The spool and delivery

What you'll learn

• what the spool is and where it lives;
• how records are sent in the background;
• what happens when the network is down, and after a restart;
• the limit on how many records can wait;
• how to make a call fail when its record cannot be delivered.

What the spool is

The spool is a small database file on your disk. The SDK saves each record there first, before your function returns. Then a background thread sends it to Foxy Audit.

This order matters. Once the record is in the spool, it survives a crash, a restart or a network outage. Nothing is lost if the network is slow.

• Where it is: ~/.foxy-audit/spool.sqlite3 by default. On Windows, ~ is your user folder. You can move it with FOXY_SPOOL_PATH.
• When it is made: when your program makes a client with an API key set. With no key, the SDK does not make a spool.
• What is in it: hashes, labels and delivery details, and the backend's delivery receipts, which it keeps for 7 days by default. There is no prompt text and no answer text in it.
• What is not in it: your API key. Each record holds a fingerprint of the key and the address it goes to, never the key itself.
• What is next to it: commitment.key, the key your hashes are made with. The SDK makes it the first time, unless you set FOXY_COMMITMENT_KEY. It never leaves your machine, and it is limited to you in the same way as the spool. Back it up: without it, you cannot prove which text a record came from.
• Who can read it: the SDK limits the spool file, and the two helper files SQLite keeps beside it (ending -wal and -shm), as far as it can:
• on Linux and macOS, to its owner (file mode 0600);
• on Windows, when your own account owns the file, to that account alone;
• on Windows, when a program running as administrator made the file, to the Administrators group and the account the program runs as. The same person can then open it without administrator rights, and other people on the machine cannot. The SDK logs one INFO line for each such file;
• a file the SDK did not make, such as one another account owns, is left as it is.

After a restart, a program running as administrator did not make the spool file, but the file still carries the two entries the SDK gave it, so the SDK knows it is your own. A -wal or -shm file SQLite makes again gets the same two entries. A spool whose entries name another account is left as it is.

How records are sent

A background thread wakes up every second. Each time, it takes up to ten waiting records from each spool file and sends them to your backend in one request. It groups them by key and address, and each record goes only with the key it was made with.

Sending never happens inside your call: the thread runs next to your program. Saving the record does happen inside your call, on your own thread. It normally takes milliseconds, and longer while the file is being cleaned or another program has it locked.

When your program exits normally, the SDK spends up to five seconds sending what is still waiting. Anything it cannot send in that time stays in the spool for next time.

When the network is down

If a send fails, the records stay in the spool. The SDK tries again later. Each time it waits a little longer, up to five minutes between tries. Records are never thrown away because a send failed.

When the network comes back, the waiting records go out on their own. You do not have to do anything.

After a restart

When your program starts again, making the client with the same key and address wakes up delivery. Records left over from last time are sent first, in the background.

If the program was killed hard, for example with kill -9, the exit step did not run. The records are still in the spool, and they go out on the next start.

Step 1: Look at what is waiting

foxy doctor shows the spool first, before anything else.

What you should see, when nothing is waiting:

  [OK] Local delivery: 0 event(s) pending in the spool.

If something on this machine needs you, that line turns into [X], and a line under it says what. Lines under it that start with an arrow are notices: records waiting under another key or address, or a clean-up of the file that is still due. A notice can appear under a green line, and it never changes the exit code. For more detail, use foxy spool status. The foxy command chapter lists every line.

The limit on waiting records

A spool holds up to 100,000 waiting records for each key and address. When that limit is reached, new records are refused, not saved. Nothing already in the spool is thrown away to make room, because a thrown-away record is evidence nobody would ever see.

When a record is refused, a warning is logged. If you use audit_required=True, your call gets an error instead. The whole file also has a ceiling, at twice that limit.

You can change the limit with FOXY_SPOOL_MAX_ROWS. A value of 0 turns the limit off. Only do that if you would rather let the file grow than refuse records.

If your API key changes

Each record is sent only with the key and address it was made with. If you change your key or your backend address, records made with the old ones stay in the spool, and the new settings do not send them. Setting the right key fixes new records only.

To send the waiting ones, you have two choices:

1. Run foxy spool drain once with the old key and address set. It sends that key's records. This works only while the old key still works.
2. Run foxy spool rekey to move the old key's records to the new key. Only do this when both keys belong to the same workspace. The command reads the old key from FOXY_OLD_API_KEY or asks you for it, and it needs --from-backend if the address changed too. It asks you to confirm, and it shows you exactly what it will move.

rekey needs the old key itself, but not a working one: a key that was revoked can still be moved from. A key that is lost is different. Its waiting records can never be sent or moved.

Both commands are explained in The foxy command. Neither one deletes a record.

Step 2: Make a call fail when its record cannot be delivered

By default, delivery is best effort: your call returns as soon as the record is saved in the spool. For some work, you may want the call to fail if its record cannot reach the ledger. Turn on audit_required:

from foxy_audit import FoxyClient, AuditRequiredError

foxy = FoxyClient(audit_required=True)


@foxy.audit(policy="default")
def ask_model(prompt):
    return your_model.generate(prompt)


try:
    answer = ask_model("Summarize this contract.")
except AuditRequiredError:
    answer = None   # no proof of this call, so do not use the answer

Now each call waits for the backend to confirm it has the record. It waits for your timeout setting, which is 5 seconds by default, or at least 1 second. If that record is already on its way when the time is up, the wait runs on until that send ends. If the confirmation does not come, the call raises AuditRequiredError.

audit_required needs an API key. With no key, nothing is sent, so there is nothing to confirm, and no call fails.

One thing surprises people. The record was usually saved in the spool before the wait began. So after an AuditRequiredError, the record may still reach the ledger later. The error means "no confirmation in time", not "lost". The one exception is a full spool: then the record was refused and never saved, and the error's cause is SpoolFull.

In a container

A container's own disk is thrown away when it restarts. If the spool is there, the records made in the moments before a restart are lost with it. Put the spool on a volume that survives restarts:

export FOXY_SPOOL_PATH=/data/foxy/spool.sqlite3

The commitment key file is kept next to the spool, so it moves to the volume too. In a serverless function, which has no volume, set FOXY_COMMITMENT_KEY from your secret store instead. Otherwise each start makes a new key.

Upgrading from an older version

Version 1.18.0 stopped keeping API keys in the spool. When a newer SDK opens an older spool file, it takes the keys out of the records it can deliver itself, a batch at a time. Later, in the background, within the hour, it rebuilds the file, so the old keys do not stay in unused space inside it. Until then, foxy doctor shows a notice about it. foxy spool compact does the rebuild at once.

• If several programs share one spool file, upgrade them all at the same time. An older program cannot send records the newer one has changed. It keeps trying and failing until it is upgraded. The records are kept, never lost. foxy doctor, run with one program's key, shows the other programs' waiting records as a notice.
• If you copied an old spool file somewhere before the upgrade, that copy still holds your key. Make a new key in the dashboard to be safe.

If it goes wrong