Foxy Audit
What you'll learn
- what the testbed is, and what in it is real;
- how to try prompts in it, in the terminal or in your browser;
- how to run its scored test set;
- every flag it takes.
What the testbed is
When you install foxy-, you also get a package called foxy_. It is a small assistant for one of three kinds of work: health care, finance or law. It sits behind the same checks your own program would use.
- The checks are real. They are the same rules, the same ruleset and the same decisions a real record would carry.
- The replies are not. By default the assistant does not call any AI model. Its answers are fixed pieces of text, and the testbed says so next to each one.
It needs nothing beyond the SDK itself. By default it makes no network call and uses no key, so a run gives the same result on any machine.
Step 1: Try some prompts in the terminal
python -m foxy_testbed --sector healthcare
The testbed first prints a short banner. It names the tag and the mode, and it says what that setting checks and what it does not. Then type a prompt and press Enter. Type /, or press Ctrl-D, to leave.
What you should see, for two prompts from a real session:
you> What are the visiting hours?
------------------------------------------------------------------------------
[ALLOWED]
nothing fired. The prompt was delivered as typed.
rules (none fired)
reason none
reached yes, the provider was called with this prompt
ruleset 2026.09.3
reply (mock placeholder: no fixture for this prompt, not model output):
[no fixture for this prompt] The mock provider has no written answer for
it, so there is nothing to show you here -- and inventing one would make
this demo dishonest. The enforcement result above is real and was
produced by the same guard a live provider runs behind. Point --provider
at openai or gemini with your own key for a real answer. (prompt digest
e372251969ca)
------------------------------------------------------------------------------
you> My SSN is 123-45-6789
------------------------------------------------------------------------------
[BLOCKED]
the prompt never left this machine. The provider was not called, so there
was no reply to withhold.
rules phi.ssn_pattern
reason phi
reached no, the provider was never called
kept back phi.ssn_pattern
ruleset 2026.09.3
No reply: the prompt was stopped before the provider was called.
------------------------------------------------------------------------------
The health care assistant uses the hipaa tag and block mode. The first prompt was safe, so it was sent. The mock provider has a written answer only for the testbed's own test prompts, typed exactly, so for this one it says it has none. The second prompt had a social security number in it, so it was stopped before any model saw it.
Step 2: Run the scored test set
Each kind of work comes with a set of test prompts. Some should be stopped. Others should be answered, because an assistant that refuses everything is useless. The testbed scores both.
python -m foxy_testbed --sector healthcare --probe all
What you should see: a long report that ends with a summary line. This one is from a real run:
PASS | enforcement 7/7 | assistance 6/6 | 2 known gap(s) open
- Enforcement counts the risky prompts that were stopped.
- Assistance counts the safe prompts that were answered.
- Known gaps are risks that nothing catches today, such as a medical record number. They are listed, so the score never reads as full cover. A gap does not fail the run.
Step 3: Try it in your browser
python -m foxy_testbed.web
This opens a page on your own machine, at http://127.0.0.1:8787. You type a prompt and watch the checks decide. The page only listens on your own machine, and there is no flag to change that.
Step 4: Use a real model, if you want to
The testbed can call a real model with your own provider key:
python -m foxy_testbed --sector finance --provider openai
It reads that key from --, or else from OPENAI_ or GEMINI_. Your prompts then go to that provider, under your key, as they would from your own program. They still do not go to Foxy Audit.
To send the testbed's records to your Foxy Audit ledger too, add --, and set FOXY_ to your backend's address, such as https://. Without that address, the records go to a backend on your own machine. With no value after the flag, it reads FOXY_. Without the flag, the testbed sends nothing to Foxy Audit, whatever is set on your machine.
--, -- and -- work only in the chat. The scored test set sends no records and checks no rows, so it refuses them.
-- and -- are two different keys: one for the model provider, one for Foxy Audit.
Flags of python -m foxy_testbed
| Flag | What it does |
|---|---|
-- |
the kind of work: healthcare, finance or legal. It is required. |
-- |
runs the scored test set instead of the chat |
-- |
observe, block or redact. The testbed's default is block. |
-- |
mock, openai or gemini. The default, mock, needs no network and no key. |
-- |
uses a different model from the provider's default |
-- |
the key for a live model provider |
-- |
your Foxy Audit key, so the turns reach your ledger. With no value, it reads FOXY_. It also needs FOXY_. Refused with --. |
-- |
your own JSON export of your ledger, so the testbed can check a row against it. Refused with --. |
-- |
the salt file, for rows made with salts. Refused with --. |
Flags of python -m foxy_testbed.web
| Flag | What it does |
|---|---|
-- |
the port on 127.0.0.1. The default is 8787. |
-- |
the mode the page opens in. The default is block. |
-- |
mock, openai or gemini. The default is mock. |
-- |
uses a different model from the provider's default |
-- |
the key for a live model provider |
-- |
your Foxy Audit key. With no value, it reads FOXY_. It also needs FOXY_. |
-- |
your own JSON export, so the page can check a row |
-- |
the salt file, for rows made with salts |
-- |
does not open a browser window |
The web page has no -- flag: you pick the kind of work on the page.
If it goes wrong
| What you see | Why | What to do |
|---|---|---|
No module named foxy_ |
The SDK is not installed in this Python | Run python -. |
| The replies say there is no fixture | The mock provider has no written answer for that prompt | That is expected. The check result above it is still real. |
| The browser does not open | The machine has no browser, or you passed -- |
Open http://127.0.0.1:8787 yourself. |
| Nothing reaches your ledger | Records are only sent with --, and only to the address in FOXY_ |
Add --, and set FOXY_ and FOXY_. |
Could not start: -- |
The scored test set sends no records | Drop the flag, or run the chat instead. |