Response scanning

What you'll learn

• what the response scan looks for, and why;
• the three settings: observe, block and off;
• which setting to choose, and what block costs;
• how the scan works on answers that arrive in pieces;
• what the scan does not promise.

Why check the answer

The prompt checks ask: what is my program sending to the model? The response scan asks a different question: what will my program do with what came back?

An answer can hurt the program that uses it. It might hold a web page script that runs when the answer is shown. It might hold a database command that deletes a table, or a link to a private address inside your network. It might repeat a secret or a person's details. This family of risks is known as LLM05 in the OWASP list of risks for AI programs.

What it looks for

The rule catalogue lists every rule and what it matches.

It does not look for prompt injection in answers. An answer that says "ignore all previous instructions" is usually the model quoting or explaining the phrase, and flagging it would be noise.

The three settings

The scan is a set of patterns, not a reader of meaning. It flags what its rules match, and nothing else. An answer that does harm in a way no rule matches passes under every setting.

Set it on the client, or in the environment:

from foxy_audit import FoxyClient

foxy = FoxyClient(response_scan="block")

or FOXY_RESPONSE_SCAN=block. There is no setting on a single decorator, on purpose. A blocked answer raises an error in whatever code called your function, so it should be one choice for the whole program, not a surprise in one place.

Which to choose

observe is the default because a mistaken finding must not break your code. Under block, a flagged answer raises an error inside your program, and nobody should meet a new error in production only because they upgraded a package. observe records what block would have stopped, so you can see how often it would fire before you turn it on.

For PHI and other regulated data, turn on block: FoxyClient(..., response_scan="block"), or FOXY_RESPONSE_SCAN=block in the environment. Under observe, an answer that repeats a patient's details, a secret or a harmful script still reaches your code, and it is only recorded. Under block, one that the scan flags does not. An auditor will ask about that difference.

What block costs:

• A flagged answer becomes an error your code must handle: FoxyResponseBlocked. It is not a FoxyPolicyBlocked, because the model call did happen and it was paid for.
• A mistaken finding costs you a real answer. Watch the observe records for your own traffic first.
• An answer that arrives in pieces is cut at the first flagged piece. The pieces before it have already reached your code. The next section explains this.

The default does not change: observe stays the default for everyone, and block is a choice you make for each program.

Step 1: Handle a blocked answer

With block on, catch the error where you call your function:

from foxy_audit import FoxyClient, FoxyResponseBlocked

foxy = FoxyClient(response_scan="block")


@foxy.audit(policy="default")
def ask_model(prompt):
    return "Here is a page: <script>alert(1)</script>"


try:
    print(ask_model("Make me a web page"))
except FoxyResponseBlocked:
    print("The answer was blocked, so it was not shown.")

What you should see:

The answer was blocked, so it was not shown.

The record says response_blocked, and it lists the rule that fired. The answer itself is never saved or sent.

Answers that arrive in pieces

Some programs show the model's answer as it arrives, a few words at a time. This is called streaming. A piece that has already reached your code cannot be taken back.

Under block, the SDK does three things for a streamed answer:

1. It checks each piece before passing it on.
2. It keeps the last 256 characters of the answer and checks them together with the next piece. So a match split across two pieces is still caught. Under hipaa and gdpr, that includes a social security number split like 123-45 and -6789.
3. It stops the stream at the flagged piece. That piece and the rest never arrive.

Two limits follow from this, and they are real:

• A pattern whose two halves are more than 256 characters apart is not caught.
• The check adds a little time to each piece.

The error is raised inside your loop, at the moment the stream is cut. So put the try around the loop, not only around the call:

from foxy_audit import FoxyClient, FoxyResponseBlocked

foxy = FoxyClient(response_scan="block")


@foxy.audit(policy="default")
def stream_answer(prompt):
    yield "Here is a page:"
    yield "<script>alert(1)</script>"


try:
    for piece in stream_answer("Make me a web page"):
        print(piece)
except FoxyResponseBlocked:
    print("The rest of the answer was blocked.")

What you should see:

Here is a page:
The rest of the answer was blocked.

Under observe, a stream is checked once, after it ends.

A stream that was cut after some pieces were sent is recorded with the event type stream and the decision response_truncated, not as blocked. The pieces that were sent are in your program, so calling that a block would be untrue. Only a stream cut at its first piece, before anything reached your code, is recorded as response_blocked.

It never changes an answer

There is no "redact" for answers, in any mode. Changing text that your code will read, store or show could break your program, so the scan only records or blocks.

When it cannot read the answer

The scan understands the answer shapes of the common model libraries, and plain text. When it meets a shape it does not know, it records one of two rules:

Neither one ever blocks an answer. "We could not read this" is missing evidence, not a finding.

When evidence is required too

If you also set audit_required=True and the record cannot be delivered, a blocked answer still raises FoxyResponseBlocked. The error has audit_delivery_failed set to True, so you can tell the two problems apart. With the ask_model from Step 1:

import logging

from foxy_audit import FoxyResponseBlocked

try:
    answer = ask_model("Make me a web page")
except FoxyResponseBlocked as error:
    if error.audit_delivery_failed:
        logging.error("An answer was blocked, and its record was not confirmed.")

A blocked prompt works differently: under audit_required=True it can raise AuditRequiredError instead of FoxyPolicyBlocked. Modes shows how to catch both.

If it goes wrong