How to Test AI Prompts

To test an AI prompt, run it against a fixed set of realistic inputs, judge each output against written criteria, and keep a record of exactly what was run and what came back. This guide describes the seven-criterion rubric and the test record format used for prompts on this site, and how to apply both to your own. A single good-looking answer is not a test: the point is to find out how a prompt behaves across the inputs and repeat runs it will actually meet.

What testing can and cannot tell you

A language model's output can vary between runs of the same prompt, and it can change when the vendor updates the model. So a test result is an observation about one prompt, on one model version, on one date, with one set of inputs. That is still valuable: it tells you whether the prompt does its job under known conditions and where it breaks. It does not certify that the prompt will produce a correct answer for you. OpenAI's documentation frames evaluation as the way to know whether a change made things better or worse, and recommends pinning model versions so results stay comparable; Anthropic's advises defining success criteria and building test cases before iterating on wording.

Before you run anything

1. Write down the task. One or two sentences on what the prompt is supposed to produce and for whom.
2. Describe the expected output. Not the exact words, but what must be present, what must be absent, and the format. Do this before looking at any output, so that a fluent answer does not redefine your standard.
3. Assemble test inputs. Aim for a small set that covers:
   • a typical case;
   • a minimal case, with sparse or short input;
   • a long or messy case;
   • an edge case, where a required detail is missing or two requirements pull against each other;
   • for prompts used in products, an input that tries to steer the model away from its instructions.
4. Fix the conditions. Same model and version, same settings, a fresh conversation for each run.

The seven-criterion rubric

Each criterion is a question about the prompt, answered by reading its outputs.

1. Instruction clarity

Did the model interpret the prompt the way it was meant? Look for signs of misreading: a section answered at the wrong level, a term taken in a different sense, a clarifying question that the prompt should have pre-empted. Failures here are faults in the prompt's wording, and they contaminate every other criterion, so fix them first.

2. Task completion

Did the output do the whole job? Check every requested part against the output. Partial completion, such as four of five sections, or a plan where a finished draft was asked for, counts as a miss even when the parts present are good.

3. Context handling

Did the model use the material and variables it was given, accurately? Look for supplied details that were ignored, facts that were altered, and content that appears in the output without any basis in the input. For prompts that take long documents, check details from the middle of the document, not only the beginning and end.

4. Constraint adherence

Were the stated limits respected? Word counts, exclusions, "use only the supplied notes", tone requirements, and instructions about what to do when information is missing. Count them one by one; constraints are the easiest criterion to check objectively and the most commonly violated.

5. Output structure

Does the response match the requested format? Headings in the specified order, the right columns, valid JSON if JSON was asked for, no preamble where none was wanted. If the output feeds another tool, test by actually feeding it in.

6. Reliability

Does the prompt behave the same way again? Run the same input several times and run the different inputs in your set. Wording will differ; what matters is whether the structure, completeness and constraint adherence hold. A prompt that is excellent on one run in three is not ready for reuse.

7. Practical usefulness

Could a competent person use this output for its intended purpose with a reasonable amount of editing? This is the criterion that requires judgment from someone who knows the task. An output can pass the first six and still be too generic, too cautious or pitched at the wrong level to be worth having.

The test record

For every test, record the following nine fields. The format is deliberately plain so that anyone can repeat the run.

Task:               What the prompt is meant to produce, for whom
Model and version:  Exact model name and version or snapshot ID
Date:               Date of the run
Input conditions:   Variable values used, any attached material,
                    settings (e.g. temperature, reasoning mode),
                    interface (app or API), number of runs
Expected output:    What a good response must contain, written
                    before the run
Actual output:      The response, saved in full and unedited
Observed strengths: What the output did well, per rubric criterion
Weaknesses:         Where it fell short, with the specific passage
Limitations:        What this test does not show (inputs not tried,
                    claims not verified, other models not run)

Three of these fields are often skipped and should not be:

• Model and version. "ChatGPT" or "Claude" is not enough; behavior differs between versions of the same product.
• Expected output. Without it, strengths and weaknesses are measured against nothing.
• Limitations. Every test is partial. Stating what was not covered stops a reader, including you later, from assuming more than the test supports.

Save actual outputs verbatim. A summary of an output cannot be re-examined when a question comes up.

Why numerical scores are not published

Prompt pages on this site describe strengths, weaknesses and limitations in words and do not carry a score such as 8.5 out of 10. There are four reasons.

• A number implies precision the method does not have. Rubric judgments are qualitative readings of a small number of runs. Converting them to a decimal makes them look measured when they are assessed.
• Results are tied to conditions. A score would hold only for the model version, date and inputs in the record. Detached from those, as scores always become when quoted, it would be misleading, and it would go stale whenever a model is updated.
• Scores invite comparisons that are not valid. A debugging prompt and a brand voice prompt are judged on different tasks with different inputs. Ranking them on one scale says nothing useful.
• A single figure hides the information you need. "Follows the format reliably but pads the summary when the input is short" tells you whether the prompt suits your case. A 7 does not.

A described observation, alongside a record that lets you repeat the test, is more honest and more useful than a rating.

Testing a change

When you revise a prompt:

1. Change one thing.
2. Re-run the entire input set, not only the case that prompted the change.
3. Compare old and new outputs side by side; the prompt comparison tool is built for viewing two versions together.
4. Keep the previous version until the new one is at least as good on every input.

Fixes commonly cause regressions elsewhere: a tighter length limit that removes a required section, an added example that narrows the range of outputs. Only the full set shows this.

Scaling up

For prompts running in an application, the same rubric can be partly automated. Structure and many constraints can be checked in code: does it parse, is it under the limit, are the required keys present. Criteria that need judgment can be assessed by a second model call given the rubric and the expected output, an approach both OpenAI's and Anthropic's documentation describe; spot-check those gradings by hand, since a grader is itself a prompt that can be wrong. Re-run the suite whenever the model version changes.

When to re-test

• The prompt text changes.
• The model or its version changes.
• The kind of input changes, for example longer documents or a new audience.
• A real-world output surprises you. Add that input to the set.

Testing sits at the end of the writing loop described in prompt engineering and is especially important for system prompts and prompt chains, which run on inputs their author never sees. For a prompt whose output is itself a set of tests, see the unit tests prompt.