> ## Documentation Index > Fetch the complete documentation index at: https://daily-ms-ws-body-url-encode.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Eval Suites > Spawn agents and run many scripted and simulated scenarios concurrently from a single manifest with pipecat eval suite. `pipecat eval run` tests scenarios against an agent you started yourself. A **suite** goes one step further: you list agents and scenarios in a manifest, and `pipecat eval suite` spawns each agent with its eval transport on its own port, runs its scenarios, tears it down, and aggregates the results, several runs at a time. Suites are the right tool when you have more than one agent, more than a handful of scenarios, or want a single command for CI. They are also how a [simulation](/pipecat/evals/simulated-scenarios) runs more than once: each run gets a fresh agent. Pipecat's own release evals are a manifest with 100+ example agents plus this command. ## The manifest ```yaml manifest.yaml theme={null} concurrency: 4 # how many runs execute at once runs_dir: eval-runs # logs + recordings go to // record: false # record conversation audio (audio-mode scenarios) scenarios_dir: scenarios # scenario names resolve to /.yaml # How to start each agent. {python}, {bot}, and {port} are substituted per run. spawn: "{python} {bot} -t eval --port {port}" suite: - bot: bots/support-agent.py scenarios: [greeting, capital_question, multi_turn] - bot: bots/sales-agent.py concurrency: 2 # at most two of this entry's runs at once scenarios: [greeting, weather_function_call] - bot: bots/booking-agent.py scenarios: [greeting, simulated/book_table] # a simulation, listed the same way - bot: bots/vision-agent.py runner_body: path: scenarios/vision-body.yaml # optional --runner-body data, as a file scenarios: [vision_describe] - bot: bots/turn-agent.py name: openai/gpt-4o-mini # the entry's label, since two entries share this bot runner_body: data: { model: gpt-4o-mini } # or written inline scenarios: [turn_completion] - bot: bots/turn-agent.py name: groq/llama-3.3-70b runner_body: data: { model: llama-3.3-70b } scenarios: [turn_completion] ``` Paths in the manifest (`bots_dir`, `scenarios_dir`, `runs_dir`, the `bot:` entries) resolve relative to the manifest file, so a manifest is portable: check it into your repo and run it from anywhere. A scenario name can include a subfolder (`simulated/book_table`), and a name ending in `.yaml` is a path relative to the manifest. A `scenarios:` entry names a [scenario file](/pipecat/evals/scenario-configuration#scenario-files), and the file contributes one run per scenario it holds, named `/` after the file's `name:` and the scenario's own. Scripted scenarios and simulations are listed the same way, and each scenario says which it is: `turns:` makes it scripted, `persona:` makes it a simulation. Pipecat's release evals keep the two in `scenarios/scripted/` and `scenarios/simulated/` folders, which is a convention worth copying. Scenarios are reusable across agents. One `greeting` scenario can cover every agent in the suite. ### Entry options An entry can carry three optional keys beside `bot:` and `scenarios:`: * **`runner_body:`** supplies session data the agent would normally receive in a `/start` request body, passed to it as `--runner-body`. It holds either `path:`, a YAML or JSON file resolved relative to the manifest, or `data:`, the body itself as a mapping, which the suite writes to a file among the run's logs. An agent given a file runs with the file's directory as its working directory, so a relative path inside the body (a vision agent's image, say) resolves next to the file. A body that holds such paths belongs in a file for that reason. * **`name:`** is the entry's label: what the dashboard, the `-p` filter, `results.jsonl`, and the artifact file names use. It defaults to the `bot:` path, so it is only needed when several entries share a bot and differ in their `runner_body:`, as when sweeping a model. Two entries may not run the same scenario under one label. * **`concurrency:`** caps how many of the entry's runs are in flight at once, for a provider that rate-limits. Without it, the entry runs as many at once as the suite has free. See [Concurrency](#concurrency). A bare `runner_body: ` is deprecated since 1.11.0 and will be removed in 2.0.0. Write `runner_body: { path: }` instead. ## Running a suite ```bash theme={null} pipecat eval suite manifest.yaml ``` In a terminal, a live dashboard shows each run's status, a running tally, and total time. When piped (in CI, or driven by a coding assistant), it streams one plain result line per run instead. The command exits `0` only if every run passes. Useful flags: ```bash theme={null} pipecat eval suite manifest.yaml -p support # only entries whose name or bot path contains "support" pipecat eval suite manifest.yaml -s greeting # every scenario in the greeting file pipecat eval suite manifest.yaml -s book_table/available # one scenario of a file pipecat eval suite manifest.yaml -k simulation # only simulations (or: -k script) pipecat eval suite manifest.yaml -c 8 # 8 runs at a time pipecat eval suite manifest.yaml -n nightly # output to eval-runs/nightly/ pipecat eval suite manifest.yaml -a # record conversation audio pipecat eval suite manifest.yaml -d # save full per-pipeline debug logs pipecat eval suite manifest.yaml -r 5 # run each pair 5 times ``` `-s` takes a run's full `/` name or either half of it, so a file's name selects every scenario it holds, and a scenario's name selects it in whichever file it appears. Everything except the `suite:` list can live in the manifest or be passed on the command line (the command line wins), so a manifest can be as minimal as a `suite:` list. ### Concurrency `concurrency:` is how many runs execute at once, and the suite keeps that many going until none are left. Each time a run finishes, the next one comes from the first entry in manifest order that still has runs, so an entry's scenarios finish together and no slot sits idle while any entry has work, however few entries the manifest has. When runs repeat (`repeat:`, or a simulation's own `runs:`), every entry's first attempt runs before any entry's second. An entry whose provider rate-limits sets its own `concurrency:`, and never has more than that many runs in flight, across its attempts. When it is full, the next free slot goes to the following entry's run instead. The suite's figure, or `--concurrency`, bounds the whole. ## Run output Each invocation writes to `//` (a timestamp when `-n` is omitted): ``` eval-runs/20260610_142200/ results.jsonl # one line per run logs/ bots_support-agent.py__greeting__greeting.log # the agent process output bots_support-agent.py__greeting__greeting.eval.log # the harness's decision trace bots_support-agent.py__greeting__greeting.debug.log # per-pipeline harness logs (-d only) recordings/ bots_support-agent.py__greeting__greeting.wav # conversation audio (record: true or -a) ``` An artifact's name is the bot path, then the entry's `name:` when it has one, then the scenario's `/` name with its slash written as a double underscore, so `bots/turn-agent.py` under the label `openai/gpt-4o-mini` running `turn_completion/short_answer` writes `bots_turn-agent.py__openai_gpt-4o-mini__turn_completion__short_answer.log`. `results.jsonl` carries one line per run, and every line names its `bot`, its entry's `name`, its `scenario`, its `attempt`, and its `kind`, `script` or `simulation`. A scripted record carries the outcome, duration, failures (each with a `kind`), per-turn results, and paths to its artifacts. Each turn lists its `expectations`, one per expectation the turn resolved, with whether it `passed` and what it `matched`: the marker of an `llm_marker`, a function call's signature, a reply's text. What a passed run matched is only recorded here. A simulation record carries the outcome (`passed`, `succeeded`, `error`), how the run ended (`ended_by`), the persona's turn count, every metric's score, value, `failure_kind`, and per-turn verdicts (`yes`, `no`, or `none` when the judge gave none), the judge's reason, the persona's own `end_call` claim, and the whole conversation as `messages`. Lines are appended as each run finishes, so an interrupted sweep keeps everything already done. Runs that didn't pass also carry `events_seen`. Each run executes in its own process, so a harness that loads local audio models does so on its own, and a crash in one run doesn't stop the others. `results.jsonl` is written by `pipecat eval suite`. `pipecat eval run` doesn't produce one. When a run fails, start with the `.eval.log` decision trace: it's a timestamped record of every event the harness saw, what it matched, what the judge said, and why an assertion failed. The agent's own log sits next to it. ## Testing one agent with many scenarios If you just want to run a batch of scenarios against an agent you already have running, you don't need a manifest. `pipecat eval run` accepts multiple scenario files and shares the suite's dashboard and tally: ```bash theme={null} pipecat eval run scenarios/*.yaml --bot-url ws://localhost:7860 pipecat eval run scenarios/ --bot-url ws://localhost:7860 # the whole directory ``` A directory expands to its `.yaml` and `.yml` files in filename order, non-recursively, and files and directories can be mixed in one invocation. A directory holding no scenarios is an error rather than an empty run. By default the agent is left running afterward so it can serve more evals; pass `--stop-bot` to shut it down when the batch finishes. ## Running a simulation more than once A persona doesn't say the same thing twice, so one run of a simulation proves little. A simulation's own `runs:` field tells the suite how many times to run it, and every run must pass. The suite reports a pass rate per simulation with a ✓ or ✗ beside it, and exits `1` when any run failed: ``` Pass rate: bots/booking-agent.py book_table/available 3/3 (100%) ✓ ~12.4s each ``` A run that errored (the agent never came up, the persona's LLM failed, the judge gave no verdict on the goal) is reported but kept out of the rate: it says nothing about whether the agent did its job. ## Repeating a run A behavior with a race in it passes sometimes. `repeat:` in the manifest, or `--repeat` / `-r` on the command line, runs every (bot, scenario) pair N times and reports a pass rate per pair instead of a single verdict: ```yaml theme={null} # manifest.yaml repeat: 5 suite: - bot: bots/support-agent.py scenarios: [interruption] ``` A repeated sweep is attempt-major: every entry's first attempt runs before any entry's second, each entry's scenarios as a block, so every bot meets the same machine conditions in the same stretch and a transient slowdown shows as a band across all of them rather than a regression in one. Artifact filenames gain an attempt suffix (`__001`) only when `repeat` is above 1, so a single pass keeps the filenames it always had. A repeated sweep always exits `0`. A pass rate isn't a pass or a fail, so the threshold is yours to choose: read `results.jsonl` and decide. Leave `repeat` unset for the CI gate below. A `repeat` set on the command line or in the manifest applies to every run, simulations included, even when it is `1`. It overrides a simulation's `runs:` and turns the requirement into a measurement: rates are reported and the exit code stays `0`. Leave `repeat` unset to let each simulation run its own `runs:` and gate on them. ## Suites in CI The exit code makes suites CI-ready with no extra glue: ```yaml theme={null} # e.g. GitHub Actions - name: Run behavioral evals run: pipecat eval suite manifest.yaml ``` For deterministic, key-free CI runs, prefer text-mode scripted scenarios and a judge endpoint you control. Simulations run their persona on the local Ollama model by default, so they need no extra key, but they say something different each run by design, so give them `runs: 3` and expect them to take longer. `-k script` runs the scripted half alone when you want a fast gate on every push and the simulations on a schedule. Audio-mode scenarios work in CI too, but need the harness's TTS and STT services available (local models by default, which also need more CPU).