# Runbook — "look at this conversation, something's off" > **Symptom.** *"mirá esta conversación, algo anda mal"* · *"esta conversación salió mal, no sé qué > pasó"* · *"this one went badly, can you check"*. > > The generic entry point, and the one that most often ends in a wrong answer. The operator has a > feeling, not a diagnosis. Your job is to convert it into one specific claim before touching > anything — and the instinct to reach for the prompt first is what produces "it's the prompt" as a > conclusion when the cause was configuration. ## Step 0 — get the claim, not the vibe Read the conversation and name **one** thing that should have happened differently, in a sentence that could be true or false. "The tone is off" is not a claim. "It offered a discount that does not exist" is. Then classify it. Everything downstream depends on this split, and it takes one read: | The agent… | Go to | |---|---| | said something factually wrong | **A — content** | | said something it should never say, or narrated its own logic | **B — leakage** | | went quiet when it should have spoken | `agent-not-answering.md` | | kept talking when it should have stopped | **C — no stop condition** | | sounded wrong, but nothing it said was false | **D — register** | ## Step 1 — check what the agent actually had loaded Before calling anything an invention, confirm what the agent was working from **at the time it spoke**. Prompts and data change; a message that looks like a hallucination is frequently the agent faithfully repeating what its knowledge base said that day. - **Test:** `get_agent_config`, then check its `current_prompt_version_id` against the agent's `changelog.md` to find what was actually deployed on the day the message was sent — not what is live now. - **Stop condition:** the claim is in the loaded data → **it is a data bug, not a model bug**. Fix the data. Do not add a rule telling the agent not to say it. This step reverses the diagnosis often enough to be worth doing every time. --- ## A — The agent said something wrong ### A1 — Stale data duplicated across files Facts written into several SDK files get updated in some and not others, and the agent then mixes current and stale versions. - **Test:** search **all** SDK files for the wrong fact, not just the knowledge base. - **Fix:** update every occurrence, then reduce the duplication. *Evidence: a wellness-retreat agent, twice in two weeks — event timing and a retreat location encoded in around fifteen places and updated in none of them.* ### A2 — The fact is nowhere and the model filled the gap - **Test:** search the SDK for the topic. Nothing at all is the signal. - **Trap:** a negative search only counts if your search terms match how the prompt actually phrases it. We have written up an "invention" that turned out to be a real instruction, missed because the search used the operator's vocabulary and the prompt used the creator's. - **Fix:** add the fact to the knowledge base. If the agent should refuse instead, demonstrate the refusal in an example. *Evidence: a pet-care agent — a hardcoded keyword response recommending medication, and a coverage area the knowledge base got wrong. Both were data, not model behavior.* ### A3 — Two files disagree and the model picked one When the prompt says one thing and the examples show another, the examples usually win. This is not the model being wrong; it is the configuration being contradictory. - **Test:** find the instruction and the examples covering the same situation and compare them. - **Stop condition:** they conflict → **you're done**. - **Fix:** make the examples match the intended behavior. Changing only the rule will not hold. *Evidence: an accounting agent — the examples and a data file contradicted the prompt on handoff, two to one, and the agent revealed the handoff. A B2B services agent, 2026-07-17 — the examples disqualified a lead the flow explicitly allowed.* --- ## B — The agent said something it should never say ### B1 — Reasoning leak The agent narrates its internal logic to the lead — thresholds, guard names, what it is about to verify. Distinctive and fixable; see `knowledge/reasoning-leak.md` for the full treatment. - **Fix:** two independent levers, a prompt fix and a platform-level output filter. Use the filter for anything shaped like a leak, and the prompt fix for the habit. Do not try to solve it with more rules about not explaining. ### B2 — A resource or link went out with no context, or to the wrong lead Usually a routing conflict: a topic map or keyword index authored before a newer campaign keyword existed, now matching two ways. - **Test:** `list_triggers` plus the agent's resource routing. Look for a topic that two entries can claim. - **Fix:** remove the ambiguity rather than adding a tiebreak rule. *Evidence: a personal-brand agent — a topic index written before a campaign keyword went live; the overlap sent a podcast URL with no context. Pure removal resolved it.* ### B3 — Format the platform strips or the operator banned Inverted punctuation, em dashes, emoji, bullet lists. Two different owners: some formatting is removed by the platform after the model writes it, some is the model copying the style of its own SDK files. - **Test:** compare what the model produced to what the lead received. If they differ, it is the platform's formatting layer, not the prompt. - **Fix:** platform formatting is configuration. Style the model produces is fixed by cleaning the SDK files it is imitating — and format prohibitions have to stay as explicit rules, because an example cannot demonstrate the absence of something. *Evidence: an education agent, validated — inverted punctuation stripped by a default formatting rule that applied because the agent's own formatting config was unset, not by anything in the prompt.* --- ## C — The agent kept going when it should have stopped - **Test:** `list_contact_limits` and `list_custom_properties`. Is there a terminal property, and does anything consume it? - **Stop condition:** no contact limit gated on the terminal property → **you're done**. The agent has no way to know it should stop. - **Fix:** `upsert_contact_limit` gated on that property, plus a notification so the operator is told. A property with nothing consuming it is inert. - **Do not** solve this with a prompt rule about when to stop talking. Stopping is a platform capability; the agent cannot see conversions. *Evidence: recurring. A wellness-retreat agent — it continued three or more exchanges past the natural endpoint with unqualified leads. A finance agent — a refund complaint needed silence; the platform layer existed but the first message still got through because the evaluator flips the property only after the agent has answered. If the first message must not go out, the gate has to be a trigger-level guard, not a property.* --- ## D — It sounded wrong but nothing was false This is a register complaint, and it is legitimate — but it is not a bug, and rules will not fix it. - **Test:** find three messages the operator dislikes and three from the creator's own writing covering the same moment. - **Fix:** replace or add examples drawn from what the creator actually writes. Style lives in the examples; a rule describing tone produces a different robot, not a warmer one. - **Trap:** do not attribute a style rule to the creator unless they literally said it. *Evidence: a personal-brand agent — three versions chasing a mirroring problem with rules before the fix moved to examples. A high-ticket coaching agent — the operator wanted a specific opener and a merged greeting; supplying their wording resolved it directly.* --- ## When to stop You have gone far enough when you can state the cause in one sentence naming a layer — data, example, trigger, property, contact limit, formatting, or model. If you cannot, do not guess at the prompt. File it with: the conversation id, the specific message, what should have happened, the config version in force at the time, and which of A–D you ruled out.