# Platform Internals — backend mechanics & known gotchas

> `platform-features.md` documents the **operator feature surface** (what you can configure). This doc covers the
> **backend mechanics and known bugs** that decide whether a config actually behaves as expected. When an agent
> "works locally but misbehaves in prod", or a follow-up "is configured but never fires", the answer is almost
> always here. Source of truth: the `mk1-chat` / `mk1-tasks` platform repos and the field investigations in
> `ninjo-prompts/knowledge/platform/`.

---

## 1. The outbound message pipeline (order matters)

When an agent generates a reply, it passes through these stages **in order** before reaching the lead
(`mk1-chat/app/routes/api+/messaging+/main-handler.ts`):

1. **LLM generation** — the model produces the response text (may follow tool calls in the same session).
2. **Property replacement** — custom-property / UTM placeholders are substituted into the text.
3. **CoT reasoning-leak filter** (`app/utils/messaging/cot-filter.ts`) — strips/blocks reasoning leaks. See §4.
4. **Message splitter** — chunks the text into multiple IG/WA messages on `\n\n` and length boundaries.
5. **Send** via the Meta API.

Two consequences worth internalizing:
- The CoT filter runs **after** generation, so it's a net, not a cure — a leak inline in the same paragraph as the
  real reply survives it (see §4). Prevention lives in the prompt (`knowledge/reasoning-leak.md`).
- Chunking, follow-up timing, contact limits, and notifications are **platform behaviors** — do not re-implement
  them as prompt rules (PAT-043 in `AGENTS.md`).

## 2. Runtime Contact-Context injection (the trace blind spot)

The static SDK (`system.md` + v5Config) is **not** the whole system prompt the model sees in production. At runtime
`mk1-chat/app/services/context.service.ts` appends, after the cached static prefix:

- `## Contact Context ##` — `firstName`, `countryCode`, and the contact's **custom properties**.
- `## Conversation Summary (Historical Context) ##` — a rolling summary of prior turns.
- Publication / document / RAG blocks where applicable.

**Why this bites you:**
- A conversation whose visible message array is just `"Si"` can still carry heavy priming from this injected block.
- **Langfuse does NOT log the `system` block** (it's cached/sent separately), and the local harness only loads the
  static SDK files. So if local reproduction shows 0 leaks but prod leaks, the gap is almost always this injected
  Contact Context / Conversation Summary. To inspect it, read `context.service.ts` or pull the contact's
  `custom_properties` + conversation summary.
- Custom properties are set by an **external evaluator**, never by the conversational agent. The SDK controls the
  agent's *messages* only — never assume a prompt rule can set a property.

## 3. Known bugs & limitations (symptom → mechanism → mitigation)

Each of these is a real, documented production failure mode. Know them before you tell an operator "it's working".

### 3a. Workflow-condition NULL handling (silently blocks follow-ups)

- **Symptom:** A follow-up / custom-notification workflow is configured correctly but **never fires** (0 fires ever).
- **Mechanism:** Workflow conditions in `mk1-tasks` (Go) evaluate `NOT_EQUALS` / `NOT_CONTAINS` against a custom
  property. If the contact has **no row** for that property (very common — ~81-98% of boolean props accumulate
  NULLs because `default_value` is metadata only and doesn't auto-populate rows), the comparison yields SQL NULL →
  "condition not met" → blocked. `mk1-chat` wraps these in `COALESCE` and is NULL-safe; `mk1-tasks` historically
  did not. Real casualties: erikaespinal_investor (4 workflows on `call_booked`, 3977 NULLs, 0 fires), mentor.mitch
  (`Booking follow-up 2h`, 9866/10069 NULLs, 0 fires), micaelagallardobio (`23 Hours Follow up`, 0 fires since the
  condition was added).
- **Operator mitigation:** Use `NOT_EQUALS X` to mean **"fire unless X is confirmed"** (NULL passes — good for
  "default opted-in" semantics). Use `EQUALS false` only when you genuinely require explicit data (NULL will block).
  If a rescue/booking follow-up isn't firing, suspect a `NOT_EQUALS`-over-NULL condition first.

### 3b. `needs_attention=true` blocks ALL follow-ups (incl. rescue workflows)

- **Symptom:** A rescue / re-engagement workflow that targets conversations where the lead spoke last and the agent
  didn't answer **doesn't run** on exactly those conversations.
- **Mechanism:** `needs_attention` is the UI inbox "unread by team" flag — set TRUE on every INBOUND
  (`conversation.server.ts`), reset FALSE on every agent OUTBOUND (`send-message.server.ts`), with no auto-reset if
  the agent decides not to respond. The follow-up poller (`mk1-tasks/repository/conversation_repository.go`) filters
  `needs_attention = FALSE`, so any conversation waiting on the team is excluded — which is precisely the population
  a rescue workflow exists to recover. Measured blocking: vicunacoach `economic-barrier-static` 58.8%, gaston
  FollowUp ~42%, andrefit ~28%.
- **Status / mitigation:** No clean operator-side fix today (a platform-side opt-in to fire on unread conversations
  is the likely fix). Until then, a rescue workflow's reach is silently capped — set expectations accordingly and
  lean on `min_minutes_since_last_inbound` as the safety gate.

### 3c. Reel-reply DM keyword bug

- **Symptom:** A lead replies to a Reel with a keyword (e.g. "PSF") and the DM_KEYWORD trigger **doesn't fire**,
  even though the same keyword typed as a plain DM works.
- **Mechanism:** Instagram delivers a reel-reply as a DM with the reel URL as an **image attachment**. The platform
  generates a single ALL_DM trigger; the evaluator sees the image and returns `activated:false`, consuming the
  batch before DM_KEYWORD ever registers an entry.
- **Mitigation:** Add a COMMENT_KEYWORD fallback where reel engagement is common; the structural fix (let DM_KEYWORD
  evaluate the text field independently of attachments) is a platform change.

### 3d. CoT filter can't strip inline leaks

See §4 and `knowledge/reasoning-leak.md`. The filter only splits on `\n\n`; a leak in the same paragraph as the
real reply is blocked wholesale or passes. This is a limitation, not a bug — but it's why prompt prevention matters.

## 4. The CoT reasoning-leak filter (config surface)

Runtime post-filter, **on by default for every agent**, ~52 global regex patterns. Per outbound message it returns
`pass` / `stripped` (keeps the clean `\n\n` paragraph) / `blocked` (`NO_RESPONSE - cot_filter_blocked`). Per-agent
config lives at `config.responseFilter { enabled, extraPatterns, disabledDefaults }` — a sibling of `config.v5Config`
in the same `agent_configs.config` JSON (read at `main-handler.ts:2006`). Set it with `update_agent_config`'s
`response_filter` field. Use `responseFilter`, never `responseFormatting` (the latter overrides all default
formatting rules and breaks voice). Full treatment, including prompt-side prevention, is in `knowledge/reasoning-leak.md`.

## 5. Forensic map — where traces live

When you need to inspect what the model actually generated (tool calls, stop reason, intermediate text):

| Source | Has | Keyed by | Note |
|---|---|---|---|
| **Langfuse** ⭐ | per-call input message array + output (text + `tool_use`) + `finish_reason` + usage | `sessionId = conversation_id` | Does **NOT** log the `system` block (so the static prompt + Contact Context are invisible here). |
| **LiteLLM proxy** | token counts (incl. cache split), cost, timing, model, tags | `session_id` / team id | Content storage is OFF — metadata only, no message bodies. |
| **CloudWatch** | gateway stdout (`conversationId`, SSE event counts, stream health) | log group `/ecs/{env}-ninjo-cortex` | Timing + stream health, not raw tokens. |
| **PG `messages`** | final user-facing text only | `conversation_id` + `direction` | No tool calls, no intermediate reasoning. `message_metadata` is for IG referral context, not LLM traces. |

Rule of thumb: Langfuse for "what did the model say and call", LiteLLM for "what did it cost / cache", PG for "what
did the lead actually receive". The system prompt itself is the blind spot — reconstruct it from `context.service.ts`
+ the contact's properties when local repro disagrees with prod.

> **For the operator working through MCP:** none of these stores are reachable from the cortex-gateway MCP tools.
> What you *can* pull is `get_conversations` (delivered messages), `get_agent_metrics`, and `get_agent_insights`.
> When those aren't enough to diagnose, say so and name the trace store an engineer should check — don't guess.