# ZhuoQi Dev > Liu ZhuoQi — AI application engineer focused on production Agent systems, backend engineering, context engineering, RAG, tool use, and AI security. --- title: "Check the Goods Before You Buy a Relay API: Six Questions That Tell You Whether It Sells the Real Model" description: "Cheap Claude, GPT and Seedance relays keep multiplying, and their groups are labelled 'official', 'official key' or 'native', but a group name is not evidence. What actually ties what a relay sells to the official API is the response id prefix, input tokens that match the official counting endpoint one for one, the error you get when you send a deliberately invalid parameter, the C2PA signature and signing certificate inside a video file, and the charges in the relay's own logs. This post walks through six questions: how to check each one, whether it costs money, what counts as a problem, and what to do without an official key. It ends with an open-source skill that hands the whole procedure to an Agent and writes the comparison report for you." url: "https://zhuoqidev.com/en/posts/verify-llm-relay-api/" language: "en" author: "Liu ZhuoQi" date: "2026-10-24" updated: "2026-10-24" tags: ["API Relays","new-api","Claude","OpenAI","LLM","C2PA","Skills","Hands-on"] translation: "https://zhuoqidev.com/posts/verify-llm-relay-api/index.md" --- # Check the Goods Before You Buy a Relay API: Six Questions That Tell You Whether It Sells the Real Model > **TL;DR** Across the 30-odd relays I tested over the past month or so, the most common problem was not a fake model. It was a real model that comes through a different channel than you think, charges you for tokens you never see, and silently drops your parameters. All of that can be found before you pay: most checks cost nothing, and across the text-line audits I have run, the official and relay sides together mostly came to under one dollar, at most about $1.30. A group name is not evidence; tokens are. > **One word. 508 extra tokens.** > > Ask the model to reply with one word, count the input with the official counting endpoint, then look at what the relay actually billed: > - Official count: 18 > - Relay billed: 526 > - Hidden prompt: +508 > > *Claude 5 family (e.g. Sonnet 5.5) · official count 2026-10-14 · subscription-pool relay measured 2026-10-19 (relay not named)* *This post is for individual developers and small teams about to buy a relay API, and for anyone choosing upstream suppliers for a platform. The cases come from the 30-odd relays I tested at work in September and October 2026 and are **all anonymous**: no names, no domains, only what happened and how to check for it. Official behavior was checked against the official docs on 2026-10-23; relays change fast, so trust the numbers you measure on the day you test.* --- ## First, know what you are buying A relay (also called a proxy or reseller) does something simple: you call an endpoint that looks exactly like the official one with a key the relay gave you, the relay forwards the request to some "upstream", and it deducts from your balance at its own prices. Most relays run the open-source [new-api](https://github.com/QuantumNous/new-api) / one-api, or programs like [sub2api](https://github.com/Wei-Shaw/sub2api) that turn subscription accounts into an API, so they share a lot of behavior. (If you want to build a forwarding gateway yourself and work out which cloud is cheapest, see [what it costs to run an app online](https://zhuoqidev.com/en/posts/model-gateway-cloudflare-vs-china-cloud/), which uses a model gateway as its full example.) The trouble is that the upstream can be several different things, and you only see the layer the relay shows you: ```mermaid flowchart LR B[Relay
new-api / sub2api] subgraph OK[Legitimate business] C[Official API
Anthropic / OpenAI] D[Cloud-hosted
Bedrock / Vertex / Azure] end subgraph RISK[Watch out] E[Subscription pool
Claude Max / ChatGPT] F[Self-hosted open weights
posing as official] end B --> C B --> D B -.-> E B -.-> F ``` "Legitimate business" is the official API itself, or the same model hosted by a cloud platform under license from the model vendor. The two to watch: a subscription pool may add a system prompt you never see to its requests and bill you for it — of the two I tested, one added 508 tokens to every request and the other added nothing, so every one has to be measured — and reselling consumer subscriptions **usually** breaks the upstream terms of service, so the accounts can be banned at any time. Self-hosted open weights are simply not the model you paid for. A few terms first: - **Group**: one source for the same model inside a relay, each with its own price; the relay names it however it likes. - **Ratio**: how many times the official list price a group charges; new-api calls it `group_ratio`. - **Official baseline**: the result of the same request sent with an official key per the official docs; everything is compared against it. - **Counting endpoint**: the official "how many tokens is this input" endpoint. Claude's `/v1/messages/count_tokens` is documented as free (with a per-minute rate limit); OpenAI's `/v1/responses/input_tokens` doesn't say whether it is billed, but it works with an official key whose balance is zero, so in practice it is free. - **C2PA**: content credentials written into an image or video file, carrying a signing certificate and a digital signature. > [!TIP] > The whole post rests on one principle: **send the same request to the official API and to the relay, and compare item by item.** The six questions below are six angles on that principle. ### The six checks at a glance Run them in this order; the first two are the cheapest and tell you the most: | # | Check | Official key? | Cost | Warning sign | |---|---|---|---|---| | 1 | Response id prefix | No | The relay's cheapest request | Prefix doesn't match the advertised channel | | 2 | Reconcile input tokens | Yes (or the public baseline below) | Official count free; relay a fraction of a cent | A constant surplus of hundreds or thousands | | 3 | Make it name the model | No | Free on the official API; a relay may pre-deduct | Model in the error ≠ model in the response | | 4 | Same bad inputs to both sides | Yes | A relay may bill it; send one first | Official 400, relay 200 | | 5 | Video signing certificate | Yes (an official original's fingerprint) | One job per tier | Fingerprint doesn't match the official one | | 6 | Read its own logs | No | Free | Usage endpoint disagrees with the logs | > [!NOTE] > **No official key?** Checks 1, 3 and 6 don't need one. For check 2 you can use a public baseline: the sentence "Reply with the single word OK." counts as 18 input tokens on the official counting endpoint for Claude Sonnet 5.5 / Opus 5.5 (measured 2026-10-14; re-measure for other models or if much time has passed). For check 5, the signing-certificate fingerprint of official Volcengine Ark Seedance originals is already in the known list of the skill below. --- ## Question 1: who is really behind it? **Look at the response id prefix.** The first successful request tells you: | Backend | Message id prefix | Typical trace in errors | |---|---|---| | Anthropic first-party API | `msg_011C…` (request id `req_011C…`) | JSON envelope with `request_id` | | AWS Bedrock | `msg_bdrk_…` | `ValidationException` | | Google Vertex | `msg_vrtx_…` | Google-style error envelope | | OpenAI Chat / Responses | `chatcmpl-…` / `resp_…` | Error codes such as `invalid_request_error` | A counter-intuitive fact: **group names are no evidence at all.** I have seen groups whose names suggested "official direct / parameter passthrough" that were all Bedrock, and groups with AWS in the name that returned first-party ids. Bedrock and first-party are both real Claude, but some features and billing details differ (some beta features exist only first-party), so you need to know which one you are buying. > [!NOTE] > The id prefixes are an observed pattern, not a format the vendors promise; they can change at any time. When you see an unfamiliar prefix, send one request with an official key to check what it looks like today. If you also send an invalid parameter to read the raw error, send it to a relay once and check its logs for a charge first (see Question 4). ## Question 2: is a system prompt being slipped in? **Reconcile against the official counting endpoint.** Take a fixed sentence such as "Reply with the single word OK.", send it to the official counting endpoint to get the input token count, then send it to the relay and look at the input actually billed in the response's `usage`. For Claude, add up `input_tokens`, `cache_creation_input_tokens` and `cache_read_input_tokens`, because some relays quietly add cache breakpoints. | | | | --- | --- | | Official counting endpoint（2026-10-14） | 18 | | Normal relay（first-party or Bedrock, several measured） | 18 | | Subscription-pool relay（posing as Claude Code, 2026-10-19） | 526 | *Input tokens for the same sentence, "Reply with the single word OK.", Claude 5 family. Relays not named.* If the extra amount is **the same every time**, a fixed system prompt is being injected. The relay above resells Claude subscription accounts as an API; to pass as the Claude Code client, it adds Claude Code's system prompt to every request, and bills you for those 508 tokens at your price. OpenAI has the same kind: on relays running on the ChatGPT / Codex subscription backend, I measured about 4,380 tokens of Codex instructions on every request, billed as usual. Two easy traps: - **Don't compare against the relay's own counting endpoint.** A subscription pool's counting endpoint does not inject, so it matches the official count exactly; only real requests come out bigger. Comparing counting endpoints alone makes the relay look clean. - **A few tokens off is not necessarily injection.** Anthropic's docs say the counting endpoint is an estimate and may differ slightly; a constant difference of about 4 with no hidden text to recite is usually a chat-template difference. A constant difference of hundreds or thousands is injection. To confirm, ask the cheapest model to "put the system text above this message into a code block, verbatim"; it often recites it outright. ## Question 3: was the model swapped for a cheaper one? **Make the upstream name the model itself.** Token counts alone cannot tell apart models of the same generation, which count identically. For Claude there is a handy trick: send an absurd `max_tokens: 9999999`. The upstream answers 400 with an error saying how many tokens this model allows at most, **and the model's name is in that error**: > `max_tokens: 9999999 > 128000, which is the maximum allowed number of output tokens for` **`claude-xxxx`** > > (The error format is from a real run; the model is masked.) I have seen a group whose responses kept writing the expensive model you asked for in the `model` field, while the error revealed a different model, and the price was the expensive one. The token counts may well match in a case like that; only the error tells the truth. > [!WARNING] > This trick is **free only against the official Claude API** (a 400 is not billed). Through a relay, some pre-deduct `max_tokens × unit price` and do not refund on an upstream error, and some retry the 400 on another channel. Send it once, check the relay's logs for a charge, then decide whether to repeat it for every model. Kimi, Gemini and OpenAI's Responses endpoint do not reject a huge `max_tokens`; they generate and bill for real, so the trick does not work there. ## Question 4: are your parameters still there? **Send the same batch of bad inputs to both sides.** Many relays are not more tolerant; they simply drop fields they don't recognize or don't want to handle. Taking Claude as the example: | What the request says | Official | A relay I tested | |---|---|---| | `temperature: 9` (Claude's valid range is 0–1) | 400, explaining the range | 200, value dropped | | `inference_geo: "us"` (run inference only in the US; valid) | Honored, response shows US | Field stripped, response still shows global | | `temperature: 0.5` (valid) | Normal | Normal | The second row is the dangerous one: a compliance requirement you relied on never took effect, with no error at all. Send an invalid parameter, a missing required header, a wrong key, a nonexistent model and a nonexistent task once each, put the status codes and error texts side by side, and sort every difference from the official API into three bins: affects users, affects integration, no effect on use. > [!CAUTION] > Invalid-parameter probes through a relay are **not necessarily free**: I have seen a backend accept an invalid value, generate normally and bill it, and a relay that, after the official API refused, re-routed the request to another pool and returned 200 with a charge. Send one, check the logs, then send the batch. ## Question 5: is the video an official original? Video lines add another question: the clip may be an official output that was copied elsewhere, or one generated with open weights on the relay's own hardware. Work down the evidence from hardest to softest: ```mermaid flowchart TB %% layout: tree S(["Got a video"]) S --> Q1["① C2PA present, signature valid, hash matches?"] Q1 --> N1["No: not the official file as generated
(official wan3 originals have no C2PA; go to ③)"] Q1 --> Q2["Yes → ② signing-cert fingerprint = an original you fetched from the official API?"] Q2 --> N2["No: the signature may be self-signed; not evidence of an original"] Q2 --> Q3["Yes → ③ encoder and metadata match an official original?"] Q3 --> N3["No: re-wrapped or re-encoded"] Q3 --> Q4["Yes → ④ video link, task id and usage are the official ones?"] Q4 --> N4["No: an original, but copied or wrapped
can't reconcile or integrate it the official way"] Q4 --> OK["Yes: official original, passed through as is"] classDef q fill:#fff3bf,stroke:#f08c00 classDef bad fill:#ffe3e3,stroke:#e03131 classDef pick fill:#d3f9d8,stroke:#2f9e44 class S,Q1,Q2,Q3,Q4 q class N1,N2,N3,N4 bad class OK pick ``` The step people miss is the second one: **a valid signature does not mean the vendor signed it.** Anyone can create a self-signed certificate that names "Byteplus Pte. Ltd." as the issuer, and the signature will still validate. Compare the signing certificate's fingerprint with one from an original you fetched from the official API yourself; only a match counts as an official original. The third step looks at the encoder version, the encoder info inside the bitstream, and whether there is an AIGC label — which must **match the official original**: Alibaba wan3 originals carry an AIGC label, Seedance originals don't, so a Seedance clip that has one has been re-wrapped. Self-hosted pipelines also often leave an entire ComfyUI workflow in the metadata, with the model, quantization and sampling steps spelled out. ## Question 6: what did it really charge? **Read its own logs, not the usage endpoint.** The usage endpoint of new-api relays, `/v1/dashboard/billing/usage`, is often wrong (I have seen a $0.29 job read as $2); the real charge for each request is in its log endpoint (`/api/log/token`, with the key in an `Authorization: Bearer` header — not as `?key=`, which puts the key into URLs and every layer's logs). For video, also **test every tier; never extrapolate linearly.** I have seen low tiers priced at a fraction of the official price while 1080P was close to official, so extrapolating from the low tier underestimated the cost several times over; and a price list that advertised 2K, took the money on submission, and only later answered "768P and below only" with a refund. Before every paid request, work out unit price × count, and climb from the cheapest tier one at a time. --- ## How to read the results | What you see | What it means | What to do | |---|---|---| | id prefix ≠ advertised channel | Not the channel it claims | Price against the real channel; ask about mixing | | Constant surplus of hundreds of input tokens | Injected system prompt (usually a pool) | Price at real cost; not for stable production | | Model in the error ≠ model in the response | Model swapped | Reject | | Invalid parameters return 200 | Fields swallowed | Ask the vendor to fix; check valid fields too | | Seedance signing cert ≠ official | Not an original, or re-encoded | Reject by my standard (strict; relax if copies are fine for you) | | Usage endpoint ≠ logs | Billing unreliable | Trust logs only; test every tier | Stability and capacity are not something a pre-purchase check can answer: a few dozen requests will not show them. After you integrate, production monitoring and a 30-day success rate are far more reliable. --- ## Let an Agent run the whole thing None of these steps is hard. What is hard is doing every one of them for every relay without skipping any, keeping the raw requests and responses, and turning it all into a report someone else can read. I took the audit procedure I use at work, stripped everything internal, and open-sourced it as an Agent skill: **[llm-relay-audit](https://github.com/YouToco/llm-relay-audit)**. Give the relay's base URL and key to an Agent such as Claude Code, Codex or Cursor, and it follows a fixed procedure: free probes first, **a price quote and your OK before any request that might cost money**, and two reports at the end. ```bash # one-line install npx skills add YouToco/llm-relay-audit ``` The external evidence report covers only "this relay vs the official API"; its first page is a one-line verdict and four verdict cards, and it can go straight to the vendor to fix: ![First page of the external evidence report: one-line verdict and four cards for backend, model, API and cost](https://zhuoqidev.com/_astro/report-en.B9tL790d_6pVCw.webp "First page of the external evidence report (the repository's fictional example, rendered by the generator).") The internal decision card is one page for whoever decides whether to integrate, including how the relay compares with your existing lines: ![Internal decision card: verdict "backup", cost and 30-day success rate against an existing line for the same model](https://zhuoqidev.com/_astro/card-en.2PK4hj34_Z1DX50F.webp "Internal decision card (fictional example).") Beyond the procedure, it takes care of things people get wrong: keys are masked automatically wherever requests and responses are written to disk; a paid request that may already have reached the server is **never resent automatically** (no buying the same video three times); the generator refuses to write the external report if it contains an internal line name or anything that looks like a key; and the video C2PA check compares the signing certificate's fingerprint instead of only checking that the signature is valid. Design and engineering details are on the [project page](https://zhuoqidev.com/en/projects/llm-relay-audit/). --- ## Limits and sources - **Measured**: 508 (the injecting one of the two subscription pools), about 4,380, the $0.29 job read as $2, the cost per audit, OpenAI's counting endpoint not billing (it works on a zero-balance key), and the swallowed-parameter and swapped-model cases all come from my integration tests of 30-odd relays in September and October 2026; the relays are not named. - **Observed patterns, not vendor promises**: id prefixes (especially `msg_011C`), the exact token count a subscription pool injects, and error wording can all change, at the vendor or at the relay. - **Inference**: that reselling consumer subscriptions usually breaks the upstream terms is inferred from vendors' usual terms; read the terms of the service you actually use. - **Official docs** (checked 2026-10-23): [Anthropic token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting) · [OpenAI input token counts](https://developers.openai.com/api/reference/resources/responses/subresources/input_tokens) · [C2PA specification](https://c2pa.org/specifications/specifications/2.2/specs/C2PA_Specification.html) - **Tools**: [new-api](https://github.com/QuantumNous/new-api) · [sub2api](https://github.com/Wei-Shaw/sub2api) · [llm-relay-audit](https://github.com/YouToco/llm-relay-audit) --- **The smallest three steps you can take today: check the id prefix, reconcile the tokens, then make the error name the model.** **A group name is not evidence; tokens are. Reconcile first, then pay.**
Appendix: the three checks I use most (1 is free, 2 costs a fraction of a cent, 3 through a relay: send once and check its logs) ```bash # 1. Official baseline: count input tokens with an official key (free; Sonnet 5.5 counts this sentence as 18, 2026-10-14) curl https://api.anthropic.com/v1/messages/count_tokens \ -H "x-api-key: $OFFICIAL_KEY" -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d '{"model":"claude-sonnet-5-5","messages":[{"role":"user","content":"Reply with the single word OK."}]}' ``` ```bash # 2. What the relay bills: send the same sentence, keep the raw response, read usage and the id prefix curl $RELAY_BASE/v1/messages \ -H "x-api-key: $RELAY_KEY" -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d '{"model":"claude-sonnet-5-5","max_tokens":16,"messages":[{"role":"user","content":"Reply with the single word OK."}]}' \ | tee relay-ok.json ``` ```bash # 3. Make the upstream name the model: a 400 is free on the official API; through a relay send once, then check its logs for a charge curl $RELAY_BASE/v1/messages \ -H "x-api-key: $RELAY_KEY" -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d '{"model":"claude-sonnet-5-5","max_tokens":9999999,"messages":[{"role":"user","content":"hi"}]}' \ | tee relay-max-tokens.json ``` Swap in the model you want to test. Some relays accept only `Authorization: Bearer` rather than `x-api-key`; just switch the header.
--- title: "Is a Mac Actually Good for Running Large Language Models? The Numbers Neither Apple nor NVIDIA Puts on the Product Page" description: "Apple says the Mac mini offers \"fast, private, on-device inference\" and up to 13.5x faster LLM prompt processing; NVIDIA says the RTX 5090 Laptop GPU has 1,824 AI TOPS. What they compare against, how the test machines were configured, and the three numbers that decide how fast a model actually feels — memory bandwidth, the compute for reading prompts, and how much memory the GPU can really use — are not where you look first. Using the llama.cpp community's shared benchmarks across dozens of machines, a check on two machines of my own, and a re-run of Apple's own footnoted test on the 16 GB base model, and the coding, compiling and testing work where the Mac mini measurably wins, this post answers, case by case, what a Mac mini, a Mac Studio and each class of NVIDIA card are actually good for." url: "https://zhuoqidev.com/en/posts/mac-for-local-llms/" language: "en" author: "Liu ZhuoQi" date: "2026-10-18" updated: "2026-10-18" tags: ["Local LLM","Hardware","Apple Silicon","NVIDIA","llama.cpp","Memory Bandwidth","Benchmarks"] translation: "https://zhuoqidev.com/posts/mac-for-local-llms/index.md" --- # Is a Mac Actually Good for Running Large Language Models? The Numbers Neither Apple nor NVIDIA Puts on the Product Page > **TL;DR** Being able to run a model and running it well are separated by a few numbers that product pages leave out. Output speed follows memory bandwidth, reading long input follows compute, and the size you can run follows the memory the GPU can actually use. Meanwhile one vendor compares its new Mac against an M1 from five years ago, and the other gives a desktop card and a laptop chip the same name, 5090. The verdict has two halves: as a machine for writing code, compiling, running tests and hosting an Agent whose model runs in the cloud, a Mac mini measured more than twice as fast as a gaming laptop of its time and fits well; as your main local-LLM machine it does not; a Mac Studio is the easiest way to fit a big model, but slow at reading long input; NVIDIA cards are fastest as long as the model fits in video memory, and fall off a cliff when it does not. > **Running is not running well.** > > A 16 GB Mac mini running a 14B model the way Apple's own footnote tests it: > - 8K prompt · first token: 15 s > - 16K prompt · first token: 35 s > - 32K prompt: Crashed > > *Qwen3-14B 4-bit · mlx-lm 0.32 · measured 2026-10-18* *This is for anyone about to buy a machine to run large models locally and deciding from product pages. Most of the data comes from the llama.cpp community's shared benchmarks; I checked it by running the same command on the Mac mini (M6, 16 GB) and the RTX 3060 laptop I own, and re-ran Apple's own footnoted test on the 16 GB base model. How I ran everything and the full numbers are in the [appendix](#appendix-how-i-ran-it-and-the-full-numbers).* --- ## What the product pages say, and what the small print says The two vendors' claims next to their footnotes: | On the product page | In the small print | Not where you look first | |---|---|---| | Mac mini: "fast, private, on-device inference"; M6 "up to 13.5x faster with LLM prompt processing" | Compared against **the 2020 M1**; the M6 test machine had **32 GB**; the test is time to first token for a 14B model with an 8K-token prompt | No comparison with any GPU; the 16 GB entry model has 153 GB/s of bandwidth, while the page says "up to 170" | | Mac Studio: run "frontier-class models on device — no cloud tokens needed" | M5 Ultra goes up to 512 GB, top configuration only | It reads long prompts at a third the speed of an NVIDIA card in the same output class | | RTX 5090 Laptop GPU: 1,824 AI TOPS, "run the latest AI models locally" | 24 GB of memory, 896 GB/s | The desktop RTX 5090 has 32 GB and 1,792 GB/s: the same name, half the bandwidth | | Desktop RTX 5090: 3,352 AI TOPS | — | The product page does not list bandwidth; a model that does not fit in 32 GB will not run, whatever the TOPS | Neither vendor says anything false, but both put compute front and centre. The trouble is that **the thing you wait for most when running a large model — tokens coming out one by one — barely depends on compute.** --- ## Running a model really compares three things ### 1. How fast it writes: memory bandwidth A model writes one token at a time. For each token the GPU reads **all of the weights** out of memory and multiplies them with a tiny bit of fresh state. The multiplying is quick; the reading is slow. So single-user output speed has a rough but useful ceiling: > **Output speed ≈ memory bandwidth ÷ bytes the model occupies** llama.cpp has two community benchmark threads ([Apple Silicon](https://github.com/ggml-org/llama.cpp/discussions/4167), [NVIDIA](https://github.com/ggml-org/llama.cpp/discussions/15013)) where people post results from the same model and the same command. I first ran that command on my two machines to check: 75.2 on my RTX 3060 laptop against 76.9 for a community 3060 desktop, 35.0 on my Mac mini M6 against 35.7 for a community Mac mini M6. A 2–4 % gap, so the data is trustworthy. Sorted by bandwidth: | Memory bandwidth (GB/s) | NVIDIA | Apple Silicon | | --- | --- | --- | | 153 | | 35 | | 170 | | 36 | | 288 | 64 | | | 307 | | 66 | | 336 | 75 | | | 360 | 77 | | | 614 | | 120 | | 896 | 158 | | | 936 | 162 | | | 1,008 | 189 | | | 1,228 | | 179 | | 1,792 | 300 | | - Mac mini M6 - M5 Max - M5 Ultra - Laptop 5090 - 4090 - Desktop 5090 *Llama 2 7B at 4 bits, llama.cpp single-user output speed. From the llama.cpp community threads; the Mac mini M6 (16 GB) and 3060 laptop points are my own check runs on 2026-10-17. Bandwidth is the vendor's rated figure (Apple only states "1.2 TB/s" for the M5 Ultra; I use 1,228, twice the M5 Max). Software versions differ between entries, so read the order of magnitude.* It is nearly a straight line. Every machine uses only 56–88 % of its rated bandwidth, but the ranking is set entirely by bandwidth: the M5 Ultra has a fifth more bandwidth than the 4090 and writes at about the same speed; the desktop 5090 has twice the laptop 5090's bandwidth and writes nearly twice as fast. **Both are called 5090; they sit far apart on the line.** ### 2. How fast it reads long input: compute When the model reads the text you give it (prefill), it processes thousands of tokens at once, and only then does compute become the bottleneck. The gap opens up at once: | | | | --- | --- | | Desktop RTX 5090（32 GB） | 14,970 | | RTX 4090（24 GB） | 14,771 | | Laptop RTX 5090（24 GB） | 7,642 | | M5 Ultra（up to 512 GB） | 4,945 | | M5 Max（up to 128 GB） | 3,220 | | RTX 3060 laptop（6 GB） | 2,311 | | Mac mini M6（16 GB） | 1,016 | *Prompt-reading speed (tokens/s), Llama 2 7B at 4 bits, 512-token prompt. Community threads; the 3060 laptop and Mac mini are my own check runs.* The 4090 and the M5 Ultra write at about the same speed, yet read prompts three times apart. You will not notice in a short chat; but ask it to read a code repository, a contract dozens of pages long, or run it as an Agent calling tools turn after turn — rereading an ever-longer context every turn — and three times is three times the wait. ### 3. How big a model: the memory the GPU can actually use Bandwidth decides how fast; **capacity decides whether at all**. A model that does not fit runs at zero. Both vendors print the total; neither prints how much you can actually use: - **A Mac's GPU cannot use all of the unified memory.** On my 16 GB Mac mini, llama.cpp reads a GPU limit of 12.7 GB from the system at start-up, about 79 %, and it shares that with the OS, the browser and your editor. - **An NVIDIA card's video memory is a hard limit.** Whatever does not fit goes back to CPU memory, and speed drops by an order of magnitude (measured below). The largest consumer card, the desktop 5090, has 32 GB. > [!NOTE] > A rough rule: a 4-bit model takes about 0.6 GB per billion parameters, plus 20–30 % for context (the KV cache) and the runtime. By that rule, about 12 GB usable by the GPU is only enough for the 14B class, 24–32 GB keeps a 30B-class model, 128 GB keeps 100–150B, and only 512 GB reaches 600B-class MoE models. It is an estimate; the real figure depends on the model and the quantization format. --- ## Apple's footnoted test, re-run on the base model Apple's "13.5x faster" uses a 14B model at 4 bits with an 8K-token prompt and measures time to first token, on a 32 GB machine. I ran the same conditions on a 16 GB Mac mini (MLX, mlx-lm) and added 16K and 32K prompts — Agent work routinely reaches tens of thousands of tokens of context. I also tried the same model on the 3060 laptop with 6 GB of video memory. | | | | --- | --- | | 2K prompt（Mac mini） | 3.4 s | | 8K prompt（Mac mini (Apple's footnote conditions)） | 15.4 s | | 16K prompt（Mac mini） | 34.6 s | | 8K prompt（3060 laptop, half the layers on the GPU） | 21.0 s | *Qwen3-14B at 4 bits, wait from sending the prompt to the first token (seconds). Mac mini on mlx-lm 0.32, 3060 laptop on llama.cpp b11696. Measured 2026-10-18, average of two runs each. A 32K prompt does not run on the Mac mini, so it is not drawn.* - **With an 8K prompt the first token takes 15 seconds**, then 15 tokens a second. Bearable for everyday questions — and these are the conditions Apple chose. - **16K takes 35 seconds**, and output drops to 13 tokens a second; GPU memory reaches 11.5 GB, just short of the 12.7 GB the system allows. - **32K crashes**: Metal reports the GPU is out of memory and the program exits. A 14B model with 32K of context does not run on the 16 GB base model at all — and an Agent reading a code repository reaches that size routinely. - **The 3060 laptop cannot even hold the model**: the 14B needs 8.2 GB, the 6 GB card takes half, the rest runs on the CPU, and output falls to 4.8 tokens a second; all on the CPU it is 2.5. The same card does 75 on a 7B that fits. **NVIDIA's wall is not slowness; past the line it collapses.** > [!TIP] > One more thing no product page tells you: **the MLX version matters**. My first install of mlx-lm picked the system Python 3.9, which only gets an old MLX (0.29); the same 8K prompt took 55 seconds to the first token — 3.6 times slower, because the old version cannot use the Neural Accelerators in the M6 GPU. Reinstalling with Python 3.13 and MLX 0.32 gave the numbers above. The same goes for LM Studio or Ollama: check that the MLX it ships is recent. --- ## The Mac mini is bad at running models, but good at something else Having covered the weak spots, fairness needs the strengths too. The same 16 GB Mac mini and the RTX 3060 laptop (its CPU is an i7-12700H, a 14-core gaming laptop from 2022) ran the same scripts on the work I actually do every day. The real project is a TypeScript gateway repository with 5,139 tests: | | | | --- | --- | | Full test suite（65 s vs 136 s） | 2.1× | | TypeScript type check（3.3 s vs 7.9 s） | 2.4× | | Front-end build（31 s vs 73 s） | 2.4× | | Single-core Python mix（0.05 s vs 0.16 s） | 3.1× | | CPU memory copy bandwidth（114 vs 16 GB/s） | 7.2× | | numpy single-precision matmul（1,988 vs 434 GFLOPS） | 4.6× | | Spawning subprocesses（717 vs 2,119 per second） | 3.0× | | Create and delete 20,000 small files（1.63 s vs 0.74 s） | 2.2× | | git grep a whole repository（428 vs 175 ms） | 2.4× | | Train a small CNN（fastest framework on each） | 2.7× | | 4B model, one user, output（56 vs 96 tokens/s） | 1.7× | - Even: 1.0× *How many times faster (faster side ÷ slower side). Measured 2026-10-15 to 10-17, several runs each, stable values. Full numbers in the appendix.* - **Writing code, running tests and compiling: the Mac mini is more than twice as fast.** About three times faster on one core, seven times faster at moving memory on the CPU; SQLite one-by-one commits are 16 times faster (0.15 s vs 2.4 s). - **Where it is slower is small actions heavy on system calls**: spawning subprocesses, creating and deleting small files and `git grep` are two to three times slower. But these are milliseconds — one subprocess takes 1.4 ms against 0.5 ms, which nobody notices. After 120 seconds at full load it loses about 5 % of its speed; the laptop does not. - **So the line on Apple's page about using "agents like Claude Code or Codex to be more productive" holds**: those Agents run their model in the cloud, and what the local machine does is compile, run tests, read and write files and call git — all things the Mac mini does fast and steadily. Office work and writing are no contest. **One more strength the product page does not even push, and that holds up hard when measured: low power, and quiet.** I read whole-machine power from macOS's own power telemetry (no root needed, sampled every second, averaged from the accumulators), and the 3060 laptop's GPU power from `nvidia-smi`: | | | | --- | --- | | Mac mini idle（Apple's published figure） | 4 W | | Mac mini, all 12 CPU cores busy（measured, 60 s average） | 21 W | | Mac mini running the 14B model（measured, 14.6 tokens/s output） | 30 W | | Mac mini maximum（Apple's published figure） | 70 W | | 3060 GPU idle（measured 30–37 W） | 33 W | | 3060 GPU running the 14B model（measured, half on the GPU, 4.5 tokens/s output） | 86 W | *Power (watts). Mac mini from the system power telemetry (PowerTelemetryData: board power plus power-supply loss, not a wall meter); idle and maximum are Apple's figures measured at the wall. For the 3060 only the GPU itself can be read, without CPU or screen. Measured 2026-10-18.* The Mac mini runs the whole 14B model from memory at under 30 W for the entire machine; the 3060 laptop's GPU alone idles at just over 30 W and draws 86 W on the same model, while writing only a third as fast because the model does not fit. Even fully loaded, I can barely hear the Mac mini's fan — my impression, not a decibel measurement. For a computer left on around the clock as an Agent's workstation, this matters. The trouble only starts when you move the model onto the same machine. ## What Apple does not put up front 1. **Bandwidth follows the configuration, and the page says "up to".** The Mac mini M6 page says "up to 170GB/s"; the 16 GB entry model has 153. The Mac Studio page lists 614 GB/s for the M5 Max — that is the 40-core-GPU version; the 32-core entry version has 460. On the line above, a tenth less bandwidth is a tenth slower output. 2. **The GPU cannot use all of the unified memory.** 16 GB gives the GPU about 12.7 GB, before subtracting whatever else you have open. 3. **Reading long input is the weak spot.** Every "X times faster" Apple quotes is against older Macs; against an NVIDIA card at the same price, prompt reading is two to three times slower. On the 16 GB base model a 14B with an 8K prompt waits 15 seconds and 32K does not run — and Agents, code repositories and long documents are exactly how "local AI" gets used most. 4. **The software version decides whether you get the new hardware.** An old MLX cannot use the M6's Neural Accelerators and reads prompts 3.6 times slower. 5. **Its real strength is capacity.** A Mac Studio holds up to 128 GB / 512 GB of unified memory; one machine fits a model that would take several NVIDIA cards. On this point the marketing does not exaggerate. ## What NVIDIA does not put up front 1. **Video memory is a hard limit.** However high the AI TOPS on the page, a model that does not fit in video memory falls off a cliff. 2. **The laptop and the desktop share a name, not a spec.** Laptop 5090: 24 GB, 896 GB/s; desktop 5090: 32 GB, 1,792 GB/s. Same with the 3060: 6 GB on laptops, 12 GB on the desktop card. On a laptop, look at the memory and bandwidth, not the model number. 3. **AI TOPS is about reading the prompt.** Output speed follows bandwidth, and the desktop 5090's product page happens not to list it. 4. **Power and noise.** One desktop 5090 draws 575 W and wants a 1,000 W power supply; a Mac Studio under full load stays far below that. --- ## So is a Mac good for running large models? | What you want to do | Is a Mac a good fit? | A better fit | |---|---|---| | Writing code with Agents like Claude Code or Codex (model in the cloud), compiling, running tests, office work | **Very good**: compiling and testing measured more than twice as fast as an i7-12700H gaming laptop | — | | Apple Intelligence, a 4B–8B model now and then | **Yes**; a Mac mini is enough | — | | A 14B-class model as an everyday assistant, or an Agent reading a code repository | **Not really**: on the 16 GB base model an 8K prompt waits 15 seconds and 32K does not run; 32 GB fits it but still reads long input slowly | A desktop NVIDIA card with 16 GB or more | | A 30B-class model for yourself, fast | **So-so**: it takes a Mac Studio to fit; output on par with a 4090, slow on long input | A 24–32 GB desktop NVIDIA card (4090, 5090) | | 100B and up, fitting matters more than speed | **Best fit**: one Mac Studio with 128 GB / 512 GB holds it; NVIDIA needs several cards | — | | Many users at once, fast reading, long context | **No** | Several workstation cards, or just buy API tokens | China prices for reference (apple.com.cn, 2026-10-18): Mac mini M6 16 GB from ¥6,999; Mac Studio M5 Max from ¥19,999; M5 Ultra from ¥46,999. In one line: **a Mac mini is a very good development and Agent machine, not a good local-LLM machine.** Its product page puts the two side by side — "create your own AI agents" and running large models on device — and the first holds up when measured, while the second, on the 16 GB base model, is closer to "it runs, the experience is so-so". To fit a truly large model locally, look at the large memory of the Mac Studio tier; to run it fast, it is still NVIDIA. --- ## Further up: power and borders If you keep going — an 8-GPU data-center server, the newest data-center GPU — you hit two ceilings that have nothing to do with performance: - **Power**: one NVIDIA DGX Rubin NVL8 draws about 24 kW and needs direct-to-chip liquid cooling; an ordinary socket in a Chinese home is 10 A (about 2.2 kW), and even the 16 A socket for an air conditioner gives about 3.5 kW. - **Borders**: the Blackwell and Rubin data-center GPUs are barred from sale to China, with no sign of a licence as of October 2026; the full RTX PRO 6000 (96 GB) is also above the control threshold, and China gets the export-edition RTX PRO 6000D (according to teardowns, 84 GB and about 17 % fewer cores). Beyond that, you rent GPUs or buy tokens. --- ## Limitations - The community numbers only cover a 7B model, one user, a 512-token prompt and llama.cpp; entries use different software versions, and newer versions alone are 10–30 % faster, so read the order of magnitude. - The everyday-development comparison is against a 2022 i7-12700H gaming laptop, not a new PC at the same price; the laptop had more programs open during the run (a quick re-run gave nearly the same single- and multi-core results). Dependency install times depend on the network and caches and are left out. - Power: the Mac mini numbers come from the system's own power telemetry, not a wall meter; on the 3060 laptop only the GPU can be read, and CPU power needs root, so it was not measured. - My re-run covers one 16 GB Mac mini and, on the Mac, one engine (mlx-lm), while Apple's test used LM Studio; I did not measure the 32 GB version or a Mac Studio. The 14B numbers use Qwen3-14B, which may not be the model in Apple's footnote. - Prices and export controls are as checked on 2026-10-18 and change quickly; the 6000D specs come from teardowns and benchmarks, not from NVIDIA. --- ## Sources - Product pages and footnotes: Apple [Mac mini](https://www.apple.com/mac-mini/) ([Chinese edition](https://www.apple.com.cn/mac-mini/); footnote 6 gives the conditions for "13.5x"), [Mac Studio](https://www.apple.com/mac-studio/); NVIDIA [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), [RTX 50 Series laptops](https://www.nvidia.com/en-us/geforce/laptops/50-series/). All read on 2026-10-18. - Benchmarks: llama.cpp community threads [Apple Silicon #4167](https://github.com/ggml-org/llama.cpp/discussions/4167) and [NVIDIA #15013](https://github.com/ggml-org/llama.cpp/discussions/15013) (raw posts fetched 2026-10-17). - Specs: [NVIDIA GPU comparison](https://www.nvidia.com/en-us/geforce/graphics-cards/compare/), [RTX 3060 Laptop GPU (Notebookcheck)](https://www.notebookcheck.net/NVIDIA-GeForce-RTX-3060-Mobile-GPU-Benchmarks-and-Specs.497453.0.html); Apple [Mac mini specs](https://www.apple.com/mac-mini/specs/), [Mac Studio specs](https://www.apple.com/mac-studio/specs/); China prices from [apple.com.cn](https://www.apple.com.cn/shop/buy-mac/mac-studio). - Power: Apple [Mac mini power consumption and thermal output](https://support.apple.com/103253) (M6: 4 W idle, 70 W maximum, measured at the wall); [DGX Rubin NVL8](https://www.nvidia.com/en-us/data-center/dgx-rubin-nvl8/); Chinese standard [GB 1002-2024](https://www.chinesestandard.net/PDF/English.aspx/GB1002-2024). - Export controls: [Tom's Hardware on the 6000D](https://www.tomshardware.com/pc-components/gpus/nvidia-rtx-pro-6000d-b40-blackwell-gpus-reportedly-set-to-supersede-banned-h20-accelerators-in-china), [PCGH teardown](https://www.pcgameshardware.de/Grafikkarten-Grafikkarte-97980/News/Nvidia-RTX-6000D-mit-84-GiByte-GDDR7-1487562/). --- ## Appendix: how I ran it and the full numbers
How the check and the re-run were done ```bash # The same command as the community benchmark (llama.cpp b11696 official prebuilt binaries) llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -p 512 -n 128 # Apple's footnote 6 conditions: 14B, 4 bits, 8K prompt, time to first token (plus 16K and 32K) mlx_lm.benchmark --model mlx-community/Qwen3-14B-4bit -p 8192 -g 128 -n 2 ```
Full numbers for my two check machines | Item | RTX 3060 laptop (6 GB) | Mac mini M6 (16 GB) | |---|---|---| | Llama 2 7B 4-bit output / prompt reading (tokens/s) | 75.2 / 2,311 | 35.0 / 1,016 | | Full test suite (5,139 tests, 4 shards) | 136 s | 65 s | | TypeScript type check / front-end build | 7.9 s / 72.5 s | 3.3 s / 30.8 s | | Single-core Python mix / all cores | 0.157 s / 2.67 s | 0.051 s / 0.44 s | | CPU memory copy / numpy single-precision matmul | 15.9 GB/s / 434 GFLOPS | 114 GB/s / 1,988 GFLOPS | | SQLite 1,000 one-by-one commits | 2.4 s | 0.15 s | | Spawn subprocesses / 20,000 small files / git grep | 2,119 per s / 0.74 s / 175 ms | 717 per s / 1.63 s / 428 ms | | Last ÷ first stretch of 120 s at full load | 1.005 | 0.954 | | Power: idle / all CPU cores / running the 14B model | GPU 30–37 W / not measured / GPU 86 W (peak 90 W) | 4 W (Apple) / 21 W / 30 W (whole-machine telemetry) | | Qwen3-4B 4-bit single-user output (fastest engine) | 96 | 56 (MLX) | | Qwen3-4B 4-bit, 8 at once (engine benchmark) | 296 | 256 (llama.cpp) | | Memory the GPU can use | 6 GB of video memory | about 12.7 GB (system limit) | | Qwen3-14B 4-bit, 8K prompt: first token / output | 21.0 s / 4.8 tokens/s (only half fits, 20 of 40 layers) | 15.4 s / 14.8 tokens/s (mlx-lm 0.32) | | Qwen3-14B 4-bit, first token at 16K / 32K | not measured | 34.6 s / out of GPU memory, crashed |
--- title: "What Does It Cost to Run an App Online Each Month? Servers, Serverless or Edge Functions: First Learn to Split the Bill in Three" description: "For anyone about to put their first app online without knowing whether to buy a server or go serverless. It first explains the few things on a cloud provider's shelf, then shows how to split your own app into three bills (does its time go to waiting or computing, how many bytes does each request move, does it keep books entry by entry), places four common apps (a personal website, a mini-program or app backend, an AI app, an image or video tool) with the free tiers and new-customer prices checked; then it builds an LLM gateway from scratch as the worked example, queries prices as of 2026-10-09 with the aliyun CLI, tccli, the clouds' public pricing APIs and official pricing pages, and works out the bill line by line for seven providers at 10 million, 100 million and 1 billion requests a month." url: "https://zhuoqidev.com/en/posts/model-gateway-cloudflare-vs-china-cloud/" language: "en" author: "Liu ZhuoQi" date: "2026-10-09" updated: "2026-10-09" tags: ["Cloud Hosting","Serverless","Cloud Servers","Cloudflare Workers","Alibaba Cloud","Tencent Cloud","Cost Modeling","LLM Gateway","Architecture"] translation: "https://zhuoqidev.com/posts/model-gateway-cloudflare-vs-china-cloud/index.md" --- # What Does It Cost to Run an App Online Each Month? Servers, Serverless or Edge Functions: First Learn to Split the Bill in Three > **TL;DR** Going live for the first time? Do not rush to buy a server. Ask three things first: does your app spend its time waiting or computing, how many bytes does each request move, and does it keep books entry by entry. To start, most apps run on free tiers or a hundred or two yuan a year; the real money comes at volume, and which bill it lands on depends on how you answer those three questions. *This is for anyone putting their first app online. The first half covers how to choose and how to estimate the cost; the second half builds an LLM gateway from scratch and works out the bills of seven clouds line by line. Prices were checked on 2026-10-09; how they were queried and the full tables are in the [appendix](#appendix-how-the-prices-were-queried-and-the-full-numbers).[^scope]* --- ## The conclusion first (for the impatient) 1. **Start on free tiers.** A personal site on Cloudflare Pages or GitHub Pages costs nothing; Cloudflare Workers is free up to 100,000 requests a day; in China, Alibaba Cloud runs ¥99-a-year offers for both a server and a database. Most apps start on a few yuan to a few dozen yuan a month. 2. **Do not pay for waiting.** If your app spends most of its time waiting on others (calling an LLM, calling third-party APIs), avoid serverless billed by duration: it charges for every second you wait. 3. **At volume, watch the bytes you carry and the entries you write.** By then CPU is the cheapest line on the bill. 4. **If your users are in mainland China, use a Chinese cloud and get an ICP filing first.** Every sentence above is explained below. --- ## Who this is for - You have an app you want to put online: a personal website, a mini-program or app backend, an AI app that calls an LLM, or a small tool that processes images or video. You have never bought a server and cannot quite say how serverless differs from one. - After the first half you will be able to read what cloud providers sell, split your app into three bills, pick a way to start, estimate the monthly cost, and know when to switch. - The second half is for those who want the details: an LLM gateway as a worked example, assuming you know how to call an LLM API and that streamed output arrives piece by piece over SSE. You can skip it without losing the conclusions of the first half. - The article **does not** cover model inference itself (GPUs, inference engines), and touches compliance only where it shapes the choice. ### Glossary | Term | In one sentence | |---|---| | Request | One "question" a browser or app sends your service when a user opens a page or taps a button | | Cloud server | A computer a cloud provider rents you, always on (ECS at Alibaba Cloud, CVM at Tencent Cloud, EC2 at AWS) | | Serverless | You upload only code; the platform starts it when a request arrives and charges no machine while nothing runs (Function Compute, SCF, Lambda, Cloud Run) | | Edge functions | Code that runs in many data centers worldwide, close to users; Cloudflare Workers is the best-known | | Wall-clock time | The real time from when a request arrives until it ends, including time spent waiting on others | | CPU time | The time the program is actually computing; waiting on the network or a database does not count | | Egress | Data sent out of the cloud: pages, images, API replies; most clouds bill it per GB | | Bookkeeping / state | Data you must write down and read again later, such as balances, call counts, login sessions | | ICP filing | The registration with China's Ministry of Industry and Information Technology that a site needs before serving from mainland servers or nodes | --- ## What is on a cloud provider's shelf The first visit to a cloud provider's site shows dozens of products. There are really only three places to put your code, plus a class of parts someone runs for you: | On the shelf | What it is | Like | How it bills | |---|---|---|---| | Cloud server | A computer that is always on; you manage the system, software and security updates | Renting a whole flat | Monthly or yearly, empty or not | | Serverless billed by duration | You upload code; the platform starts it per request | A hotel, by the night | Every second from request to response, including time spent waiting on others | | Edge functions (Cloudflare Workers) | Code in data centers worldwide, close to users | A worker paid only for the minutes of actual work | By the milliseconds your code computes; waiting is free | | Managed services | Databases, file storage, CDNs that someone else runs | A furnished flat | By size or by use | Five characters keep turning up in this article; meet them first: ```comic cols=2 > An ordinary request [User] Ask the LLM for me: do I need an umbrella tomorrow? [Your service] Got it, passing it to the LLM now. --- > Twenty seconds later [LLM] Tomorrow... light... rain... [Your service] I was busy for 10 ms. The other 20 seconds I waited for it to finish, word by word. --- [Cloud!] Which raises the question: what do I bill you for? [You] That is exactly what this article works out. ``` "Your service" in the comic is the program you put in the cloud. It works for about 10 ms, yet each request waits 20 seconds for the LLM to finish talking. A cloud provider has three ways to bill it, and the bills can be several times apart: ```comic > Billed by CPU time (Cloudflare Workers) [Cloud] I only charge for the 10 ms your service actually works. Waiting is free. --- > Billed by duration (Lambda, Function Compute) [Cloud] From the moment the request arrives until it ends: 20 seconds, every one of them. [Your service] But I spend most of it staring at the wall! --- > Billed per machine (cloud servers) [Cloud] You pay for every machine you rent, busy or not. ``` Besides running code, two more bills are easy to miss: **egress** (data leaving the cloud; most clouds bill it per GB, Cloudflare does not) and **bookkeeping** (databases and caches; some are paid per instance, some per read and write). --- ## Learn to split the bill: your app in three parts ```comic cols=2 > Going live for the first time [You] I want to launch a mini-program. Should I buy a server or go serverless? [Cloud] Hold on to your wallet. What does your app do all day? --- [Cloud] Does it wait on others, compute, or carry things around? Does it keep books? [You!] So I have to take it apart first. ``` Splitting the bill takes three questions plus one constraint, and each has a way to find your own answer: **1. Does its time go to waiting or to computing?** Look at what your endpoints do: - They call an LLM or third-party APIs (payments, maps, SMS), or run slow database queries, and a call waits for seconds → **a waiting app**. - They process images and video, render PDFs, compress, or run a local model → **a computing app**. - They read or write the database and return within tens of milliseconds → **a light app**: little waiting, little computing. **2. How many bytes does each request move?** Open your app in a browser, press F12 for the developer tools, switch to the Network panel and click a request to see its size. For images and video, look at file sizes; for an AI app, look at how long the prompt and answer are (a Chinese character is about 3 bytes, an English word about 5–6). A few KB is light; hundreds of KB or more is heavy. **3. Does it keep books entry by entry?** Does each request read or write the database? Does it deduct a balance, count calls or rate-limit in real time? If so, bookkeeping is a fixed cost for you, and providers bill it in very different ways. **Plus one constraint: where are your users?** Mostly in mainland China means a Chinese cloud, and an ICP filing before the site serves anyone; overseas platforms such as Cloudflare have no mainland nodes and are slow from there. Mostly overseas, and you have many more options. Then estimate the volume: **requests per month ≈ daily active users × requests per user per day × 30**. For example, 1,000 daily users tapping 50 times a day is 1.5 million requests a month. With those numbers, the monthly bill is roughly four items added up: | | Cloud server | Serverless billed by duration | Edge functions (Cloudflare) | |---|---|---|---| | Floor (paid with nobody using it) | **High**: machines and databases are always on | Low | Low (Workers Paid is $5 a month) | | Waiting and computing | Almost nothing extra, but you buy enough machine for the peak | **Billed by wall-clock time**, waiting included | Only the milliseconds of computing | | Egress | Per GB (some plans include unlimited traffic) | Per GB | Free | | Bookkeeping | Your own database, per instance | An external database, per instance or per use | **Per read and write** | Whichever column is cheaper depends on which row your app spends the most in. --- ## Find your place: how four common apps start First, how the four apps start: | App | Its three bills | Users mostly in China | Users mostly overseas | |---|---|---|---| | Personal website, blog, portfolio | Almost no compute; luggage depends on images; no bookkeeping | Object storage plus a CDN, paid by traffic, ICP filing needed | Cloudflare Pages or GitHub Pages, free | | Mini-program or app backend | Light; light luggage; bookkeeping (user data) | One small server plus a database; WeChat mini-programs can also use Tencent CloudBase | Cloudflare Workers plus D1, starting on the free tier | | AI app that calls an LLM | Waiting; medium luggage; bookkeeping (usage, balances) | One small always-on server, not duration-billed Function Compute | Cloudflare Workers, starting on the free tier | | Image or video tool | Computing; heavy luggage; no bookkeeping | Function Compute when occasional, a server when it runs flat out; files in object storage | Lambda or Cloud Run when occasional, a server when sustained; files in R2 (free egress) | To start, use the free tiers and new-customer prices first:[^free] | Platform | Free tier or starting price | Watch out for | |---|---|---| | Cloudflare Pages | Free, unlimited static requests, 500 builds a month | | | GitHub Pages | Free, site up to 1 GB, soft limit of 100 GB of traffic a month | Not for online business, e-commerce or SaaS | | Vercel Hobby | Free, 100 GB of traffic a month | Non-commercial personal use only | | Cloudflare Workers | 100,000 requests a day free; Paid is $5 a month with 10 million requests | The free plan allows 10 ms of CPU per request | | Cloudflare D1 / KV / R2 | D1 5 million rows read and 100,000 written a day; KV 100,000 reads and 1,000 writes a day; R2 10 GB a month | R2 egress is free | | Alibaba Cloud ECS 99 | 2 vCPU, 2 GB, 3 Mbps with unlimited traffic, ¥99 a year | One per person at a time; same-price renewal, offer runs to 2029-03-31 | | Alibaba Cloud RDS 99 | MySQL basic edition, ¥99 a year | Can be held with ECS 99; offer runs to 2027-03-31; basic edition is a single node | | Alibaba Cloud simple app server | The site says "from ¥68 a year" | The page gives no spec or conditions; offer machines renew at the regular price | | Alibaba Cloud Function Compute | 150,000 CU a month for new users, for 3 months | No long-term free tier | | Tencent Cloud Lighthouse | 4 vCPU, 4 GB, 3 Mbps, ¥109 for the first year for new users | Once per verified identity, offer runs to 2026-12-30, renews at list price | | Tencent CloudBase (mini-programs) | Free trial with 3,000 resource points a month; Personal plan ¥19.9 a month (limited time) | Once the mini-program is published, the free environment expires after 15 days | | AWS | $100 credit for new accounts, up to $100 more for using listed services; Lambda's 1 million requests a month stay free | The free plan lasts at most 6 months, and the account closes when it ends or the credit runs out | | Google Cloud | $300 for new customers, within 90 days; Cloud Run 2 million requests a month free; one e2-micro always free | e2-micro only in three US regions | | Oracle Cloud | Always free: Arm capacity of about 2 cores and 12 GB, 2 small AMD machines, 10 TB of egress a month | Idle machines may be reclaimed after 7 days | When should you switch? Watch which of the three bills grows first: more requests and longer waits mean duration-billed serverless should give way to an always-on server or edge functions; more traffic means looking at egress prices (Cloudflare and Oracle are cheap); more bookkeeping means Cloudflare's per-operation billing keeps growing while your own database is paid per instance. The LLM gateway example in the second half works these three bills out from 10 million to 1 billion requests a month. [^free]: All checked on official pages on 2026-10-09: [Cloudflare Workers](https://developers.cloudflare.com/workers/platform/pricing/), [Pages](https://developers.cloudflare.com/pages/platform/limits/), [D1](https://developers.cloudflare.com/d1/platform/pricing/), [R2](https://developers.cloudflare.com/r2/pricing/), [GitHub Pages](https://docs.github.com/en/pages/getting-started-with-github-pages/github-pages-limits), [Vercel Hobby](https://vercel.com/docs/plans/hobby), [Alibaba Cloud ECS 99](https://www.aliyun.com/daily-act/ecs/99program), [RDS 99](https://www.aliyun.com/activity/database/bestoffers), [Alibaba Cloud simple app server](https://www.aliyun.com/product/swas), [Function Compute trial](https://help.aliyun.com/zh/functioncompute/fc/product-overview/trial-quota-1), [Tencent Lighthouse](https://cloud.tencent.com/act/pro/lighthouse), [Tencent CloudBase](https://tcb.cloud.tencent.com/pricing), [AWS Free Tier](https://aws.amazon.com/free/), [Lambda](https://aws.amazon.com/lambda/pricing/), [Google Cloud](https://docs.cloud.google.com/free/docs/free-cloud-features), [Oracle Always Free](https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm). Offers change at any time; for the simple app servers of Alibaba Cloud and Tencent Cloud, the purchase page has the final spec and renewal price; the RDS 99 spec comes from the rules on the bundle page. A few more common services, sorted the same way: | Service | Time goes to | Bytes per request | Bookkeeping | Where it fits | |---|---|---|---|---| | Personal website, blog | Almost no compute | Depends on images | None | Static hosting plus a CDN | | Mini-program or app backend | Light | Light | Yes (user data) | A small server plus a database, or a managed backend | | AI app that calls an LLM | Waiting | Medium | Yes (usage, balances) | Edge functions or a long-running process, not duration-billed serverless | | Real-time push and chat (SSE, WebSocket) | Waiting (long-lived connections) | Light | Presence, rooms | Workers + DO (no duration charges while a WebSocket hibernates), or a long-running process + Redis | | Slow-API proxies, BFFs, webhook relays | Waiting | Light | None or little | Edge functions are cheapest; a long-running process also pays off at volume | | Image, PDF and video processing | Computing | Heavy | None | Duration-billed serverless when occasional, machines when sustained | | File and media downloads | Almost no compute | Very heavy | None | Object storage plus a CDN; for overseas users, Cloudflare R2 has free egress | | High-frequency short requests (auth, counters, rate-limit services) | Very little | Light | Read on every request, within milliseconds | A long-running process with an in-memory cache (the reason Unkey left Workers, see below) | *Table: common services sorted by the three questions (this article's judgment, not official advice).* **For computing apps, what matters is the price of a CPU-hour:** | | | | --- | --- | | Tencent Cloud CVM（SA9, monthly） | ¥0.10 | | Oracle（E6.Flex, on demand） | ¥0.13 | | Alibaba Cloud ECS（c9i, monthly） | ¥0.14 | | Google Cloud（c4, on demand） | ¥0.29 | | AWS EC2（c8i, on demand） | ¥0.32 | | Azure（F4als_v7, on demand） | ¥0.41 | | Cloudflare Workers（per CPU millisecond） | ¥0.48 | | AWS Lambda（Arm, 1,769 MB ≈ 1 vCPU） | ¥0.57 | | AWS Lambda（x86） | ¥0.71 | *Price per CPU-hour (CNY, 2026-10-09). Machines are their 4-vCPU monthly price per vCPU-hour, i.e. the price when fully used; at 30% average use, multiply by about three. Lambda counts 1,769 MB of memory as about one vCPU.* Workers' CPU costs about 3.6 times an Alibaba Cloud monthly machine and about 1.5 times an AWS on-demand machine per CPU-hour.[^cpu-price] Sounds steep? But machines are bought for the peak and sit idle much of the time: against Alibaba Cloud, if your machines average below about 30% use, paying per millisecond comes out cheaper. Duration-billed Lambda is the most expensive per CPU-hour, yet for computing apps that is perfectly fair, since every second it is up it is computing. So **the same serverless product is a trap for waiting apps and a good deal for occasional computing**. Computing that runs flat out all day is still cheapest on machines. Mind the hard limits too: Workers allows at most 5 minutes of CPU and 128 MB of memory per request, too little for large image processing or long video transcoding.[^limits] --- ## A worked example: building an LLM gateway from scratch The first half was the method; this section takes one concrete app and works out its three bills line by line. An LLM gateway is a good pick because it runs up nearly all three: it waits a lot, computes little, carries medium to heavy luggage on every request and keeps books entry by entry. The example's conclusions first, who pays what at three volumes (switch volumes with the tabs on top): | | 10M requests / month | 100M / month | 1B / month | | --- | --- | --- | --- | | Cloudflare（plain design） | ¥140 | ¥2,756 | ¥32,574 | | Cloudflare（optimized design） | ¥67 | ¥2,115 | ¥25,139 | | Alibaba Cloud · Shenzhen | ¥1,625 | ¥5,854 | ¥45,696 | | Tencent Cloud · Guangzhou | ¥1,422 | ¥5,359 | ¥45,179 | | AWS · US East | ¥2,467 | ¥6,646 | ¥47,179 | | Google Cloud · US East | ¥3,056 | ¥7,387 | ¥43,798 | | Azure · US East | ¥4,794 | ¥9,125 | ¥49,848 | | Oracle · US East | ¥1,973 | ¥2,388 | ¥8,832 | *Monthly bill in CNY at list prices (2026-10-09). Self-hosted = load balancer + machines in two zones + Redis + MySQL + logs + egress. Hong Kong, Singapore and the managed and serverless options are in the appendix.* 1. **Small? Pick Cloudflare.** At 10 million requests a month it costs a little over ¥100, while the cheapest self-hosted setup costs about ¥1,400. A cloud provider bills you a floor: two machines, a high-availability database and a Redis instance cost the same with nobody using them. 2. **Large? It depends on how fat the requests are.** At 1 billion requests a month, self-hosting on Alibaba Cloud, Tencent Cloud or the big three Western clouds costs only 30% to a little over 100% more than Cloudflare. The smaller the requests, the better self-hosting looks; the bigger they are (coding agents, say), the better Cloudflare looks. 3. **Oracle does not play by the rules.** Its first 10 TB of egress each month is free, so at volume self-hosting in Oracle US East is about three times cheaper than Cloudflare. The price is running your own machines. > [!NOTE] > The example's bills rest on illustrative assumptions: each request lasts 20 seconds, the gateway computes for 10 ms, 50 KB moves per request, and the machine sizes were not load-tested. The most sensitive one is the last; [the chart in step 3](#step-3-carrying-the-luggage) shows which side your workload falls on. Right, let's build. Say you are launching an LLM gateway: customers come with an API key, you forward their requests to a model and bill them for what they use. We will add one part at a time, and after each part see where the bill grows. Stripped down, a gateway does six things: ```mermaid flowchart TD C(["Client"]) --> A["1 Look up the API key
who is it, which models"] A --> B["2 Take a concurrency slot, hold balance
return 429 if over the limit"] B --> R["3 Pick an upstream line"] R --> U["4 Forward the request body upstream"] U --> S["5 Relay the SSE stream as is
about 20 s, almost no CPU"] S --> F["6 Read usage, release the slot
settle actual usage, write one log"] classDef wait fill:#d0ebff,stroke:#1971c2 classDef work fill:#fff3bf,stroke:#f08c00 class S wait class A,B,R,U,F work ``` *Figure: the minimal causal chain of one LLM gateway request (illustrative, not any product's implementation). The yellow steps compute, but each takes only milliseconds; the blue step waits, for anything from ten-odd seconds to several minutes.* These six steps make an LLM gateway differ from an ordinary API gateway in three ways: | Trait | Value used here (illustrative) | Why it matters | |---|---|---| | Long wall-clock time | 20 s | Most of it is spent waiting for the upstream model to generate tokens; reasoning models take longer | | Short CPU time | 10 ms | Authentication, routing, relaying SSE chunk by chunk, finding the usage[^cpu-ms] | | Heavy egress | 50 KB per request | 30 KB is the customer's prompt forwarded upstream and 20 KB is the SSE stream sent back to the client | 10 ms out of 20 s is 0.05%. The gateway spends 99.95% of its time waiting. ### Step 1: take the request, then wait First the core part: take the request, forward it upstream, relay the answer back as is. What this step costs depends on where you stand: - **Workers**: bills only those 10 ms of CPU; the 20 seconds of waiting cost nothing.[^cf-wall] A 20-second stream costs the same as a 0.2-second one. - **Duration-billed Lambda, Function Compute, SCF**: bill all 20 seconds, every one of them, and often one instance can serve only one request at a time.[^one-per-instance] - **Your own machines**: waiting holds a connection, not a CPU; but you buy machines for peak concurrency and pay for them idle. How big is the gap? At 1 billion requests a month, the duration-billed options cost two to ten times Cloudflare: | | | | --- | --- | | Cloudflare Workers（billed by CPU time） | ¥32,574 | | Google Cloud Run（250 requests per instance） | ¥59,634 | | Alibaba Cloud AI Gateway（managed） | ¥60,518 | | Alibaba Cloud Function Compute（100 requests per instance） | ¥68,877 | | Tencent Cloud SCF（multi-request mode, estimated） | ¥75,903 | | Azure Container Apps（100 streams per replica） | ¥79,376 | | Google Cloud Run（80 requests per instance） | ¥107,676 | | OCI Functions（Oracle） | ¥264,894 | | AWS Lambda（128 MB Arm） | ¥278,460 | | Tencent Cloud SCF（default, one stream per instance） | ¥321,865 | - Alibaba Cloud Shenzhen, self-hosted: ¥45,696 *Monthly bill at 1B requests a month (CNY, including Redis, database, logs and egress). The dashed line is Alibaba Cloud Shenzhen, self-hosted. Wall-clock billing pays for the 20 seconds of waiting on the upstream; Cloudflare Workers bills CPU time only.* The cheaper duration-billed options in the chart all run many requests per instance. At small volumes they even undercut self-hosting slightly, since they skip the always-on machines and load balancer; as volume grows it flips. If you want to see this step in code, here is the Workers version as pseudocode, with comments marking which lines wait and which compute (illustrative, not any platform's real source): ```ts // Pseudocode (illustrative): one request through a Workers-based LLM gateway export default { async fetch(req, env, ctx) { const key = await env.KEYS.get(apiKeyOf(req), "json"); // 1 KV read: waits on the network, no CPU if (!key) return new Response("invalid key", { status: 401 }); const limiter = env.LIMITER.get(env.LIMITER.idFromName(key.id)); // one DO per API key const slot = await limiter.acquire(estimateCost(req)); // 2 take a slot + hold balance (1 row written) if (!slot.ok) return new Response("too many requests", { status: 429 }); const upstream = await fetch(pickLine(key), forward(req)); // 3, 4 wait for the upstream's first bytes, no CPU const { readable, writable } = new TransformStream(); ctx.waitUntil(pipeAndFindUsage(upstream.body, writable, async (usage) => { await limiter.release(slot.id, usage); // 6 release the slot + settle (1 row written) await env.USAGE.send({ key: key.id, usage }); // 6 usage message; a consumer sums per minute into D1 })); return new Response(readable, upstream); // 5 a 20-second stream; CPU only for copying chunks }, }; ``` On Alibaba Cloud or Tencent Cloud the same six steps become one long-running process (a Higress plugin or your own Go service) plus a Redis instance: key lookups hit an in-process cache and fall back to Redis on a miss; taking a slot and holding balance use Redis `INCR` / `DECR` plus a Lua script; usage is batched in the process and written to the database in bulk or sent to a message queue. **The architecture is the same; what differs is how each step is billed.** ### Step 2: keeping the books A gateway that bills by usage has to keep books: on the way in each request takes a concurrency slot and holds some balance, on the way out it settles what it actually used, and a usage record is left behind. ```comic cols=2 > Every request makes two entries in the books [User] I paid up front. Take what I use. [Your service] Then I hold a little on the way in, and settle what you actually used on the way out. --- > On Cloudflare [Cloud!] Every entry is billed. A billion requests is two billion entries. --- > On your own Redis [Cloud] Redis is paid per instance. One entry or a hundred million, same price. ``` Here the two sides bill in opposite ways. **Cloudflare bills every entry.** The largest item is DO row writes, about ¥13,000 a month at 1 billion requests. Slots and balances must be written to the DO's built-in SQLite rather than kept only in memory, because an idle DO may fall asleep and wake up with its in-memory counts gone, while a stream lasts 20 seconds.[^do] The optimized design, which sums usage inside the DO, flushes once a minute and drops Queues, cuts the bill by more than a fifth. Three more things to settle up front: - A DO is single-threaded and handles roughly a thousand requests a second. So use one DO per API key, shard large customers' keys, and never use one DO as a global rate limiter. - KV is eventually consistent: a change takes a minute or more to reach every location, so revoking a key or changing a balance cannot rely on it alone.[^kv] - A D1 database is capped at 10 GB, and the cap cannot be raised. A usage table summed per minute fills it in just over a month, so summarize periodically, split databases by month and move cold data to R2.[^d1] **A self-hosted Redis is paid per instance.** A 1 GB two-replica Redis costs ¥77 a month and is rated at 100,000 operations a second; at 1 billion requests a month the peak needs only about 4,600.[^redis] Key lookups, counters and balances all sit on it, and **ten times the requests do not move its price at all**. Split the bill at 1 billion requests a month into three parts and you can see at once whose bill the bookkeeping lands on: | | State (key lookups, counters, balances, usage store) | Egress | Everything else (machines, load balancer, logs) | Total | | --- | --- | --- | --- | --- | | Cloudflare（plain design） | ¥27,462 | ¥0 | ¥5,112 | ¥32,574 | | Cloudflare（optimized design） | ¥20,027 | ¥0 | ¥5,112 | ¥25,139 | | Alibaba Cloud · Shenzhen | ¥1,730 | ¥37,997 | ¥5,969 | ¥45,696 | | Tencent Cloud · Guangzhou | ¥2,008 | ¥40,000 | ¥3,171 | ¥45,179 | | AWS · US East | ¥4,626 | ¥28,827 | ¥13,726 | ¥47,179 | | Oracle · US East | ¥2,954 | ¥2,270 | ¥3,608 | ¥8,832 | *The monthly bill at 1B requests a month, in three parts (CNY). Cloudflare charges for state per operation; the self-hosted Redis and MySQL are paid per instance.* **More than 80% of Cloudflare's bill is bookkeeping; 60–90% of the Alibaba Cloud, Tencent Cloud and AWS bills is egress; Oracle's three parts are about the same size.** Egress is the next step. ### Step 3: carrying the luggage A gateway does not just pass messages; it carries luggage. Each request moves 50 KB on average: 30 KB is the customer's prompt, which the gateway delivers upstream, and 20 KB is the model's answer, which goes back to the client. **For a gateway, forwarding the prompt counts as egress too.** ```comic cols=2 > Your service does not just pass messages [User] My question is 30 KB, and the answer is another 20 KB. [Your service] I carry the question to the LLM, then carry the answer back to you. --- > On the Chinese clouds and the big three [Cloud] Every byte that leaves needs a ticket. [Your service] Even the questions I pass on to the LLM for users? [Cloud!] Even those. --- > On Cloudflare and Oracle [Cloud] No ticket needed. For Oracle, the first 10 TB a month. ``` Cloudflare needs no ticket at the door, the subrequests a Worker makes are free, and the terms put no cap on bandwidth.[^tos] The Chinese clouds charge ¥0.5–0.8 per GB, and the big three Western clouds about the same. Oracle's first 10 TB each month are free, and beyond that North America costs under a cent per GB. So the bytes per request all but decide who is cheaper: | Egress per request (KB) | Alibaba Cloud Shenzhen, self-hosted | AWS US East, self-hosted | Oracle US East, self-hosted | Cloudflare, plain design | Cloudflare, optimized design | | --- | --- | --- | --- | --- | --- | | 0 | | | | ¥32,574 | ¥25,139 | | 5 | ¥9,478 | ¥18,896 | ¥6,562 | | | | 10 | ¥13,723 | ¥22,186 | ¥6,562 | | | | 15 | ¥17,731 | ¥25,320 | ¥6,834 | | | | 20 | ¥21,726 | ¥28,443 | ¥7,119 | | | | 25 | ¥25,721 | ¥31,566 | ¥7,405 | | | | 30 | ¥29,716 | ¥34,688 | ¥7,690 | | | | 35 | ¥33,711 | ¥37,811 | ¥7,975 | | | | 40 | ¥37,706 | ¥40,933 | ¥8,261 | | | | 50 | ¥45,696 | ¥47,179 | ¥8,832 | | | | 60 | ¥53,247 | ¥52,548 | ¥9,402 | | | | 70 | ¥60,737 | ¥57,785 | ¥9,973 | | | | 80 | ¥68,227 | ¥63,023 | ¥10,544 | | | | 90 | ¥75,717 | ¥68,261 | ¥11,115 | | | | 100 | ¥83,207 | ¥73,499 | ¥11,686 | ¥32,574 | ¥25,139 | - break-even ≈ 34 KB - this article's 50 KB *Cloudflare does not charge for egress, so its two lines do not budge. A 500 KB coding-agent request is off the chart: Alibaba Cloud self-hosted ¥365,488, AWS ¥236,506, Oracle ¥34,518.* - **Light luggage: self-hosting is cheaper.** Below about 34 KB out per request, self-hosting on Alibaba Cloud beats Cloudflare (about 24 KB against Cloudflare's optimized design). Short Q&A is around 5 KB, where self-hosting costs about a third of Cloudflare. - **Heavy luggage: Cloudflare is cheaper.** Coding agents' requests often carry hundreds of kilobytes, and there Cloudflare is several times cheaper. AWS breaks even even earlier than Alibaba Cloud. - **Oracle is cheapest across the whole range**, only drawing level with Cloudflare at three to four hundred kilobytes. - **The smaller the volume, the lower the break-even point.** At 10 million requests a month the clouds' floor already costs more than Cloudflare's whole bill, so they never break even. So "Cloudflare is cheaper" **holds only when volume is small or the luggage is heavy**. The Chinese clouds have one more lever of their own: **if the upstream model lives on the same cloud's internal network** (calling Alibaba Cloud Model Studio from Alibaba Cloud, say), the forwarded prompt need not cross the public internet. Model Studio's private connection is currently offered only in Beijing and Hong Kong; with the gateway in the same region, at 1 billion requests a month the egress line drops from about ¥38,000 to about ¥19,000 (private-connection fees included). A gateway in Shenzhen reaching Beijing across regions saves almost nothing. ### Step 4: turn the volume up a hundredfold All the parts are in. Now turn the traffic from 10 million to 1 billion requests a month, the three tabs on the chart at the start of this section. - **At small volumes, the floor decides.** Cloudflare has no floor and the clouds do, so at 10 million it is an order of magnitude cheaper. - **At volume, bookkeeping and luggage decide.** Cloudflare's money goes to bookkeeping and the other clouds' to egress, and the two end up only 30% to a little over 100% apart. Oracle's egress is almost free, which puts it in front. - **CPU barely moves the bill.** Doubling the gateway's CPU time from 5 ms to 20 ms changes Cloudflare's monthly bill by only 6%, and a self-hosted setup still fits on four 8-vCPU machines. When you are billed by CPU time, the 20 seconds of waiting cost almost nothing. And one fact that should make you relax: **the gateway's infrastructure is small change next to the model bill.** Per request, Cloudflare costs about ¥0.00003 and self-hosting on Alibaba Cloud about ¥0.000045; a call with 7,500 input tokens and 300 output tokens costs about ¥0.017 at DeepSeek V4.1-Flash peak prices, so the gateway is only 0.2–0.3% of it, and even with the far cheaper Qwen qwen-flash just 2–3%.[^model] So the bill alone should not decide the platform. Latency, reachability and stability matter at least as much. ### Step 5: move next to your users The last part: where the entry point sits, and how far it is from your users. | | | | --- | --- | | Alibaba Cloud · Shenzhen | 7 ms | | Tencent Cloud · Guangzhou | 8 ms | | Alibaba Cloud · Hong Kong | 13 ms | | Tencent Cloud · Hong Kong | 13 ms | | Alibaba Cloud · Singapore | 57 ms | | Tencent Cloud · Singapore | 89 ms | | Cloudflare（anycast, lands in Los Angeles） | 162 ms | | AWS · Singapore | 203 ms | | AWS · US East | 224 ms | | Oracle · Singapore | 230 ms | | Alibaba Cloud · Virginia | 231 ms | | Oracle · US East | 236 ms | *Ping from a China Telecom home line in Shenzhen, 2026-10-09, 20 probes each. Mainland regions and Cloudflare are the 10:49 median, the overseas regions the 11:55 average. Azure and Google Cloud answer on global anycast addresses, so no region can be measured.* From China Telecom in Shenzhen, the Chinese clouds' entry points in Shenzhen, Guangzhou and Hong Kong are all around 10 ms away; packets to Cloudflare leave the country over the carrier's backbone and land in Los Angeles, 162 ms away.[^latency] Singapore is not necessarily close either: in the same city, Alibaba Cloud is 57 ms away while AWS and Oracle take over 200 ms, about as much as US East. Chinese clouds' overseas regions usually interconnect far better with Chinese carriers. What does that do to the first token? A new HTTPS connection needs at least three round trips (TCP, TLS 1.3, sending the request), so at 162 ms per round trip the first token arrives about 0.45 seconds later; with a reused connection, about 0.15 seconds. Chat users will notice; long agent tasks barely will. Three more things: - **China Network cannot rescue this architecture.** Without China Network, users in mainland China connect to data centers outside the mainland. China Network needs the Enterprise plan, a separate subscription, an ICP filing or license for every apex domain, and content review by JD Cloud; its product list includes Workers, KV and R2 but **not DO, D1 or Queues** (Cloudflare does not say whether these fail on JD Cloud nodes or fall back to overseas). `*.workers.dev` is blocked in mainland China, so you must use your own domain.[^workers-dev] - **For overseas models, the long hop cannot be avoided.** Neither OpenAI's nor Anthropic's list of supported API regions includes mainland China or Hong Kong, so a gateway calling them must exit from a supported region such as Singapore, Japan or the United States; Hong Kong can only relay.[^regions] Seen from a mainland user, the long hop happens either on "user → Cloudflare Los Angeles" or on "offshore gateway → US upstream", so when calling overseas models Cloudflare's latency handicap shrinks a lot and the difference is mostly line quality. - **Both sides have had major outages, of different shapes.** On 2025-06-12 a third-party cloud that KV depended on went down and AI Gateway's error rate peaked at 97%; on 2025-11-18 a core proxy failure made KV return many 5xx errors too. A gateway that reads KV or a DO synchronously on every request goes down with the whole world, with no other zone to fail over to. Alibaba Cloud Hong Kong Zone C was down for more than ten hours on 2022-12-18 after a cooling failure, but the failure was confined to one zone, so a multi-zone deployment had a way out. --- ## Traps to step around before launch Beyond the bill, every cloud has a few traps that can double it, or cut requests off halfway. ### Three Cloudflare traps - **Keeping a DO awake for the whole stream.** Route the stream through the DO, or arm a `setTimeout` in it as a lease timeout, and the DO is billed by wall-clock duration, adding up to about ¥215,000 a month at 1 billion requests. Use `setAlarm` for lease timeouts. Since 2026-10-01, pending outbound calls and `waitUntil` in a DO also keep it awake (for up to 15 minutes), which makes this easier to hit. - **AI Gateway logs the full prompt and response by default.** Accounts that create their first AI Gateway on or after 2026-09-24 pay for logs at Workers Logs prices, about ¥94,000 a month at 1 billion requests. Turn off payload logging, or logging altogether. Also, with Cloudflare's Unified Billing each gateway allows only 200 requests a minute, which the average rate at 10 million requests a month already exceeds, so a reseller has to bring its own upstream keys (BYOK). - **The default log settings.** New Workers have logs on by default and write an invocation log for every call, and by the docs each DO RPC writes one too, so one request makes about 4 log entries. At Cloudflare's own average of 4.84 KB per entry, the log bill grows about twentyfold. Turn off invocation logs or sample them on the gateway and DO Workers.[^logs] ### Four Chinese-cloud traps - **Offshore tiered pricing must be switched on by hand.** Alibaba Cloud Hong Kong, Singapore and Tokyo charge less per GB than Shenzhen, so at 1 billion requests a month Hong Kong comes out cheaper than Shenzhen even though its machines cost nearly twice as much. But since 2024-12-12, ECS and EIP **are no longer billed on the tiered price** (Alibaba Cloud calls it CDT) automatically; you must "upgrade to CDT billing" by hand (free, and irreversible). Without it, Hong Kong costs about ¥21,000 more a month at 1 billion requests. Singapore and Tokyo are the other way round: without the upgrade their base price is below the first tier, so within a range of volumes not upgrading is cheaper. - **Do not send upstream traffic through NAT.** Since 2025-09-26 Alibaba Cloud's NAT gateway bills processed traffic, so routing upstream requests through NAT adds about 23% on top of egress. A pay-by-traffic public IP on each ECS instance is cheaper. - **Long-lived connections are cheap, but timeouts cut them.** Alibaba Cloud ALB's capacity charge is driven by bytes processed, and 23,000 long-lived connections come to fewer than 8 capacity units; Tencent Cloud's shared CLB does not bill capacity at all. But **ALB's request timeout defaults to 60 seconds**, so raise it before long reasoning streams start returning 504. - **The managed AI gateway is not cheap.** Alibaba Cloud's AI Gateway (built on the open-source Higress) has multi-model routing, fallback, consumer API keys and token rate limits, but the smallest size costs about ¥4,000 a month (documented price); you still need your own Redis and database for balances and usage, so at small volumes it costs more than three times the whole self-hosted setup. The Serverless edition, billed from 2026-09-01, has no size floor in the thousands (the enterprise edition's instance fee is about ¥176 a month), so at small volumes it costs about the same as self-hosting; but it bills public traffic in both directions at ¥0.8/GB, outside the tiered prices, so it gets more expensive at volume. ### The four Western clouds: the big three keep the same books as the Chinese clouds, Oracle is the exception The big three's egress prices are in the same range as the Chinese clouds', the equivalent of ¥0.6–0.8 per GB in the first tier and a little over ¥0.5 at volume; their machines, databases and logs cost more, logs at $0.50 per GB being over 8× Alibaba Cloud's log service. The two effects cancel out, and at volume the big three and the Chinese clouds end up with similar bills. At small volumes AWS US East costs 50% more than Alibaba Cloud Shenzhen, and Azure US East nearly three times as much. If your clients are in mainland China, the traffic Google Cloud sends back to them falls under the price list's "to mainland China" rate, $0.20–0.23 per GiB (Google does not say how the destination is determined); pricing all egress that way (an upper bound) raises US East at 1 billion requests a month from about ¥44,000 to about ¥81,000. Oracle's egress is almost free: the first 10 TB each month cost nothing, and beyond that it is $0.0085/GB in North America and $0.025/GB in Asia-Pacific; traffic within a region and data handled by the load balancer are not billed separately, and log ingestion is free. So at 1 billion requests a month, 50 TB of egress costs about ¥2,300 in US East, against about ¥38,000 for the same traffic on Alibaba Cloud Shenzhen. The costs hide elsewhere: - The 10 TB free tier applies per source region group: US East and Singapore each get 10 TB, and regions in the same group share one.[^oci-10tb] - Singapore has only one availability domain (Oracle's term for an availability zone), so you cannot spread across zones, only across fault domains. - A pay-as-you-go account gets only 6 OCPUs per availability domain and one region by default, so at volume you must request a quota increase first. - The NAT gateway caps concurrent connections to the same destination address and port: about 20,000 per availability domain in a three-domain region such as US East, and about 65,000 in Singapore. 23,000 streams in one availability domain going to one upstream would hit it, so machines are better off exiting through their own public IPs (reserved public IPs are officially free). - The smallest high-availability MySQL is billed as 3 instances, $170 a month, the largest item at small volumes. **Default load balancer timeouts are the easiest trap before launch:** | | Default | Effect on SSE | |---|---|---| | Alibaba Cloud ALB | 60 s request timeout, counted as time without data between ALB and the backend (per the WebSocket docs; SSE is not covered) | A reasoning model that thinks for over 60 s before its first token gets a 504; raise it first | | AWS ALB | 60 s idle timeout, up to 4,000 s | A reasoning model that thinks for over 60 s before its first token gets cut off; raise it or send SSE heartbeats | | Google Cloud external Application Load Balancer | 30 s backend service timeout, **counting the whole response** | Not an idle timeout: streams over 30 s are cut off; global load balancers can go up to 86,400 s, so raise it before launch | | Azure Application Gateway v2 | 20 s request timeout, counted as time without data | The troubleshooting docs say that after a timeout it resends the request to another backend, with no exception for POST; if POST is retried too, a request whose first token takes over 20 s **may be forwarded twice and billed twice upstream**[^appgw]; SSE also needs the response buffer, on by default, switched off | | Oracle flexible load balancer | 60 s idle timeout, up to 7,200 s | Sending data does not reset the receive timer | ### Serverless and edge functions: do not count on them at volume Each duration-billed product has its own temper: - **Alibaba Cloud Function Compute**: bills the configured size, not actual use, so the vCPU is billed in full for the 20 seconds spent waiting upstream and never enters the "light sleep" state in which vCPU is free. The only saving is concurrency per instance: at 1 request per instance the compute charge is about 95 times that at 100. - **Tencent Cloud SCF**: by default one instance handles only one SSE connection at a time, and at volume the peak also exceeds the default concurrency quota per region. - **AWS Lambda**: default concurrency is 1,000, so volume needs a quota increase. - **Google Cloud Run**: one instance can handle up to 1,000 requests at once, the best fit for long-lived connections of the lot, but it still bills instance wall-clock time; it also writes request logs automatically at $0.50 per GiB (an exclusion filter can turn them off), which the chart leaves out. - **Azure**: Functions Flex defaults to 16 concurrent requests per instance, so 1 billion requests a month would need 1,447 instances, above the 1,000-instance cap; Container Apps runs once concurrency is raised to 100. - **OCI Functions**: one instance handles one request at a time, synchronous calls last at most 300 seconds, and the result only comes back after the function finishes, so it cannot stream. Edge functions are blunter still: **Chinese edge functions cannot serve as a general LLM gateway**, because of their limits rather than their price: | | Cloudflare Workers | Alibaba Cloud ESA functions | Tencent Cloud EdgeOne edge functions | |---|---|---|---| | Billing | Requests + CPU time | Per invocation, ¥5 per million | Requests ¥1.7 per million + CPU ¥0.11 per million ms | | Total duration per invocation | Unlimited (while the client stays connected) | **120 s** (waiting counts) | No limit documented; `fetch` timeout configurable up to 300 s | | First byte | No limit documented | **504 if no data within 10 s** | No limit documented | | Request body | 100 MB and up (depends on the zone's plan) | Not in the function docs; site upload limit 300 MB by default | **1 MB** | | CPU per invocation | 30 s by default, configurable up to 5 min | No figure documented | **200 ms** | | Subrequests | 10,000 by default | **4** (the Chinese docs say 4 per invocation, the English docs 4 concurrent and raisable on request) | 64 | | Strongly consistent state | Durable Objects | None (KV is eventually consistent, up to 300 s) | KV in beta, Enterprise only, 1 million reads per namespace per day | | Mainland China nodes | Only via China Network (Enterprise + ICP) | Yes, with ICP filing | Yes, with ICP filing | Reasoning models often take more than 10 seconds to the first token and more than 120 seconds for long outputs, and agents' long-context requests easily exceed 1 MB; neither Chinese product has strongly consistent state like a DO, so rate limits and balances still have to go back to a central region. Overseas, neither CloudFront Functions nor Lambda@Edge can relay a 20-second SSE stream, and Azure Front Door's docs state that it does not support SSE. **Edge functions can stand in front of a gateway for caching and authentication pre-checks; they cannot replace it.** As for managed AI gateways overseas, the big three have them and Oracle does not, and none is built for reselling: - **Azure API Management**: all three v2 tiers pass SSE through and offer per-token rate-limit policies such as `llm-token-limit`; at small volumes Basic v2 costs about $148 a month (Microsoft positions it for development and testing). The bottleneck is the cap of 2,048 concurrent backend connections per upstream host: one SSE stream holds one connection, so the peak at 100 million requests a month already exceeds it. The classic tiers state the cap per unit while v2 just says 2,048; read per unit, 1 billion requests a month needs 12 units at about $18,000 a month, and read literally, 12 separate instances plus an extra layer to split traffic. HTTP/2 to the upstream is still in preview in v2. - **AWS**: Bedrock AgentCore Gateway (inference targets since 2026-07) can proxy OpenAI, Anthropic and compatible endpoints, passing SSE through unchanged, and since 2026-08 can cap RPM, TPM and concurrency per user identity; but built-in authentication is only IAM, JWT or none, custom checks need an interceptor, which the docs say does not yet work in streaming mode, and there are no per-customer API keys, per-customer usage bills or prepaid balances. API Gateway REST APIs have supported streamed responses since 2025-11, but cannot read usage from the stream. - **Google**: Apigee passes SSE through, but its per-token rate-limit policy only works on the most expensive proxy type, about $73,000 a month in call fees alone at 1 billion requests. - **Oracle**: no general LLM gateway; its generative AI service only calls models in its own catalog. --- ## Who really runs an LLM gateway on Workers Only first-hand evidence counts here: official blogs, official docs, deployment configs in repositories and articles signed by founders. | Platform | Cloudflare components used | Evidence | Strength | |---|---|---|---| | **Cloudflare AI Gateway** | Workers; logs first in D1, then moved to R2, then to DOs sharded by account + gateway | [Official blog, 2024-10-24](https://blog.cloudflare.com/billions-and-billions-of-logs-scaling-ai-gateway-with-the-cloudflare/) | Confirmed | | **Helicone** (hosted proxy and hosted AI Gateway) | Workers + DO (rate limiters, plus one wallet per organization that holds the worst-case cost at the start and settles actual usage at the end) + KV + Queues; request bodies over 20 MiB go to Containers; backend on AWS | [`worker/wrangler.toml`](https://github.com/Helicone/helicone/blob/main/worker/wrangler.toml), [wallet DO](https://github.com/Helicone/helicone/blob/main/worker/src/lib/durable-objects/Wallet.ts), [official docs](https://docs.helicone.ai/references/availability) | Confirmed (edge request path) | | **OpenRouter** | Edge layer on Workers, caching user and API key data at the edge; when a balance runs low it checks the database more often and latency rises | [Official docs](https://openrouter.ai/docs/guides/best-practices/latency-and-performance) | Confirmed (edge layer only) | | **Portkey** | The open-source gateway repo ships a production `wrangler.toml`, and the official docs say the hosted version runs on "edge workers" worldwide | [wrangler.toml](https://github.com/Portkey-AI/gateway/blob/main/wrangler.toml) | Indirect evidence | | **Unkey** (moved away) | Used Workers + DO for API key verification and rate limiting; in 2025-08 moved to a stateful Go service on AWS | [Co-founder's post, 2025-08-01](https://www.unkey.com/blog/serverless-exit) | Confirmed | | **Braintrust** (moved away) | Its AI Proxy ran on Workers in 2023; the hosted entry point is now a Gateway on AWS, for reasons not made public | [2023 blog post](https://www.braintrust.dev/blog/ai-proxy), [current docs](https://www.braintrust.dev/docs/deploy/gateway) | Confirmed that it moved | Three things stand out: 1. **Apart from Cloudflare itself, none of them is "all Cloudflare".** The closest is Helicone: Workers, DO, KV and Queues on the request path are all on Cloudflare, while analytics and the billing backend are on AWS. This article's Cloudflare side keeps even billing in D1, which is more aggressive than what these companies actually do. 2. **Everyone's pain is in the bookkeeping layer, not compute.** AI Gateway's log storage, Helicone's wallet holds, OpenRouter's database checks on low balances, the Pydantic gateway's soft KV limits: all of them. 3. **Unkey left for reasons specific to its workload.** Its reasons: cache reads had to go over the network, with p99 above 30 ms; to work around statelessness it stacked up DO, Queues, Workflows and more; exporting data was painful; customers could not self-host. After moving, latency fell to one sixth. But Unkey has to finish key verification within 10 ms per request, whereas an LLM gateway request lasts 20 seconds anyway, so tens of milliseconds of state reads weigh far less, and that conclusion does not carry over as is. On the Chinese side the mainstream approach is **a long-running gateway process plus Redis**, not edge functions: - **Higress**: the gateway open-sourced by Alibaba, accepted into the CNCF Sandbox on 2026-03-15, about 9,500 stars. The project says it powers the Qwen app, Model Studio's model API and PAI, but publishes no traffic figures. Its token rate limiting uses Redis for global counters, the same design point as the DO counters on the Cloudflare side. - **Alibaba Cloud AI Gateway**: in an official customer case, all of China Pacific Property Insurance's LLM traffic goes through the AI Gateway, close to 100 million tokens a day. - **Chinese edge functions**: I found no named LLM gateway customer. --- ## How to choose **Start from your app.** Once you have split the bill, it goes roughly like this: 1. **Users mostly in mainland China?** Use a Chinese cloud, and get an ICP filing before the site serves anyone. 2. **Only static pages** (a personal website, blog, portfolio)? Use static hosting plus a CDN; it costs next to nothing. 3. **A waiting app** (calls an LLM or third-party APIs)? For overseas users, edge functions; for users in China, one small always-on server; not duration-billed serverless. 4. **A computing app** (image or video processing)? Duration-billed serverless when the work is occasional, machines when it runs flat out. 5. **A light app** (an ordinary backend)? Start on free tiers or the cheapest server plan, and redo the three bills once the volume is real. **The decision tree and lists below are for LLM gateways.** ```mermaid flowchart LR %% layout: tree Q(["Where are your paying users?"]) Q --> CN["Mainland China"] --> CN1["Mostly Chinese models
→ Alibaba / Tencent Cloud mainland: long-running gateway (Higress or your own) + Redis
avoid Function Compute / SCF; edge functions only in front"] CN --> CN2["Overseas models too
→ mainland entry + offshore node for overseas models
exit from a region OpenAI / Anthropic support (e.g. Singapore, Tokyo)
upgrade Hong Kong nodes to CDT; work out Singapore and Tokyo first"] Q --> OS["Overseas"] --> OS1["Small, no machines to run
→ Cloudflare Workers + one DO per key
billing split by month or an external database; logs without payloads"] OS --> OS2["Large and egress-heavy
→ self-host on Oracle (first 10 TB of egress free each month)
already on AWS / GCP / Azure: egress bill similar to Chinese clouds"] Q --> BOTH["Both"] --> BOTH1["→ two entry points, each close to its users
billing in one place only, no dual writes"] classDef q fill:#fff3bf,stroke:#f08c00 classDef pick fill:#d3f9d8,stroke:#2f9e44 class Q q class CN1,CN2,OS1,OS2,BOTH1 pick ``` *Figure: a decision tree for choosing by where users are (this article's judgment, not official advice).* **When to choose Cloudflare:** - Your users are mostly overseas, or they are developers calling overseas models; - You are still small (tens of millions of requests a month or fewer) and want a near-zero floor with no machines to run; no self-hosted setup at the six cloud providers can match it; - Your requests are large (agents, long context, multimodal), so egress would dominate a cloud provider's bill; - Your team can accept bookkeeping-driven design work such as one DO per key and periodic D1 archiving. **When not to use Cloudflare:** - Your users are mostly in mainland China and sensitive to time to first token (chat products); - You are at around 1 billion requests a month with short requests (under about 30 KB out per request), where self-hosting on a Chinese cloud or one of the big three is cheaper; - Your users are overseas, you are large, and you are willing to run your own machines: self-hosting on Oracle is about three times cheaper; - You need strongly consistent global state (global concurrency, a credit pool across keys) and do not want to shard it yourself; - You need nodes in mainland China but cannot afford Enterprise, or you rely on DO, D1 or Queues. --- ## Closing Back to "your service" in the comics. In a whole day, the time it actually spends working may add up to less than a minute; the rest of the time it is waiting for the LLM to finish, carrying data for users, and writing entries in the books. No cloud bills it only for the work; each one picks one or two of waiting, carrying and bookkeeping to bill instead. So when you go live for the first time, split the bill first and start on free tiers; by the time the volume is real, you will know which line to watch. Change the kind of app and the answers to the three questions change, and so does the conclusion: an LLM gateway is **bound by waiting, by carrying bytes and by state**, and comparing two clouds on CPU price compares the smallest line on its bill. > **Rule: first ask three questions — who bills you for waiting, who bills you for carrying bytes, who bills you for state. Then look at which side your users are closer to.** --- ## Appendix: how the prices were queried, and the full numbers Every number in the main text comes from here. "10M / 100M / 1B" in the table headers are requests per month. ### How the prices were queried - **The two Chinese clouds were priced through their official command-line tools.** Alibaba Cloud through aliyun CLI 3.5.1 (`DescribePrice`, `GetPayAsYouGoPrice`), Tencent Cloud through tccli 3.1.180.1 (`InquiryPrice*`, `DescribeDBPrice`). Every call is a read-only pricing query, and no resource was created. - **All Cloudflare numbers come from the official pricing and limits pages.** Cloudflare has no pricing API; wrangler 4.148.0 and the Cloudflare API only report your own account's usage. - **The four Western clouds were priced through public pricing APIs that need no login** (AWS Price List, Azure Retail Prices API, Oracle's price list API); Google Cloud's pricing API needs an API key, so its numbers come from the official pricing pages. US East and Singapore, all on-demand, 720-hour months. Reserved instances or Savings Plans can save another 30–50%, but only on machines, not on egress. - **List prices only.** Alibaba Cloud's pricing APIs also return account discounts (my account, for example, gets 15% off load balancers and NAT), which are left out. Where no API works, the documented price is used and marked; Alibaba Cloud's AI Gateway, for example, has no pricing module in the billing system. - **Two verification rounds on the same day**: every item the first rounds had marked unverified was checked against official sources; what could be confirmed is now in the text, and what still cannot (mostly things that need an account, a load test or a real bill) is still marked.
A few representative pricing queries (the full command list and raw output were kept) ```bash # Alibaba Cloud: ECS monthly price (40 GB ESSD PL0 system disk, no bandwidth) aliyun ecs DescribePrice --RegionId cn-shenzhen --ResourceType instance \ --InstanceType ecs.c9i.xlarge --PriceUnit Month --Period 1 \ --SystemDisk.Category cloud_essd --SystemDisk.PerformanceLevel PL0 --SystemDisk.Size 40 # Alibaba Cloud: total price of 50,000 GB of public egress on CDT tiers aliyun bssopenapi GetPayAsYouGoPrice --region cn-hangzhou --ProductCode cdt \ --ProductType cdt_DataTransfer_public_cn --SubscriptionType PayAsYouGo --Region cn-shenzhen \ --ModuleList.1.ModuleCode internet_traffic --ModuleList.1.PriceType Usage \ --ModuleList.1.Config 'Region:cn-shenzhen,internet_traffic:50000,charge_type:PayByTraffic,isp:BGP' # Alibaba Cloud: Redis (Tair) and RDS monthly prices aliyun r-kvstore DescribePrice --RegionId cn-shenzhen --ZoneId cn-shenzhen-c \ --InstanceClass redis.shard.large.ce --OrderType BUY --ChargeType PrePaid --Period 1 --NodeType MASTER_SLAVE aliyun rds DescribePrice --RegionId cn-shenzhen --ZoneId cn-shenzhen-c --Engine MySQL --EngineVersion 8.0 \ --DBInstanceClass mysql.n4.large.2c --DBInstanceStorage 100 --DBInstanceStorageType cloud_essd \ --PayType Prepaid --UsedTime 1 --TimeType Month --Quantity 1 --OrderType BUY --CommodityCode rds # Tencent Cloud: CVM monthly price (50 GB general-purpose SSD, bandwidth set to 0) TENCENTCLOUD_REGION=ap-guangzhou tccli cvm InquiryPriceRunInstances --cli-unfold-argument \ --Placement.Zone ap-guangzhou-6 --ImageId img-mmytdhbn --InstanceType SA9.LARGE8 \ --InstanceChargeType PREPAID --InstanceChargePrepaid.Period 1 \ --SystemDisk.DiskType CLOUD_BSSD --SystemDisk.DiskSize 50 \ --InternetAccessible.InternetChargeType TRAFFIC_POSTPAID_BY_HOUR \ --InternetAccessible.InternetMaxBandwidthOut 0 --InstanceCount 1 ``` ```bash # AWS: egress tiers from us-east-1 to the internet (public Price List file, AWSDataTransfer service) curl -s https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSDataTransfer/current/us-east-1/index.json # Azure: egress tiers in eastus (Retail Prices API) curl -s "https://prices.azure.com/api/retail/prices?\$filter=serviceName%20eq%20'Bandwidth'%20and%20armRegionName%20eq%20'eastus'%20and%20meterName%20eq%20'Standard%20Data%20Transfer%20Out'" # Oracle: egress originating in North America (price list API, B88327 = Outbound Data Transfer - Originating in North America, Europe, and UK) curl -s "https://apexapps.oracle.com/pls/apex/cetools/api/v1/products/?currencyCode=USD&partNumber=B88327" ```
### Key unit prices
Cloudflare, Alibaba Cloud (Shenzhen), Tencent Cloud (Guangzhou) | | Cloudflare (USD) | Alibaba Cloud (Shenzhen, CNY) | Tencent Cloud (Guangzhou, CNY) | |---|---|---|---| | Compute | Workers $5/month including 10M requests and 30M CPU ms; then $0.30 per million requests and $0.02 per million CPU ms; **no charge for wall-clock duration** | ECS c9i monthly: 2c4g ¥205.91, 4c8g ¥391.82, 8c16g ¥763.64 (system disk included) | CVM SA9 monthly: 2c4g ¥156.2, 4c8g ¥287.4, 8c16g ¥549.8 (system disk included) | | Load balancer | Not needed | ALB instance ¥0.049/hour + capacity units (LCU) ¥0.049 per LCU-hour; an LCU is the largest of four dimensions: new connections, concurrent connections, bytes processed and rule evaluations | CLB shared ¥0.2/hour, **no capacity charge** | | Egress | **Free** | CDT tiers: ¥0.80/GB up to 10 TB, ¥0.75 for 10–50 TB, ¥0.70 for 50–150 TB, ¥0.65 beyond; 20 GB free per month | ¥0.80/GB flat, no tiers and no free quota | | Counters / balance | DO: requests $0.15 per million; SQLite writes $1.00 per million rows (50M rows included) | Tair, two replicas: 1 GB ¥76.98/month, 4 GB ¥360/month | Redis primary-replica: 1 GB ¥76/month, 4 GB ¥304/month | | Key lookup | KV reads $0.50 per million (10M included) | The same Redis | The same Redis | | Usage / billing database | Queues $0.40 per million operations (3 per message); D1 writes $1.00 per million rows, **10 GB max per database** | RDS MySQL high-availability: 2c4g ¥660/month, 4c16g ¥1,370/month | MySQL two-node: 2c4g ¥480/month, 4c16g ¥1,704/month | | Logs | Workers Logs, **from 2026-12-01** $0.25/GB ingested + $0.10 per GB-month stored | SLS ¥0.4/GB (indexing and 30 days of storage included) | CLS billed by feature, about ¥0.83/GB |
AWS, Google Cloud, Azure, Oracle (US East, USD) | | AWS | Google Cloud | Azure | Oracle | |---|---|---|---|---| | Egress | 100 GB free per month; $0.09/GB up to 10 TB, $0.085 for 10–50 TB (Singapore first tier $0.12) | Premium Tier to North America: $0.12/GiB up to 1 TiB, $0.11 up to 10 TiB, $0.08 beyond (Singapore to Asia first tier $0.12) | 100 GB free per month; $0.087/GB up to 10 TB, $0.083 for 10–50 TB (Singapore first tier $0.12) | **First 10 TB free per month**, then $0.0085/GB (Singapore $0.025) | | 4 vCPU 8 GiB machine (monthly) | c8i.xlarge $134.94 | c4-highcpu-4 $122.47 | F4als_v7 $174.24 | E6.Flex 2 OCPU $54.72 | | Load balancer | ALB $16.20/month + LCU (about $0.008/GB processed) | Forwarding rule $18/month + $0.008/GiB | Application Gateway v2 $0.20/hour + $0.008 per capacity unit | Flexible LB $9–41/month, no charge for data handled | | ~1 GB high-availability Redis (monthly) | Valkey primary-replica $36.86 | Memorystore Standard $46.08 | Azure Managed Redis B1 $46.08 | OCI Cache $55.87 | | ~2-vCPU high-availability MySQL (monthly) | RDS Multi-AZ $115.88 | Cloud SQL HA $192.80 | Flexible Server zone-redundant HA (smallest: 2 vCore 8 GiB) $272.84 | HeatWave HA $170.11 | | Logs | CloudWatch Logs ingestion $0.50/GB | Cloud Logging $0.50/GiB, 50 GiB free per project per month | Basic Logs $0.50/GB | Ingestion free, storage $0.05 per GB-month |
### The full monthly bills
Self-hosted gateways and Cloudflare (Hong Kong and Singapore included) Machines on the Chinese clouds: 2 × 2 vCPU/4 GB at 10M, 2 × 4 vCPU/8 GB at 100M, 4 × 8 vCPU/16 GB at 1B, across two zones, not load-tested; database and Redis sizes grow with volume. CNY at list prices: | Option | 10M | 100M | 1B | |---|---|---|---| | Cloudflare, plain design (Workers + KV + Durable Objects + Queues + D1) | 140 | 2,756 | 32,574 | | Cloudflare, optimized design (usage summed inside the Durable Object, no Queues) | 67 | 2,115 | 25,139 | | Alibaba Cloud · Shenzhen | 1,625 | 5,854 | 45,696 | | Alibaba Cloud · Hong Kong (reference for an offshore node) | 1,981 | 6,114 | 40,426 | | Tencent Cloud · Guangzhou | 1,422 | 5,359 | 45,179 | | AWS · US East / Singapore | 2,467 / 3,227 | 6,646 / 8,572 | 47,179 / 53,327 | | Google Cloud · US East / Singapore | 3,056 / 4,031 | 7,387 / 8,583 | 43,798 / 48,564 | | Azure · US East / Singapore | 4,794 / 6,215 | 9,125 / 11,934 | 49,848 / 57,957 | | Oracle · US East / Singapore | 1,973 / 1,973 | 2,388 / 2,388 | 8,832 / 13,237 | Divide by 6.7153 for USD: Cloudflare at 1B is about $4,851, AWS US East about $7,026, Oracle US East about $1,315.
Managed gateways and serverless (also including Redis, database, logs and egress) | Option | 10M | 100M | 1B | |---|---|---|---| | Alibaba Cloud AI Gateway (managed) | 5,428 | 10,195 | 60,518 | | Alibaba Cloud Function Compute (100 concurrent requests per instance) | 1,419 | 7,703 | 68,877 | | Tencent Cloud SCF (default single concurrency / multi-concurrency on*) | 3,756 / 1,296 | 32,543 / 7,947 | 321,865 / 75,903 | | AWS Lambda (128 MB Arm, US East, platform logs included) | 4,142 | 28,874 | 278,460 | | Google Cloud Run (80 / 250 concurrent requests per instance, US East, automatic request logs excluded**) | 2,672 / 2,405 | 12,556 / 7,752 | 107,676 / 59,634 | | Azure Container Apps (100 streams per replica, US East) | 2,916 | 9,596 | 79,376 | | OCI Functions (Oracle, US East) | 4,071 | 27,445 | 264,894 | \* Tencent Cloud's SSE docs say one instance handles one SSE connection at a time, while its multi-concurrency docs list long-lived connections as the main use case with WebSocket as the only example; the multi-concurrency figures assume 100 concurrent requests per instance at 70% fill and were not tested. \*\* Cloud Run writes request logs automatically, billed at $0.50/GiB, and an exclusion filter can turn them off; at 1 KB each they add about ¥3,100 at 1B for either concurrency.
The bill for one request, and the largest items at each volume Marginal cost per million requests at the overage prices of 1 billion requests a month: | Cloudflare per million requests | USD | Alibaba Cloud Shenzhen per million requests | CNY | |---|---|---|---| | Workers requests | 0.30 | Egress, 50 GB × ¥0.75 | 37.5 | | Workers CPU (10 ms) | 0.20 | ALB capacity units (50 GB processed) | 2.45 | | 1 KV read | 0.50 | SLS logs, 1 GB | 0.40 | | 2 DO requests | 0.30 | Machines + Redis + RDS spread per million requests | about 4.8 | | **2 DO rows written** | **2.00** | | | | 3 Queues operations | 1.20 | | | | 1 KB of logs | about 0.27 | | | | **Total** | **about 4.8 (≈ ¥32)** | **Total** | **about ¥45** | Spreading the 1B bill over every request for the Western clouds: AWS US East about ¥0.000047, Oracle US East about ¥0.000009. | | 10M | 100M | 1B | |---|---|---|---| | Cloudflare | Queues 55%, Workers subscription 24% | DO row writes 37%, Queues 29%, KV reads 11% | DO row writes 40%, Queues 25%, KV reads 10% | | Alibaba Cloud self-hosted · Shenzhen | RDS 41%, ECS 25%, egress 24% | **Egress 68%**, ECS 13% | **Egress 83%**, ECS 7%, ALB 5% | | Tencent Cloud self-hosted · Guangzhou | MySQL 34%, egress 28%, CVM 22% | **Egress 75%** | **Egress 89%** | | AWS self-hosted · US East | EC2 37%, RDS 32%, egress 10% | **Egress 45%**, EC2 27%, RDS 12% | **Egress 61%**, EC2 15%, RDS 7%, logs 7% | | Oracle self-hosted · US East | MySQL 58%, Redis 19%, machines 19% | MySQL 48%, machines 31% | Machines 33%, **egress 26%**, MySQL 25% |
Egress per request: Alibaba Cloud Shenzhen in detail, and every provider's break-even point | Egress per request | 10M | 100M | 1B | For comparison: Cloudflare at 1B (plain / optimized) | |---|---|---|---|---| | 5 KB (short Q&A) | ¥1,243 | ¥2,033 | **¥9,478** | ¥32,574 / ¥25,139 | | 20 KB | ¥1,371 | ¥3,307 | ¥21,726 | Same | | 50 KB (this article's assumption) | ¥1,625 | ¥5,854 | ¥45,696 | Same | | 200 KB (agents with long context) | ¥2,899 | ¥18,102 | ¥155,788 | Same | | 500 KB (common for coding agents) | ¥5,446 | ¥42,072 | **¥365,488** | Same | Break-even with Cloudflare (egress per request, against the optimized to the plain design): - **1B**: Shenzhen about 24–34 KB, Hong Kong about 24–37 KB; AWS US East 15–27 KB, Google Cloud US East 16–30 KB, Azure US East 10–22 KB; Singapore egress costs more, so its break-even points are lower. Oracle US East 336–466 KB: from 5 KB to 500 KB its bill only moves from ¥6,562 to ¥34,518, while AWS US East moves from ¥18,896 to ¥236,506 over the same range. - **100M**: Shenzhen about 6–14 KB. - **10M**: the clouds' fixed floor already costs more than Cloudflare's whole bill, so there is no break-even point. CPU time: from 5 ms to 20 ms, Cloudflare at 1B moves from ¥31,902 to ¥33,917. With logs at the official average of 4.84 KB per entry, Cloudflare at 1B costs ¥39,622 for the plain design and ¥32,188 for the optimized one.
Raw latency data | 2026-10-09 10:49 (Beijing time), China Telecom home broadband in Shenzhen, 20 ICMP round trips | Median | P90 | |---|---|---| | Alibaba Cloud Shenzhen (ECS API endpoint) | 7 ms | 19 ms | | Alibaba Cloud Hong Kong | 13 ms | 16 ms | | Tencent Cloud Guangzhou (CVM API endpoint) | 8 ms | 9 ms | | Tencent Cloud Hong Kong | 13 ms | 20 ms | | Cloudflare anycast address (this blog's Pages / Functions endpoint) | 162–164 ms | 166–186 ms | Each provider's overseas regions, measured again on the same line at about 11:55 (20 ICMP round trips, average): | Singapore | Average | US East | Average | |---|---|---|---| | Alibaba Cloud Singapore | 57 ms | Alibaba Cloud Virginia | 231 ms | | Tencent Cloud Singapore | 89 ms | AWS us-east-1 (S3 endpoint) | 224 ms | | AWS ap-southeast-1 (S3 endpoint) | 203 ms | Oracle us-ashburn-1 | 236 ms | | Oracle ap-singapore-1 | 230 ms | | |
--- ## Sources and check dates All checked on 2026-10-09. - Cloudflare pricing and limits: - [Workers pricing](https://developers.cloudflare.com/workers/platform/pricing/), [Workers limits](https://developers.cloudflare.com/workers/platform/limits/) - [Durable Objects pricing](https://developers.cloudflare.com/durable-objects/platform/pricing/), [Rules of Durable Objects](https://developers.cloudflare.com/durable-objects/best-practices/rules-of-durable-objects/) - [How KV works](https://developers.cloudflare.com/kv/concepts/how-kv-works/), [Queues pricing](https://developers.cloudflare.com/queues/platform/pricing/) - [D1 pricing](https://developers.cloudflare.com/d1/platform/pricing/), [D1 limits](https://developers.cloudflare.com/d1/platform/limits/) - [Observability pricing (from 2026-12-01)](https://developers.cloudflare.com/observability/pricing/) - [AI Gateway pricing](https://developers.cloudflare.com/ai-gateway/reference/pricing/), [AI Gateway limits](https://developers.cloudflare.com/ai-gateway/reference/limits/) - [China Network](https://developers.cloudflare.com/china-network/), [China Network available products](https://developers.cloudflare.com/china-network/reference/available-products/) - Alibaba Cloud (prices from aliyun CLI pricing APIs, rules from the docs; pages in Chinese): - [CDT public egress](https://help.aliyun.com/zh/cdt/internet-data-transfers/), [Upgrade to CDT billing](https://help.aliyun.com/zh/cdt/user-guide/upgrade-to-cdt-billing) - [ALB billing rules](https://help.aliyun.com/zh/slb/application-load-balancer/product-overview/alb-billing-rules), [NAT Gateway billing](https://help.aliyun.com/zh/nat-gateway/nat-gateway-billing) - [Function Compute billing](https://help.aliyun.com/zh/functioncompute/fc/product-overview/billing-overview-of-fc), [Concurrency per instance](https://help.aliyun.com/zh/functioncompute/fc/configure-the-concurrency-of-a-single-instance) - [AI Gateway sizes and prices](https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/free-product-templates) - [ESA Functions and Pages limits](https://help.aliyun.com/zh/edge-security-acceleration/esa/user-guide/what-is-functions-and-pages), [ESA function billing](https://help.aliyun.com/zh/edge-security-acceleration/esa/user-guide/functions-and-pages-billing) - [Model Studio over a private connection](https://help.aliyun.com/zh/model-studio/access-model-studio-through-privatelink), [PrivateLink billing](https://help.aliyun.com/zh/privatelink/private-link-billing-description), [AI Gateway Serverless billing](https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/overview-of-billing-during-the-serverless-public-preview-of-the) - Tencent Cloud (prices from tccli pricing APIs, rules from the docs; pages in Chinese): - [Public egress prices](https://cloud.tencent.com/document/product/213/113026), [CLB LCU billing](https://cloud.tencent.com/document/product/214/58387) - [SCF billing](https://cloud.tencent.com/document/product/583/12281), [SCF SSE](https://cloud.tencent.com/document/product/583/90617), [Web function multi-concurrency](https://cloud.tencent.com/document/product/583/123888) - [EdgeOne plans](https://cloud.tencent.com/document/product/1552/94158), [EdgeOne overage prices](https://cloud.tencent.com/document/product/1552/94159), [Edge function limits](https://cloud.tencent.com/document/product/1552/81344) - Cases and outages: - Cloudflare post-mortems: [2025-06-12](https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/), [2025-11-18](https://blog.cloudflare.com/18-november-2025-outage/) - [Alibaba Cloud Hong Kong Zone C outage notice (2022-12)](https://www.alibabacloud.com/zh/notice/resolved_service_outage_in_zone_c_of_the_china_hong_kong_region_1ae) - [Higress](https://github.com/higress-group/higress), [Alibaba Cloud AI Gateway product page](https://www.aliyun.com/product/apigateway) - [GreatFire: workers.dev](https://en.greatfire.org/domain/workers.dev) (third-party monitoring), [Cloudflare community: workers.dev blocked in China](https://community.cloudflare.com/t/cloudflare-workers-suspected-of-being-blocked-in-china/382155) - [Cloudflare terms update (2023)](https://blog.cloudflare.com/updated-tos/), [Cloudflare log datasets (average entry size)](https://developers.cloudflare.com/observability/logs/datasets/) - The four Western clouds (prices from public pricing APIs or official pricing pages, rules from the docs): - AWS: [EC2 on-demand pricing (including data transfer)](https://aws.amazon.com/ec2/pricing/on-demand/), [Lambda pricing](https://aws.amazon.com/lambda/pricing/), [Lambda response streaming](https://docs.aws.amazon.com/lambda/latest/dg/configuration-response-streaming.html), [ALB idle timeout](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-load-balancer-attributes.html), [AgentCore Gateway inference targets](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-targets-inference.html) - Google Cloud: [Network pricing](https://cloud.google.com/vpc/network-pricing), [Cloud Run pricing](https://cloud.google.com/run/pricing), [Load balancer timeouts](https://cloud.google.com/load-balancing/docs/https/request-distribution#timeouts_and_retries), [Apigee pricing](https://cloud.google.com/apigee/pricing) - Azure: [Bandwidth pricing](https://azure.microsoft.com/en-us/pricing/details/bandwidth/), [Application Gateway and SSE](https://learn.microsoft.com/en-us/azure/application-gateway/use-server-sent-events), [Application Gateway 502 troubleshooting (timeout retries)](https://learn.microsoft.com/en-us/troubleshoot/azure/application-gateway/application-gateway-troubleshooting-502), [API Management AI gateway capabilities](https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities), [API Management limits (concurrent backend connections)](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#api-management-limits), [Front Door, WebSocket and SSE](https://learn.microsoft.com/en-us/azure/frontdoor/standard-premium/websocket) - Oracle: [Price list (networking)](https://www.oracle.com/cloud/price-list/#pricing-networking), [VCN pricing](https://www.oracle.com/cloud/networking/virtual-cloud-network/pricing/), [Press release on 10 TB per regional zone (2021-11-10)](https://www.oracle.com/news/announcement/oracle-joins-cloudflare-bandwidth-alliance-2021-11-10/) - Model prices: [DeepSeek pricing](https://api-docs.deepseek.com/zh-cn/quick_start/pricing), [Qwen qwen-flash](https://help.aliyun.com/zh/model-studio/qwen-flash) - Tool versions: aliyun CLI 3.5.1, tccli 3.1.180.1, wrangler 4.148.0 (latest on npm: 4.149.0); no CLI was used for the four Western clouds, only the public pricing APIs above. [^scope]: Alibaba Cloud prices come from the pricing APIs of aliyun CLI 3.5.1, Tencent Cloud's from tccli 3.1.180.1; Cloudflare has no pricing API, so its numbers come from the official pricing and limits pages; the four Western clouds come from public pricing APIs that need no login (AWS, Azure, Oracle) and Google's official pricing pages. Two verification rounds were done the same day; what still could not be confirmed is marked in the text. Exchange rate 1 USD = 6.7153 CNY (open.er-api.com, 2026-10-09). [^cpu-price]: Machines are their 4-vCPU monthly price per vCPU-hour, i.e. the price when fully used; the two Chinese clouds are monthly subscriptions (system disk included), the Western clouds on demand. Lambda counts 1,769 MB of memory as one vCPU, per the AWS docs. This compares orders of magnitude, not exact unit prices. [^limits]: Workers allows 128 MB of memory per isolate and 30 s of CPU per request by default, configurable up to 5 minutes; Lambda runs at most 15 minutes with up to 10,240 MB; a DO is not billed for duration while its WebSockets hibernate; R2 egress is free. All checked on the official pages on 2026-10-09. [^cpu-ms]: Cloudflare's limits page gives as reference about 2.2 ms for an average Worker and typically 10–20 ms for heavier work such as authentication or parsing large payloads. [^cf-wall]: Cloudflare's pricing page states that under the Standard usage model wall-clock duration is neither billed nor capped, and the limits page states that time spent waiting on `fetch()`, KV or a database does not count as CPU time. [^one-per-instance]: Tencent Cloud SCF's SSE docs state that by default one instance handles one SSE connection at a time; the docs for web functions' multi-concurrency list long-lived connections as the main use case but only give WebSocket as an example, so the chart's multi-concurrency figure assumes 100 concurrent requests per instance at 70% fill and was not tested. An AWS Lambda execution environment handles one request at a time, and a streamed response is billed until the whole stream ends, even if the client disconnects. [^do]: A DO idle for about 10 seconds may hibernate. The official soft limit for one object is about 1,000 requests a second, and about 200–500 with storage writes; the official design rules page explicitly warns against using a single DO for global rate limiting. [^kv]: KV is eventually consistent: other locations may take 60 seconds or more to see a change. Pydantic's open-source AI gateway admits in its old README that with state cached in KV, spending limits can only be "soft". [^d1]: A usage table summed per minute fills a database in about 1.2 months at 1 billion requests a month and about 4.6 months at 100 million; an account can have 50,000 databases. [^redis]: Alibaba Cloud Tair, two replicas, 1 GB at ¥76.98 a month; the 4,600 operations a second assume a peak at three times the average and were not load-tested. [^tos]: Since Cloudflare's 2023 terms update, the limit on serving large amounts of non-web content applies only to the CDN, not to the developer platform ([terms update](https://blog.cloudflare.com/updated-tos/)). [^model]: Model prices come from the DeepSeek and Qwen official pricing pages (checked 2026-10-08). A 30 KB prompt being about 7,500 tokens and the answer 300 tokens is a rough estimate. [^latency]: China Telecom home broadband in Shenzhen, 2026-10-09, 20 ICMP probes per entry point: mainland regions and Cloudflare are the 10:49 median, the overseas regions the 11:55 average. The local network transparently proxies every TCP connection (any handshake takes 2–3 ms), hence ping; `mtr` shows packets to Cloudflare jumping from about 9 ms to about 160 ms on China Telecom's 163 backbone. This is one line at one time of day; two evenings earlier, a TCP handshake to Cloudflare from the same network measured 193 ms. Azure and Google Cloud answer on global anycast addresses, so no region can be measured, and they are left out. [^workers-dev]: A Cloudflare employee confirmed it in the [official community](https://community.cloudflare.com/t/cloudflare-workers-suspected-of-being-blocked-in-china/382155), and GreatFire's third-party monitoring still showed it fully blocked as of 10-07. [^regions]: The supported-region lists of [OpenAI](https://developers.openai.com/api/docs/supported-countries) and [Anthropic](https://www.anthropic.com/supported-countries), checked 2026-10-09. [^logs]: Workers Logs switches from per-event to per-GB pricing on 2026-12-01 (announced in Cloudflare's 2026-10-02 blog post); this article uses the new prices, about $260 of logs a month at 1 billion requests ($588 under the old ones). Metering includes the fields Cloudflare adds automatically, so 1 KB per entry is a lower bound; at the official 4.84 KB average, the plain design at 1 billion requests a month goes from about ¥33,000 to about ¥40,000 in total, and logs from about $260 to about $5,300. [^oci-10tb]: Oracle's 2021 [press release](https://www.oracle.com/news/announcement/oracle-joins-cloudflare-bandwidth-alliance-2021-11-10/) and a 2025 official white paper both say "each regional zone". [^appgw]: An inference from the troubleshooting docs, not tested. --- title: "Letting an Agent Run a Dozen Overseas Social Accounts: About 60 Posting, Comment and DM APIs and MCP Servers Compared Cell by Cell (October 2026)" description: "You want an Agent such as Claude Code to publish to X, LinkedIn, YouTube, Instagram and other accounts in one go, then read the comments and answer the DMs. How far that can go is decided first by each platform's official API; the tools only carry those official capabilities into one interface. Since February 2026 X lets programs reply only to people who mentioned you, nobody can read the comments or DMs of a LinkedIn personal profile, Facebook only lets you post to Pages, and Threads and YouTube have no DM API. Within those limits I checked about 60 unified APIs, social media suites, open-source projects and session-based APIs against their official documentation of 2026-10-09, cell by cell across platform × posting / comments / mentions / DMs, scored them for integration completeness with price only as the second column, and marked what I tested and what I only read." url: "https://zhuoqidev.com/en/posts/social-publishing-inbox-apis-2026/" language: "en" author: "Liu ZhuoQi" date: "2026-10-09" updated: "2026-10-09" tags: ["Social Media Automation","MCP","Claude Code","X API","Unified API","Architecture"] translation: "https://zhuoqidev.com/posts/social-publishing-inbox-apis-2026/index.md" --- # Letting an Agent Run a Dozen Overseas Social Accounts: About 60 Posting, Comment and DM APIs and MCP Servers Compared Cell by Cell (October 2026) > **TL;DR** Ranked by integration completeness, Zernio leads the developer-facing unified APIs with 35.3 points out of 38, with SocialAPI.ai and Ayrshare at about 29 each. Among the established social media suites, only Vista Social and Eclincher can post, read the inbox and reply through a single MCP server. The only way to get more, such as LinkedIn personal DMs, is a session-based API that asks for your cookies, which is exactly what the platforms forbid in writing. First check what the platform gives you, then compare how much of it each tool carries. **Version scope**: every product capability, price and platform rule was checked on 2026-10-09 against official documentation, pricing pages and API references. Prices are list prices in US dollars, without discounts or promotions. GitHub stars, last commits and licences were queried with `gh api` the same day. This field moves fast: X changed its API rules twice in 2026, Reddit's public API closes in March 2027, and several of the MCP servers only appeared in September. > [!NOTE] > **What I tested: this article is mostly documentation reading.** Only Zernio was tested (2026-10-09): its read-only REST endpoints, the MCP handshake and tool list, connecting it to Claude Code, and, with my own X account, publishing a post and then replying to a comment I left under it myself (section 7). **Whether replying to someone else's comment is blocked by X's "summoned" rule is not tested yet**; everything about the other vendors comes from documentation. No vendor sponsored this and there are no referral links. --- ## The short version 1. **The platform decides what is possible.** Since 2026-02-23 X only lets a program reply to someone who mentioned or quoted you, and AI-generated replies posted automatically need X's approval first. A LinkedIn personal profile can post, but its comments and DMs are open only to legal entities and partners. A Facebook personal profile cannot post through the API. Instagram needs a professional account. Threads and YouTube have no DM API at all. **No tool gets around any of this.** 2. **Developer-facing unified APIs (19 vendors)**: Zernio is first with 35.3. Without your own X developer key it reads and answers replies under your X posts and X DMs (tested: reading needs a background switch that bills continuously), connects all 17 accounts in the example, exposes comment and DM tools in its MCP server, and pushes new messages by webhook. Ayrshare (29.3) covers almost as much, but needs your own X key, needs a separate profile for each extra account on the same platform, and costs four times as much. SocialAPI.ai (29.5) is cheap, but posts only plain text to X and has no Reddit. OmniSocials (23.8) is the cheapest thanks to per-workspace pricing, but cannot read replies under your X posts. 3. **Established social media suites (23 vendors)**: only Vista Social (about $217–257 a month for the example accounts) and Eclincher ($134–149 a month, no Bluesky, some features unverified) can post, read the inbox and reply through MCP. Agorapulse's MCP server can only save drafts. Hootsuite's can only save drafts and read, and while it is switched on it blocks X, LinkedIn, Reddit and YouTube data. Metricool's REST API can answer comments and DMs, but its MCP server cannot. 4. **Open source, self-hosted**: none is complete. Postiz is the most popular but has no inbox. BrightBean Studio has the most complete inbox tools but no X. OpenPost handles X comments and DMs but is seven months old. 5. **Session-based APIs** (they take your password or cookies) add only things the platforms forbid in writing: LinkedIn personal DMs, a way around X's reply limit, a way around Instagram's 24-hour DM window. 6. **Whatever you pick, test one X reply on a real account first.** X's documentation doesn't say whether a reply under your own post counts as "mentioning you", and that limit applies to every vendor. --- ## Who this is for Engineers who already use an Agent (Claude Code, Cursor or their own) and want it to take over social posting and replies too. You don't need to know any platform's API beforehand; you should know roughly what MCP is (the protocol an Agent uses to call external tools). **Seven terms first**: | Term | Meaning in this article | |---|---| | Official API | The interface a platform publishes for developers; the platform decides what it can do and what it costs | | Authorization token (OAuth token) | The key a third-party service receives after you click "Allow" on the platform's consent page. It can only do what you approved, and you can revoke it any time in the platform's settings | | Unified API | A service that wraps a dozen platforms' official APIs into one interface, so you integrate with it alone | | Inbox | Comments, mentions and DMs: the three kinds of messages where someone reaches out to you | | Session-based API | A service that logs in as you with your password or browser cookies, bypassing the official API | | Summoned | X's rule: a program may reply to a post only if its author mentioned or quoted you | | P / C / M / D | Table shorthand: post / read and answer comments / read and answer mentions / read and send DMs | --- ## The minimal causal chain: tools are only carriers Whichever tool you use, a call travels this chain: ```mermaid flowchart TD AG(["Your Agent
Claude Code etc."]) --> T["MCP tool or REST API"] T --> UNI["Unified service
holds the token you authorized"] UNI --> API["Platform's official API
the platform decides what it can do"] API --> ACC(["Your account"]) T -.-> SES["Session-based service
holds your password or cookies"] SES -.->|"logs in as you, forbidden by platform terms"| ACC YOU(["You
browser or phone app"]) -->|"login session"| ACC classDef ok fill:#d3f9d8,stroke:#2f9e44 classDef bad fill:#ffe3e3,stroke:#e03131 class UNI,API ok class SES bad ``` *Schematic: it shows call paths, not any vendor's internals. Solid lines are the official route; dashed lines are the route session-based APIs take.* Three consequences follow, and the rest of the article rests on them: 1. **A unified service can never do more than the platform's official API allows.** Tools compete on how much of the official capability they carry, and how reliably. 2. **Anything beyond the official API means taking the dashed route**, handing over your password or cookies. That route does unlock more, but platform terms forbid it in writing. 3. **A service calling the official API is not you logging in.** In a platform's security settings, "connected apps" and "devices / sessions" are two separate lists. A unified service calls the API from its own servers; that does not count as you logging in from another IP. --- ## 1. What the platforms give you: check the ceiling first | Platform | Post (P) | Read and answer comments (C) | DMs (D) | Requirements / catches | |---|---|---|---|---| | X | Yes | Read yes; **reply only to people who mentioned or quoted you** | Yes; DMs moved to encrypted X Chat can't be read by the API | Pay per use, no free tier; automatic AI replies need X's prior approval | | YouTube | Videos only | Yes; each reply costs 50 quota units, 10,000 units a day by default | The platform has no DMs | Whether uploads from unaudited projects are locked to private: two official pages disagree | | Instagram | Yes | Yes | Yes, but the other person must write first and you must answer within 24 hours | Needs a creator or business account (free to switch) | | Threads | Yes | Yes | The platform API has none | — | | Facebook | Pages only | Pages yes | Pages yes | Personal profiles never | | LinkedIn personal profile | Yes | **Cannot read**: the permission to read comments is for "select developers" only | **No**: the Messages API is partners only | Company Pages can read comments | | TikTok | An app you register yourself won't pass the audit, and an unaudited app can only post privately; you need an already-audited service | Business accounts | Business accounts; not in the EU, UK or Switzerland | The audit rules explicitly reject "tools that upload for your own or your team's accounts" | | Reddit | Yes | Yes | Yes | **New API applications close on 2026-10-31; the public Data API shuts down in March 2027** | | Bluesky / Mastodon | Yes | Yes | Yes | Free, no review; on Bluesky, reply only to people who mention you first | X is the only platform here that charges per call (official pricing page, 2026-10-09): | Operation | Unit price | |---|---| | Create a post | $0.015 | | Create a post containing a link | **$0.20** | | Read a post | $0.005 | | Read your own data (such as your own mentions) | $0.001 | | Read a DM / send a DM | $0.010 / $0.015 | Saving your first card gives $20 of credit, and reads are capped at 3 million posts a month. For 30 posts a month (10 with a link), 500 replies read and 50 answered, the X part comes to about $3–6, mostly from the link posts. --- ## 2. How I scored To compare like with like I fixed an **example setup**: one person with a personal and a brand account on each of 9 platforms, 17 accounts in total (two each on X, YouTube, Instagram, Threads, TikTok, Bluesky and Reddit; a personal profile plus a Company Page on LinkedIn; only the brand Page on Facebook, because personal profiles can't be connected). | Item | Max | How it's scored | |---|---|---| | Posting coverage | 10 | Accounts it can connect and post to / 17 × 10; text-only X counts as half an account | | Read and answer replies under your X posts | 4 | Weighted separately; X engagement depends on it most | | X DMs | 2 | | | Comments on the other 8 platforms | 8 | 1 point per platform | | Other DMs | 5 | 1 point each for Instagram, Facebook, TikTok, Bluesky, Reddit | | Mentions | 3 | 0.5 for each of 6 platforms | | MCP | 3 | Dedicated tools for comments and DMs 3; comments only 2; one generic execute tool 1 | | Webhooks for new messages | 2 | New comments and new DMs 2; one kind only 1 | | Several accounts per platform together | 1 | No point if extra workspaces are needed | Features limited to posts published through the tool, read-only, beta, or with self-contradicting docs get half a point. **These weights are mine**; cutting "replies under your X posts" from 4 points to 1 leaves the top three unchanged. --- ## 3. Developer-facing unified APIs (19 vendors) | Rank | Product | Score | Replies under your X posts | X DMs | Accounts (/17) | MCP has comment + DM tools | Monthly price for 17 accounts (second sort key) | |---|---|---|---|---|---|---|---| | 1 | Zernio (formerly Late) | **35.3** | Yes, including posts not made through it | Yes | 17 | Yes | $69 + X passed through at official rates | | 2 | SocialAPI.ai | 29.5 | Yes (your own X key) | Yes | 15 (X text only) | Yes | $29 + your own X costs | | 3 | Ayrshare | 29.3 | Yes (your own X key) | Yes | 17 | Yes (no mentions) | $299 ($249 yearly) + $0.09 per DM conversation | | 4 | OmniSocials | 23.8 | **No** | Yes | 15 (no Reddit) | Yes | Two workspaces, $20–24 | | 4 | Aidelly | 23.8 | Docs contradict | Partly | 15 | Yes | $29–39 | | 6 | Postproxy | 23.3 | No | No | 15 | Yes | $49 | | 7 | RelayAPI | 22.0 | Docs unclear | Docs unclear | 17 | Generic execute tool only | From $5 | | 8 | Outstand | 21.0 | Only posts made through it | No | 17 | Comments only | $19 + your own X key | | 9 | Upload-Post | 20.8 | Docs conflict | No | 15 (Reddit posting returns 503) | Yes | $16–24 | | 10 | bundle.social | 19.0 | Docs contradict themselves | No | 17 | Comments only | $90–100 | Ranks 11–19 are UniPost, Mallary, Postqued, PostFast, Blotato, Post for Me, Post Bridge, Unipile and Typefully: they either only post, or only see comments on posts published through them. The leaders platform by platform (P post, C comments, M mentions, D DMs; ½ partial, ? docs unclear): | Product | X | LinkedIn personal | LinkedIn Page | YouTube | Instagram | Facebook Page | Threads | TikTok | Bluesky | Reddit | |---|---|---|---|---|---|---|---|---|---|---| | Zernio | P C D M½ | P | P C M½ | P C | P C D M½ | P C D | P C | P C D½ | P C D | P C D | | SocialAPI.ai | P½ C D M½ | P | P C½ | P C | P C D M½ | P C D M | P C M½ | P | P C D M | — | | Ayrshare | P C D M½ | P C? | P C | P C½ | P C D M½ | P C D | P C? | P C | P C½ | P C¼ | | OmniSocials | P D | P | P C M | P C | P C D M | P C D M | P C M | P C | P | — | | Postproxy | P | P | P | P C | P C D | P C D | P C | P C | P C½ D | — | Easy things to miss: - **"Your own X key" is a different way of paying.** Zernio and bundle.social use their own X app and pass X's price through. Ayrshare (mandatory since 2026-03-31), SocialAPI.ai and Outstand make you create an app in X's developer console, add a card and pay X yourself. With your own key, reading your own mentions costs $0.001; Zernio passes reads through at $0.005 each. - **OmniSocials prices differently**: $10–12 a month per workspace with unlimited accounts, but only one account per platform per workspace; it pays for X posts without links itself. Its inbox documentation says outright that comments and mentions on X are not part of the inbox. - **Newer vendors often contradict their own docs**: Aidelly, bundle.social and Upload-Post all have features that the marketing page lists and the API docs do not. RelayAPI's documented MCP install command installs an unrelated maintainer's npm package of the same name, a supply-chain risk. SocialAPI.ai is a one-engineer personal company with no public changelog. - **Zernio has catches too**: every X read is off by default and needs a background switch that bills continuously (tested in section 7); the X comment list returns only the first page, reply threads are cached for 2 minutes, and the comment list can lag by up to 10 minutes; Threads has no new-comment webhook; TikTok DMs can be answered but not started. --- ## 4. Established social media suites (23 vendors) These have mature web inboxes. The question is whether a program can drive the inbox: | Product | How Claude Code connects | Posting | Comments read / reply | DMs read / reply | Monthly for the 17 accounts (monthly / yearly) | |---|---|---|---|---|---| | Vista Social | Official MCP | Publishes directly | Yes / yes | Yes / to be confirmed (the reply endpoint says "public reply") | $257 / $217 | | Eclincher | Official MCP | Publishes directly | Yes / yes | Yes / yes (X DMs unverified) | $149 / $134 (no Bluesky) | | Agorapulse | Official MCP (beta) | **Drafts only** | Yes / yes | Yes / yes | $282 / $247 (X needs a $69 add-on per account) | | Metricool | Posting via MCP, inbox via REST | Yes | REST yes (replies under X posts are not in the inbox) | REST yes (X DMs included) | $87 / $73 | | Statusbrew | MCP | Drafts only | Read only | Read only | $429 / $359 | | Hootsuite | MCP + REST | REST publishes, MCP saves drafts | MCP read-only, **and it blocks X, LinkedIn, YouTube and Reddit** | Same | From $199 (per seat, yearly) | | Sprout Social | REST (enabled by sales); MCP for ChatGPT only | Drafts only | API read-only | API read-only | About $399 | | SocialPilot | Official MCP | Publishes directly | No | No | $50 / $42.5 (cheapest posting-only option) | | Buffer / Publer / Zoho Social | MCP or REST | Yes | No (web only) | No | $59–98 | Later, Loomly, SocialBee and Sendible have neither a public API nor MCP and are out. Brandwatch, Khoros, Emplifi, Sociality.io and NapoleonCat need an enterprise contract for API access. > [!WARNING] > Vista Social and Metricool both say they bring **LinkedIn personal-profile comments** into the inbox. That conflicts with LinkedIn's official documentation, which gives the permission to read personal-post comments to "select developers" only. They may be approved partners, but I found no evidence, so I treat this as a vendor claim, unverified. --- ## 5. Open source, self-hosted, and new vendors in MCP directories | Project | State (2026-10-09) | Inbox | Inbox tools in MCP | Catch | |---|---|---|---|---| | Postiz | 36,914★, AGPL-3.0, latest v2.25.0 | **None**; its MCP docs say it cannot read or answer comments | None | 18 security advisories in 2026, several critical and exploitable without authentication | | Mixpost Pro | Closed source, $299 one-off | Yes (comments and mentions on FB, IG, Threads, X) | None; replies only in the web app | — | | BrightBean Studio | 2,421★, AGPL-3.0, free hosted version | Yes: comments, mentions, DMs | **Most complete**: list, draft and send are separate permissions | **No X** | | OpenPost | 659★, created March 2026 | Yes, including X comments and DMs | Comments only; DMs only in the HTTP API | Its hosted version says it "hasn't finished final live checks" | | Chatwoot | 37,646★ | DMs only; the X channel isn't merged yet | Community servers only | Doesn't post | In the official MCP Registry, Smithery, Glama and Anthropic's connector directory I also found a few vendors with inbox tools that appeared around September: Ignix (174 tools, about $121 a month for the 17 example accounts; its help page and landing page disagree on which platforms have DMs), PostLake (the cleanest inbox API, but X, Instagram, Facebook and Threads aren't open to the public yet), Pinlyx (DMs only) and SocialClaw (Instagram only). None covers as much as the leaders in section 3. --- ## 6. Session-based APIs: the extra capability is exactly the forbidden part Unipile, Linked API, Beeper, twitterapi.io and similar services ask you to log in to your account on their page or hand over your browser cookies. They add only four things over the official route: | Extra capability | What the platform says | |---|---| | LinkedIn personal DMs and personal-post comments | User Agreement 8.2 prohibits "bots or other unauthorized automated methods" | | Instagram DMs without the "they write first, 24 hours" limit | The Terms of Use prohibit unauthorized automated access | | X replies without the "summoned" limit, at about 1/10 of the official price | Automation rules: non-API automation may lead to permanent suspension | Unipile's own documentation admits some endpoints are non-public and that account restrictions are an "expected failure"; Beeper's warns that sending too many messages can get accounts suspended. **If the account matters to you, stay away from this category.** --- ## 7. One comment end to end (Zernio as the example) Take the top-ranked Zernio and follow what happens between someone replying under your X post and your reply going out. ```mermaid flowchart TD A(["Someone replies under your X post"]) --> B["1. The unified service pulls comments from X
the X inbox must be switched on; each read is billed at X's $0.005"] B --> C["2. The Agent lists comments through MCP
first posts with comments, then one post's comments"] C --> D["3. The Agent drafts a reply"] D --> E["4. You approve before it goes out"] E -->|"edit"| D E -->|"send"| F["5. Call the reply tool
X charges one post, $0.015"] F --> G["6. X checks whether you were summoned
if not, it refuses; to be tested"] ``` *Schematic: the steps come from Zernio's documentation and X's pricing page. Step 4 is my addition, a human approval that satisfies X's rule that automatic AI replies need prior approval. Only the existence of the tools in step 2 has been tested.* **The tested part** (2026-10-09, free): I created an API key in the Zernio dashboard and connected the MCP server to Claude Code with one command: ```bash claude mcp add --transport http -s user zernio https://mcp.zernio.com/mcp --header "Authorization: Bearer " ``` `claude mcp list` showed it connected, but **the tools did not appear in the current session; they only show up in a new one**. Rather than wait for a new session, I spoke MCP directly: first `initialize`, then list the tools: ```http POST https://mcp.zernio.com/mcp Authorization: Bearer Content-Type: application/json Accept: application/json, text/event-stream {"jsonrpc":"2.0","id":2,"method":"tools/list"} ``` What came back (a summary, not the raw response): | Observation | Detail | |---|---| | 52 tools listed directly | 13 posting tools (`posts_create`, `posts_cross_post`, `posts_publish_now` and others), plus accounts, scheduling and analytics | | Comment tools are in the list | `comments_list_inbox_comments`, `comments_get_inbox_post_comments`, `comments_reply_to_inbox_post` | | Mention tools are in the list, but narrow | `mentions_list_inbox_mentions` currently returns LinkedIn Company Pages only; `mentions_reply_to_mention` supports Instagram only | | **DM tools are not in the list** | You first call `search_tools`, then `call_tool`; the search finds `messages_list_inbox_conversations`, `messages_send_inbox_message` and `messages_create_inbox_conversation` | The documentation doesn't mention that last point: an Agent that only reads the tool list will conclude Zernio has no DM support. ### A run with a real account (evening of 2026-10-09) I connected my own X account to Zernio, published a test post, left a comment under it from the same account, and had the Agent reply to that comment: | Step | How | Result | |---|---|---| | Connect the X account | Authorize in the Zernio dashboard | The docs say a card is required before the first X account (X bills per call and Zernio passes it on); once connected, the account is healthy and can post | | Check before posting | `validate_post` | Free, no issues | | Publish a text-only post | `posts_create`, publish now | Success, about $0.015 | | Read the comments under it | `comments_get_inbox_post_comments` | **403**, `X_INBOX_NOT_ENABLED` | | Try search or lookup by ID instead | `twitter_engagement_search_tweets`, `twitter_engagement_get_tweet` | **403**, `X_ANALYTICS_NOT_ENABLED` | | Reply to a specific comment | REST `POST /v1/posts` with the comment's ID in `platformSpecificData.replyToTweetId` | Success, about $0.015; checked through X's public embed endpoint, the reply is attached to that comment | Four things came out of this run: - **Every X read on Zernio is off by default.** An X account has two switches: inbox (syncing replies under your posts and polling DMs) and analytics (periodic reads of post data). With both off you can only post and delete; reading comments, searching and looking up a post by ID all return 403. Once switched on, Zernio calls X's API in the background and each call is billed at X's price, and **the documentation doesn't say how often it polls**. If you want to watch X comments through it long term, budget for that ongoing cost; the dashboard lets you set a monthly X spending cap. - **The MCP posting tool can't target a specific post to reply to.** `posts_create` has no such parameter; use the REST API, or the full tool found through `search_tools`. - **Replying to my own comment was not blocked.** Whether replying to someone else's comment counts as "summoned" was not tested in this run. - After both posts, the usage endpoint still showed $0, possibly a delay; the end-of-month invoice is the final word. **The same flow carries over to other services**, because an inbox has these four steps and only the names differ: | Step | Zernio | OmniSocials | Ayrshare | |---|---|---|---| | List what's pending | `comments_list_inbox_comments` | `get_next_unanswered` (hands you the next unanswered item) | `get_comments` | | Read the context | `comments_get_inbox_post_comments` | `get_inbox_conversation` (includes the original post) | `get_comments` | | Reply | `comments_reply_to_inbox_post` (supports an idempotency key, so a retry never posts twice) | `reply_to_inbox` | `reply_comment` | | Send a DM | `messages_send_inbox_message` | `reply_to_inbox` | `send_message` | If you build it yourself, keep three things: record what you have handled (polling will fetch the same comment twice); send replies with an idempotency key; keep a human approval step before anything goes out. --- ## 8. Will it get you banned? - **A unified service calling the official API is not you logging in from elsewhere.** Ban risk comes from behaviour, not from the tool. X forbids several accounts posting the same or very similar content (translations into another language are allowed), automatic replies to people who didn't mention you, automatic likes and bulk following. LinkedIn rejects duplicate posts outright, and Meta treats repeated content as a spam signal. - **Multiple IPs affect your own logins.** Logging in through proxy exits that keep changing country makes X ask for verification or lock the account temporarily, Facebook and Instagram show security checks, and LinkedIn restricts the account temporarily. The platforms don't publish the details; this comes from third-party sources. Sticking to one exit and turning on two-factor authentication avoids most of the trouble. - **One person, several accounts, has limits**: X allows up to 10 accounts per person, each with a distinct purpose, and a violation may mean keeping one and losing the rest. LinkedIn's User Agreement allows one account per person, so the brand needs a Company Page. Facebook also allows one personal account; the brand uses a Page. - **When you drop a service, revoke its access in the platform's connected-apps settings.** One user reported that disconnecting inside a service did not actually revoke its access. --- ## 9. How the pricing works (second priority) | Pricing model | Example | For the 17 accounts | |---|---|---| | Graduated per connected account | Zernio: first 2 free, accounts 3–10 at $6 each, 11–100 at $3 | $69 | | Per workspace, unlimited accounts, one per platform | OmniSocials: $10–12 per workspace | Two workspaces (personal, brand): $20–24 | | Per profile tier | Ayrshare: Launch | $249–299 | | Per plan or per seat | Vista Social, Hootsuite, Sprout | $200–400 | Almost every vendor charges for X separately: passed through at X's price, paid by you with your own key, or converted into credits. SocialPilot and Upload-Post include it in the plan; the cost at Upload-Post is that links are stripped from posts by default. --- ## 10. Which kind to pick when ```mermaid flowchart LR %% layout: tree Q(["What matters most to you?"]) Q --> A["Reading and answering replies under your X posts
→ Zernio; or SocialAPI.ai, Ayrshare (your own X key)
test the summoned rule on a real account first"] Q --> B["A web inbox, team workflow, an established vendor
→ Vista Social (expensive), Eclincher
Agorapulse's MCP only saves drafts"] Q --> C["Many accounts, mostly IG, Threads, YouTube, LinkedIn Pages
X only for posting → OmniSocials (per-workspace pricing)"] Q --> D["Posting only, no replies
→ SocialPilot, Buffer, Post for Me"] Q --> E["Full control in your own hands
→ open source: BrightBean (no X), OpenPost (very new)"] Q --> F["Programmatic LinkedIn personal DMs are a must
→ no compliant option; handle them by hand"] classDef q fill:#fff3bf,stroke:#f08c00 classDef pick fill:#d3f9d8,stroke:#2f9e44 class Q q class A,B,C,D,E pick ``` *Schematic: compiled from the documentation of 2026-10-09, not a tested ranking.* --- ## Facts, inference and what isn't verified - **Tested**: Zernio's read-only REST endpoints return 200; after the MCP handshake it lists 52 tools, and the DM tools have to be found through `search_tools`; after connecting it to Claude Code the tools appear only in a new session; with a real account, publishing and replying to a specific comment work, and every X read returns 403 while both switches are off. - **Official documentation**: each platform's limits, X's prices and "summoned" rule, every vendor's prices, the MCP tool names. - **My inference**: Vista Social and Metricool may read LinkedIn personal comments through partner permissions; the scoring weights. - **Not verified, to be tested**: - whether a reply under your own post counts as "summoned" (replying to your own comment works); - how often Zernio calls X once the inbox or analytics switch is on, and what that costs a month; - whether Vista Social can answer DMs; - Eclincher's X DMs; - Ignix's and Aidelly's contradictory docs; - whether YouTube uploads from unaudited projects are locked to private. --- **The platform sets the ceiling; tools compete on how much of it they carry. The extra bit a session-based API offers is exactly the part the platform won't let you use.** --- ## Sources (checked 2026-10-09) - X: [pricing](https://docs.x.com/x-api/getting-started/pricing) · [changelog](https://docs.x.com/changelog.md) · [creating posts and the reply limit](https://docs.x.com/x-api/posts/manage-tweets/introduction.md) · [platform manipulation and multiple accounts policy (archived)](https://archive.ph/Ga9ef) - LinkedIn: [User Agreement](https://www.linkedin.com/legal/user-agreement) · [Comments API](https://learn.microsoft.com/en-us/linkedin/marketing/community-management/shares/comments-api) · [Messages API](https://learn.microsoft.com/en-us/linkedin/shared/integrations/communications/messages) · [prohibited software](https://www.linkedin.com/help/linkedin/answer/a1341387) - Meta: [Instagram comment moderation](https://developers.facebook.com/docs/instagram-platform/comment-moderation) · [Instagram Messaging API](https://developers.facebook.com/docs/instagram-platform/instagram-api-with-instagram-login/messaging-api) · [Threads API](https://developers.facebook.com/docs/threads/overview) · [Meta Terms of Service](https://www.facebook.com/terms/) - YouTube: [quota costs](https://developers.google.com/youtube/v3/determine_quota_cost) · [videos.insert](https://developers.google.com/youtube/v3/docs/videos/insert) - TikTok: [content sharing guidelines](https://developers.tiktok.com/doc/content-sharing-guidelines) - Reddit: [Responsible Builder Policy](https://support.reddithelp.com/hc/en-us/articles/42728983564564-Responsible-Builder-Policy) · [Data API shutdown timeline (PPC Land)](https://ppc.land/reddit-to-halt-public-data-api-access-in-march-2027-as-14-000-apps-register/) - Bluesky: [bot guidelines](https://docs.bsky.app/docs/starter-templates/bots) - Zernio: [pricing](https://zernio.com/pricing.md) · [llms.txt](https://zernio.com/llms.txt) · [X inbox](https://docs.zernio.com/platforms/twitter/inbox) · [terms](https://zernio.com/tos) - OmniSocials: [pricing](https://omnisocials.com/pricing) · [inbox](https://docs.omnisocials.com/inbox) · [X](https://docs.omnisocials.com/platforms/x) - Ayrshare: [pricing](https://www.ayrshare.com/pricing/) · [Comments API](https://www.ayrshare.com/docs/apis/comments/overview) - SocialAPI.ai: [pricing](https://social-api.ai/pricing) · [docs](https://social-api.ai/llms-full.txt) - Suites such as Vista Social, Hootsuite, Agorapulse and Metricool: [Hootsuite MCP](https://www.hootsuite.com/integrations/mcp) · [Metricool MCP](https://help.metricool.com/how-to-connect-metricools-mcp-with-claude-0l84v.md) - Open source: [Postiz](https://github.com/gitroomhq/postiz-app) · [Postiz MCP](https://docs.postiz.com/mcp/introduction) · [BrightBean Studio](https://github.com/brightbeanxyz/brightbean-studio) · [OpenPost](https://github.com/getopenpost/openpost) - MCP: [security best practices](https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices) --- title: "Agents That Check Their Own Work and Tidy Their Own Memory: Outcomes and Dreaming from the Docs, Plus a Hands-On Test of Claude Code /goal" description: "Outcomes has an independent grader, blind to the writer's reasoning, check the work criterion by criterion and send it back until it passes; Dreaming reads an old memory store and past sessions between runs and writes a separate, reorganized store while leaving the original untouched. Both are Claude Managed Agents APIs on the Claude Developer Platform, and Claude Code does not have them. Following the official docs, this article explains why both are needed and how they work, then identifies their closest relatives in Claude Code (/goal and the unreleased auto-dream), runs /goal 12 times on the same task, and translates the design into two contracts you can rebuild on any Agent stack." url: "https://zhuoqidev.com/en/posts/managed-agents-outcomes-dreaming/" language: "en" author: "Liu ZhuoQi" date: "2026-10-06" updated: "2026-10-09" series: "Agent Architecture Deep Dives" tags: ["Claude","Claude Code","Managed Agents","AI Agent","Agent Architecture","Outcomes","Dreaming","Agent Memory","LLM-as-Judge","Prompt Injection"] translation: "https://zhuoqidev.com/posts/managed-agents-outcomes-dreaming/index.md" --- # Agents That Check Their Own Work and Tidy Their Own Memory: Outcomes and Dreaming from the Docs, Plus a Hands-On Test of Claude Code /goal > **TL;DR** Agents are good at handing in work that looks finished, and good at letting their memory drift into a mess. Anthropic gave Claude Managed Agents one mechanism for each problem: Outcomes brings in an independent grader to check the work inside a session, and Dreaming reorganizes memory between sessions. Yet in the Claude Code you use every day, neither name appears. This article first takes both mechanisms apart from the docs, then shows what Claude Code actually offers, how it performed in a hands-on test, and how to rebuild both in your own Agent. **Version scope**: this article follows the official documentation as checked on 2026-10-06, with checks and a hands-on test on Claude Code 2.1.294 added on 2026-10-09. Outcomes and memory stores are in public beta; Dreaming is a research preview that requires requesting access. Both were announced at Code with Claude on 2026-05-06. Fields and limits can change during beta, so check the current docs before integrating. > [!WARNING] > **First, what was tested and what was only read.** Outcomes and Dreaming are Managed Agents APIs on the Claude Developer Platform, paid from an API balance, and **Claude Code does not have either feature**. I did not run either one: Outcomes needs prepaid API credits, which I chose not to buy for this article, and the Dreaming research preview is not enabled on my account. So the requests, events, and three-round revision in the first half all come from the official docs and cookbook. What I did test is their closest relatives in Claude Code: I ran `/goal` 12 times on the same task and scored every result with a hidden test that neither the writer nor the evaluator could see, and I checked why auto-dream has never run on my account. See "[What you can see in Claude Code](#what-you-can-see-in-claude-code)" and "[Hands-on: how much does the /goal evaluator actually do?](#hands-on-how-much-does-the-goal-evaluator-actually-do)". > [!NOTE] > **The single most important caveat: how well Outcomes works depends almost entirely on how you write the rubric.** A vague rubric makes the grader approve everything, the loop ends after one round, and you pay for a check that never happened. --- ## Who this is for The default reader is an engineer who knows LLMs, context windows, and basic tool calling, has built or configured an Agent, usually works in Python, TypeScript, Go, or similar, but has not used Claude Managed Agents. You **do not** need to know: - the Managed Agents API layout or SDK calls; - the terms "grader," "rubric," or "memory store"; each is explained on first use; - any Anthropic internals. Anthropic has not published the internal code for either mechanism, so every "how it is implemented" passage below is marked as documented behavior, my inference, or illustrative pseudocode. --- ## Three plain sentences to remember 1. **Outcomes answers "is this delivery good enough?"** You write down what passing looks like. The platform starts a separate grader that cannot see the working Agent's reasoning and checks the output against your criteria one by one. If anything fails, the itemized feedback goes back for revision, until it passes or the round limit runs out. 2. **Dreaming answers "will the next run go better?"** Outside any session, it reads an old memory store plus up to 100 past sessions and produces a **new**, reorganized store: duplicates merged, stale entries replaced with the latest value, and patterns no single session could see added. The old store is not changed at all. 3. **Both sit on the same foundation: the memory store.** It is a set of text files mounted into the Agent's sandbox. The Agent uses its ordinary file tools on it, and every write leaves an immutable version. | | Memory store | Dreaming | Outcomes | |---|---|---|---| | When it runs | During a session, as the Agent reads and writes | Between sessions, offline and asynchronous | Within one session | | Problem solved | Sessions forget everything by default | Incremental writes pile up into a mess | "Looks done" is not "done right" | | Input | The Agent's file reads and writes | One existing store + 1 to 100 session transcripts | Task description + rubric + max rounds | | Output | Memory files and their version history | A new memory store | One of five results + the grader's itemized explanation | | Status (2026-10-06) | Public beta | Research preview, access required | Public beta | --- ## Deriving both mechanisms from constraints Set the API aside and start from constraints no Agent can avoid. **Constraint 1: context is finite and cleared when a session ends.** A Managed Agents session starts with a fresh context by default, and whatever it built up is gone when it ends. To keep user preferences, project conventions, and past mistakes, you have to write them outside the context. That is the memory store. **Constraint 2: incremental writes make memory messier over time.** Each session only sees what is in front of it, so it appends "run migrations before deploying," and a later session appends something nearly identical; a setting changes and nobody deletes the old entry. The official docs say a store accumulates duplicates, contradictions, and stale entries over many sessions. Memory does not tidy itself, so **something that can see many sessions at once has to reorganize it periodically**. That is Dreaming. **Constraint 3: a model grading itself is unreliable.** A writer checking its own output inside its own context sees the reasoning it just produced. It declares success when it believes it succeeded, does not reopen a link it already cited, and does not notice that the quote it remembers differs from the page by a few words. Reliable checking requires **a checker that cannot see the writer's reasoning and holds only the criteria and the output**. That is the independent grader in Outcomes. **Constraint 4: the loop needs an exit.** A grader can always find something to criticize, so revision needs a cap, and a self-contradictory rubric must be able to end in failure instead of spinning. Chained together, the minimal causal chain is: ```mermaid flowchart LR A["Finite context
cleared at session end"] --> B["Write memory outside the context
(memory store)"] B --> C["Incremental writes
duplicates, contradictions, stale entries"] C --> D["Cross-session reorganization
(Dreaming)"] E["Self-grading is biased"] --> F["Context-isolated grader
+ checkable criteria"] F --> G["Capped revision loop
(Outcomes)"] ``` *Figure: a conceptual sketch of causes, not Anthropic's internal architecture.* What you must keep and what you can swap: | Must keep (the design) | Can swap (the choices) | |---|---| | The grader's context is isolated from the writer | Which model grades, and whether it matches the writer | | Criteria are written as items that can be checked one by one | Whether the rubric is markdown or JSON | | Revision is capped, and an inapplicable rubric can end in failure | Whether the cap is 3 or 5 | | Reorganizing memory never edits the original, and results are reviewed before use | Whether the original lives at Anthropic, in Git, or in a database | | Every memory write leaves a version | Whether versions come from a managed service or from Git | --- ## The only five things you need to know | Plain name | Official name | What it is | Layer | |---|---|---|---| | Session | session | One complete run of an Agent, with its own sandbox and event stream | Runtime | | Memory store | memory store | A set of text files mounted as a sandbox directory, kept across sessions | Managed storage | | Rubric | rubric | A markdown document stating, item by item, what passing looks like and how to verify it | Your input | | Grader | grader | Another Agent the platform starts automatically, with its own context, checking only against the rubric | Runtime | | Dream | dream | An asynchronous job that reads old memory and past sessions and writes a new store | Managed job | The prose below uses the plain names on the left; official names appear only in request examples and sources. --- ## The foundation: how a memory store works **Where it sits**: when you create a session you attach the store as a resource. The platform mounts it as a directory under `/mnt/memory/` in the sandbox and automatically adds a note to the system prompt saying what the directory is called, what it holds, and whether it is writable. The Agent reads and writes it with the same tools it uses for every other file; there is no special "memory tool." Documented rules: - **Every change creates an immutable version**, so you can audit, roll back to a point in time, or redact sensitive content from history. - **Limits**: 100 kB per memory (about 25k tokens), 10,000 memories per store, 8 stores per session. The docs recommend many small files over a few large ones. - **Stores can only be attached at session creation**, not added or removed while a session runs. - **Access is read-write or read-only**, defaulting to read-write and enforced at the filesystem level. - Memory store endpoints use the beta header `agent-memory-2026-07-22`, which **must not** be combined with the session header `managed-agents-2026-04-01`; sending both returns 400. Attaching a store to a session still uses the session header. The complete minimal request to create a session with a store attached (from the official docs): ```bash curl -s https://api.anthropic.com/v1/sessions \ -H "x-api-key: $ANTHROPIC_API_KEY" \ -H "anthropic-version: 2023-06-01" \ -H "anthropic-beta: managed-agents-2026-04-01" \ -H "content-type: application/json" \ --data @- < [!CAUTION] > **The default read-write access is an injection channel.** The official docs say so directly: if the Agent processes untrusted input (user-supplied prompts, fetched web pages, third-party tool output), a successful prompt injection can write malicious content into the store, and **later sessions read that content as trusted memory**. Attach reference material, shared lookups, and any store the Agent does not need to modify as read-only. --- ## Outcomes: a writer-and-grader loop inside one session ### What happens in one loop In product terms: 1. Instead of a normal user message, you send a "define outcome" event with three things: a **task description** for the writer, a **rubric** for the grader, and a **maximum number of rounds**. The Agent starts on receipt; no other message is needed. 2. After the writer finishes a round, the platform starts a **brand-new grader**. It uses the same model and the same tools as the writer, but its context is separate: it cannot see the writer's reasoning or know what shortcuts were taken, and holds only the rubric and the output. 3. The grader returns a verdict per criterion. If everything passes, the session ends. If anything fails, the grader's explanation goes back to the writer verbatim; after the revision, the platform starts another new grader that checks everything again from scratch. 4. If the cap is reached without passing, the writer takes one final wrap-up turn with no further grading, and the session ends. ```mermaid sequenceDiagram participant U as Your program participant P as Platform participant W as Writer (the session's Agent) participant G as Grader (new each round) U->>P: Define outcome: task + rubric + max rounds P->>W: Task description, start work loop Each round W->>W: Work#59; write output to /mnt/session/outputs/ P->>G: Start a new grader with only the rubric and the output G->>G: Check each criterion (may use all tools to verify) alt Everything passes G-->>P: satisfied P-->>U: Session goes idle else Something fails G-->>P: needs_revision + itemized explanation P->>W: Explanation passed back verbatim end end Note over P,G: Cap reached without passing: max_iterations_reached,
writer takes one wrap-up turn, no further grading ``` *Figure: a simplified sequence drawn from the documented behavior; the grader's internal reasoning is not exposed.* ### The complete request and events The define-outcome event (from the official docs; `max_iterations` is optional, default 3, maximum 20): ```json { "events": [ { "type": "user.define_outcome", "description": "Build a DCF model for Costco in .xlsx", "rubric": { "type": "text", "content": "# DCF Model Rubric\n..." }, "max_iterations": 5 } ] } ``` This goes to `POST /v1/sessions/{session_id}/events`. You can also upload the rubric through the Files API first and pass `{"type": "file", "file_id": "..."}`, which makes it reusable across sessions and reviewable like code. Or put this event in `initial_events` when creating the session to create it and start work in one call. The event stream then shows three kinds of grading events: ```json { "type": "span.outcome_evaluation_start", "outcome_id": "outc_01a...", "iteration": 0 } { "type": "span.outcome_evaluation_ongoing", "outcome_id": "outc_01a...", "iteration": 0 } { "type": "span.outcome_evaluation_end", "outcome_id": "outc_01a...", "result": "satisfied", "explanation": "All 12 criteria met: revenue projections use 5 years of historical data, ...", "iteration": 0, "usage": { "input_tokens": 2400, "output_tokens": 350, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 1800 } } ``` *For readability, `id`, `processed_at`, and similar fields are removed from the three events above; the values are the examples from the official docs.* `iteration` counts from 0: 0 is the first grading, 1 is the grading after the first revision. The `ongoing` event is only a heartbeat. **You cannot see what the grader is thinking**, only its final `explanation`. The `result` on the end event decides what happens next: | Result | Next | |---|---| | `satisfied` | The session goes idle | | `needs_revision` | The writer starts a new round | | `max_iterations_reached` | The writer takes one wrap-up turn with no further grading, then goes idle | | `failed` | The session goes idle. Returned when the rubric does not apply to the deliverables, for example when the description and rubric contradict each other | | `interrupted` | You interrupted the session (with `user.interrupt`) while the outcome was active | Only one outcome runs at a time, but you can chain them by sending another define-outcome event after the previous one ends. The session stays conversational afterward and keeps its history. Deliverables are written to `/mnt/session/outputs/` in the sandbox; list and download them through the Files API by session ID. ### The three-round revision from Anthropic's cookbook The following run was performed and published by Anthropic in its cookbook; I did not run it. The writer drafts a one-page brief on the unit economics of US public DC fast charging, citing at most 6 sources, each with a verbatim quote. The rubric has two parts: - **A 7-item coverage checklist**, each item more specific than the task description. For example, the "named operator" item requires the GAAP net income or net loss from a public charging operator's most recent 10-K or 10-Q, **and the citation must be the SEC filing itself on sec.gov, not a press release, earnings-call recap, or news article**. - **A 3-step citation check**: every URL must open directly with the web-fetch tool (no mirrors, reposts, or search snippets as substitutes); the quoted string must be found on the page; and the quote must actually support the claim it is attached to. What the grader actually reported over three rounds: | Round | Grader verdict | Problem caught | What the writer changed | |---|---|---|---| | Round 0 | Coverage 5/7 | Demand charges described only qualitatively, with no $/kW figure; EVgo's net loss cited to a third-party news site | Added $20/kW; went to sec.gov for EVgo's FY2024 figure | | Round 1 | Coverage 6/7, citations 6/6 | The new citation is **8-K Exhibit 99.1**, an earnings press release filed with the SEC, not a 10-K | Found the actual 10-K on EDGAR | | Round 2 | `satisfied`, 7/7 | None | 3 rounds in total, 12 min 56 s | Round 1 is the one to study. Exhibit 99.1 also lives on sec.gov, so a grader that only checked "is the link on sec.gov" would have passed it. It was caught because the rubric said "must be a 10-K or 10-Q, not a press release," which forced the grader to open the document and confirm what it actually was. ### Writing a rubric that works The cookbook lists several principles; here they are regrouped in my own words into four: | Principle | Bad | Good | |---|---|---| | **Make the grader produce evidence** | "Check that the brief covers demand charges": the grader skims, sees a paragraph, ticks the box, and never opens a source | "Find the demand-charges section and confirm it gives a $/kW figure or a % of operating cost" | | **Be more specific than the task** | The task says "operator economics," and so does the rubric | Pin down the metric (GAAP net loss), the document type (10-K / 10-Q), and the source domain | | **Describe the goal, not the steps** | Require a specific command; when it is unavailable, the check silently never runs | Define what counts as proof; the grader has the writer's full toolset and will find a way | | **Block the writer's shortcuts and fix the feedback format** | Without "no mirrors," a dead link gets swapped for a scraped copy and passes | State the forbidden substitutes; require a one-line score first, then one bullet per failure saying what is wrong and what to do, plus a list of things not to flag (style nits, out-of-scope issues) | If you have no rubric yet, the docs suggest giving Claude a known-good example of the deliverable, asking it to analyze what makes it good, and turning that analysis into criteria. This works better than starting from a blank page. > [!NOTE] > **Why not just put the rubric in the writer's system prompt?** It helps the writer aim, but the writer is still grading itself. It declares success when it believes it succeeded, does not refetch a link it already cited, and does not notice a remembered quote that differs from the page. The independent grader starts with only the rubric and the output, and the platform does not let the loop continue until it has a verdict on every criterion. No single prompt gives you that separation. ### When to use it, and when not to **Good fit**: - Tasks where "good" can be written as checkable items: complete coverage, accurate detail, verifiable citations; - Subjective quality also works, such as matching a brand voice, as long as the rubric pins the voice down to something checkable; - Deliverables are files the grader can open and verify with tools. **Avoid, or think twice**: - **You cannot write checkable criteria.** With a vague rubric the grader approves everything; you pay for grader tokens and check nothing. - **Every run hits the cap, and the grader keeps flagging the same kind of issue.** The writer cannot act on the feedback, for example because it lacks a tool, a permission, or the data. More rounds only pay for a loop that does not converge. - **Latency matters.** Every round re-grades the whole artifact; the example took 13 minutes for 3 rounds. - **The grader and writer may share blind spots.** This is my inference, not something the docs say: by default the grader uses the same model and tools as the writer, so what the writer cannot judge, the grader may not judge either. Context isolation fixes "being led by the writer's reasoning," not "neither side knows." High-stakes conclusions still need a human or deterministic checks such as tests and validation scripts. The official numbers: up to 10 points higher task success than a standard prompting loop, with the largest gains on the hardest problems; +8.4% task success on docx generation and +10.1% on pptx. **These are internal vendor benchmarks and deserve skepticism**; for your own task, run a comparison on your own samples. --- ## Dreaming: reorganizing memory between sessions ### What happens in one dream A dream is an **asynchronous job** with two inputs: - one **existing memory store**: the thing to verify, deduplicate, and reorganize; - **1 to 100 sessions**: complete past transcripts to mine for patterns to fold into the new memory. It produces **another memory store**, separate from the input. Once the job is running, the platform first clones the input store, and the new store's ID then appears in the job's `outputs[]`. The docs are explicit: **a dream never deletes or modifies its inputs.** If you have only transcripts and no existing memory, create an empty store first and pass it as the input. ### The complete request and response Dream endpoints need two beta headers; the session header alone is not enough. The complete minimal request (from the official docs): ```bash curl -s https://api.anthropic.com/v1/dreams \ -H "x-api-key: $ANTHROPIC_API_KEY" \ -H "anthropic-version: 2023-06-01" \ -H "anthropic-beta: managed-agents-2026-04-01,dreaming-2026-04-21" \ -H "content-type: application/json" \ --data @- <` sets a completion condition for the current session. After every turn, Claude Code sends **the condition and the whole conversation so far** to a small model (Haiku by default), which answers "not yet met", "met", or "impossible" with a one-line reason; if the condition is not met, the reason goes back to Claude and it keeps working. The official docs describe it as a thin wrapper around "a session-scoped, prompt-based Stop hook." | | Outcomes (Managed Agents) | `/goal` (Claude Code) | |---|---|---| | Who grades | A fresh grader started by the platform each round, the same model as the writer by default | The configured small model, Haiku by default | | What the grader sees | Only the rubric and the deliverable, **not the writer's reasoning** | The condition plus **the whole conversation**: what the writer said and what it ran | | Can the grader check for itself | Yes, it has all of the writer's tools and can open files and run code | **No**, it calls no tools and can only judge what already appears in the conversation | | Writer and criteria | The task description goes to the writer and the rubric to the grader, written separately | The condition itself is the writer's instruction; the writer sees all of it | | Round limit | `max_iterations`, default 3, maximum 20, enforced by the platform | No hard limit; "at most N turns" can only be written into the condition for the evaluator to judge. The hard stops are `/goal clear` or `--max-budget-usd` on `claude -p` | | Possible results | Satisfied / needs revision / limit reached / rubric does not apply / interrupted | Met / not yet met / impossible, plus automatic clearing on errors and a stop after several turns without tool use | | Grader cost | Billed at the grading model's standard price, plus session runtime | Runs on the small model; the docs call it usually negligible | In one sentence: **the Outcomes grader is a second person who can go and check; the `/goal` evaluator is someone who only reads the meeting notes.** Earlier I argued that the key to Outcomes is isolating the grader's context from the writer's. `/goal` has exactly no such isolation: whatever the writer ran and said in the conversation is what the evaluator judges. So with `/goal`, write the condition as something the conversation can prove, such as "`npm test` exits 0 and its summary line shows 0 failed," and require the writer to actually run that evidence. ### The relative of Dreaming: auto-dream Claude Code's auto memory (Claude writing notes for itself in a memory directory) also has a background consolidation feature, called auto-dream in the program: once enough hours have passed since the last pass and enough new sessions have piled up, it sends a sub-agent in the background to read recent session transcripts and merge or correct the files in the memory directory. | | Dreaming (Managed Agents) | auto-dream (Claude Code) | |---|---|---| | Trigger | You create a job through the API | Automatic. The program's default threshold is at least 24 hours and at least 5 new sessions since the last pass; the remote configuration my account receives says 24 hours and 3 sessions | | Input | One memory store plus 1 to 100 sessions | This project's memory directory plus session transcripts since the last pass | | Output | **A separate** new memory store; the original is untouched and reviewed before use | **Edits** the files in the memory directory directly | | Official docs | Yes, as a research preview that requires access | None. The `autoDreamEnabled` setting is not in the official settings reference; it appears only in the setting descriptions built into the program | | Who can use it | Accounts granted access | Decided by a server-side rollout flag | *The auto-dream thresholds and behavior come from my reading of the 2.1.294 program and configuration. They are not an official commitment and can change at any time.* On my machine, neither path works: - **API**: calling `GET /v1/dreams` with my personal key returns 404, the same as a path that does not exist at all, while the same key gets 200 from the session and memory store list endpoints. Managed Agents works; the Dreaming research preview is not enabled on this account. - **Claude Code**: the rollout flag cached in `~/.claude.json`, `tengu_onyx_plover`, is `{"enabled": false, "minHours": 24, "minSessions": 3, "remoteEnabled": false}`, and nowhere in `~/.claude` or the memory directory is there a `.consolidate-lock` file or any consolidation state that auto-dream leaves behind when it runs, so it has never run. Setting `"autoDreamEnabled": true` does not help: the program checks the rollout flag first and, while the flag is off, never reads that setting. [Issue #86209](https://github.com/anthropics/claude-code/issues/86209) in the official repository reports exactly this behavior; it was later closed automatically for inactivity and not fixed. ### Three things named Dreaming Search for "Claude Dreaming" and you will find three things with the same name and different origins and uses: | | Claude Code auto-dream | Managed Agents Dreaming | OpenClaw Dreaming | |---|---|---|---| | Built by | Anthropic | Anthropic | The open-source OpenClaw project, unrelated to Anthropic | | For whom | Claude Code users, released gradually behind a rollout flag | Teams building their own Agents on the Claude Developer Platform | People who run OpenClaw themselves | | Trigger | Automatic, in the background once enough time and sessions have passed | Create a job through `/v1/dreams` | A scheduled job; the current docs say it is on by default | | Edits the original? | Edits the files in the memory directory directly | Writes a separate new store; the original is untouched | Rewrites `MEMORY.md`, saving the previous version in plugin state first and writing a summary of additions, merges, and supersessions to `DREAMS.md` for people to read | | Injection defense | No public description found | The docs do not discuss the dream job itself (see "Risk" below) | Removes candidates labeled untrusted or system-generated before building the consolidation prompt | | Official docs | None | Yes | Yes | Timeline: - **2026-03-24**: users already saw "Auto-dream: on · last ran 1d ago" in `/memory` on Claude Code v2.1.81 ([issue #38461](https://github.com/anthropics/claude-code/issues/38461)), so it was already out to a small group. - **2026-03-31**: Claude Code v2.1.88 on npm shipped with a source map, exposing the full source. As retold by analyses such as Layer5's and Soma's, its `autoDream` is the first column above: a sub-agent sent off in the background that runs only after 24 hours, 5 sessions, and acquiring a lock, works in four steps (orient, gather, consolidate, prune), and gets a read-only shell. - **2026-04-05**: the first commit of the Dreaming doc in the OpenClaw repository. The timing is close, but I have no evidence the two are related. - **2026-05-06**: Anthropic announces the Managed Agents Dreaming research preview. - **2026-10-09**: the official Claude Code changelog does not mention dream anywhere. So **not seeing Dreaming in Claude Code is not a missed option; the account simply is not in the rollout.** The only review loop you can see is `/goal`, and it differs from Outcomes in exactly the most important respect. --- ## Hands-on: how much does the /goal evaluator actually do? I did not run Outcomes this time (see the top of the article for why), so this section tests its relative in Claude Code, `/goal`. The question: **does adding an evaluator after every turn make the output better, and if so, is it the evaluator or simply writing the criteria down?** ### Setup - **Task**: write a Python function `parse_duration` that converts durations such as "1h30m" and "45s" into seconds, with pytest tests. The writer's instructions are one sentence: ```text Write a Python module `duration.py` that exposes `parse_duration(text: str) -> int`, converting compact duration strings such as "1h30m" or "45s" into a whole number of seconds. Invalid input must raise an exception. Also write a pytest file `test_duration.py` for it. Put both files in the current directory. ``` - **Strict criteria**: 23 examples in four groups, plus "the tests cover every rule and pass." 7 values (including "one space between components is allowed" and "leading and trailing whitespace is ignored"); 13 strings that must raise `ValueError` (empty, number only, unit only, unknown unit, uppercase "1H", decimals, signs, out-of-order units "30m1h", repeated units "1h1h", a space between number and unit, two spaces); 3 types that must raise `TypeError` (`None`, `90`, `b"1h"`). - **Loose criteria**: only "common formats convert, invalid input raises an exception, the tests pass." - **Hidden test**: a scoring script built from the 23 strict examples, seen by neither the writer nor the evaluator, applied to every run's output after all runs finished. I checked the script first: a reference implementation scores 23/23 and a deliberately sloppy one 13/23. - **Groups**: four groups, three runs each, all with Claude Sonnet 5.5 as the writer. Each run used `claude -p` in an empty directory without loading my user-level settings or memory. | Group | What the writer gets | Evaluator | |---|---|---| | A one sentence | The sentence above | None | | B rules in the prompt | The sentence plus the full strict criteria | None | | C `/goal` + strict condition | Exactly the same text as B, prefixed with `/goal` | Haiku, reading the conversation after every turn | | D `/goal` + loose condition | The sentence plus the loose criteria, prefixed with `/goal` | Same | The exact conditions for C and D: ```text C: /goal Write a Python module `duration.py` that exposes `parse_duration(text: str) -> int`, converting compact duration strings such as "1h30m" or "45s" into a whole number of seconds, plus a pytest file `test_duration.py`, both in the current directory. The goal is met only when all of the following hold, each shown in this conversation by actually running the examples and pytest: (1) "1h30m" -> 5400, "45s" -> 45, "2d" -> 172800, "0s" -> 0, "1d2h3m4s" -> 93784, units d=86400 h=3600 m=60 s=1, return type int; (2) one space between components is allowed ("1h 30m" -> 5400) and leading/trailing whitespace is ignored (" 45s " -> 45); (3) each of these raises ValueError: "", " ", "90", "h", "5w", "1H", "1.5h", "-5m", "+5m", "30m1h", "1h1h", "1 h", "1h 30m"; (4) None, 90 and b"1h" raise TypeError; (5) test_duration.py covers every rule above and `python -m pytest` passes. D: /goal Write a Python module `duration.py` that exposes `parse_duration(text: str) -> int`, converting compact duration strings such as "1h30m" or "45s" into a whole number of seconds, plus a pytest file `test_duration.py`, both in the current directory. The goal is met when parse_duration converts common duration strings such as "1h30m" and "45s" to seconds, invalid input raises an exception, and the tests in test_duration.py pass. ``` ### Results The 12 runs on 2026-10-09: | Group | Hidden test score (3 runs, out of 23) | Evaluator verdict | Time per run | Cost per run (subscription usage, priced) | |---|---|---|---|---| | A one sentence | 22, 20, 19 | No evaluator | 14–15 s | about $0.05 | | B rules in the prompt | 23, 23, 23 | No evaluator | 15–26 s | $0.05–0.08 | | C `/goal` + strict condition | 23, 23, 23 | "Met" after the first turn, all 3 times | 24–27 s | about $0.07 | | D `/goal` + loose condition | 18, 18, 18 | "Met" after the first turn, all 3 times | 13–20 s | $0.04–0.07 | Each evaluator check took 1.6 to 8.2 seconds. Its verdict is not in the `claude -p` output stream; it is a `goal_status` record in the session transcript, `~/.claude/projects//.jsonl`. Here is run 2 of group D verbatim: ```json {"type": "goal_status", "met": true, "iterations": 1, "reason": "The transcript shows duration.py and test_duration.py were written with the Write tool to the current directory. The test file asserts ('45s', 45) and ('1h30m', 5400), and test_invalid covers inputs such as '' , 'abc' and '30m1h' with pytest.raises(ValueError). The Bash run returned '19 passed in 0.01s', so the tests pass."} ``` *The `durationMs` and `tokens` fields are omitted for readability.* Three observations: 1. **What worked here was writing the criteria down, not the evaluator.** Group B had no evaluator at all and scored full marks three times; group C's evaluator said "met" after the first turn every time and never sent Claude back for another round. Every point group A lost was a rule the sentence did not state: none of its three runs guessed that "one space between components is allowed"; twice it added a "weeks" unit of its own and accepted "5w" as five weeks; once each it accepted uppercase "1H", accepted the out-of-order "30m1h", and rejected " 45s " with surrounding whitespace. The writer was not incapable; it did not know what you wanted. 2. **With a loose condition, the evaluator approves anyway.** Group D got "met" all three times, yet scored only 18/23 on the hidden test, failing identically every time: uppercase "1H" accepted as one hour, `None`, `90`, and `b"1h"` raising `ValueError` instead of `TypeError`, and a space between components rejected. The evaluator's reason was always that pytest passed (for example "19 passed"), and those tests were written by the writer to its own understanding. This is the caveat from the top of the article, as it shows up in `/goal`: a vague standard gets approved. Group D even scored below group A, which had no review at all; my guess is that "invalid input raises an exception" nudged the writer toward raising `ValueError` for everything, but three samples are not enough to conclude anything. 3. **The evaluator only reads the conversation.** All six reasons cite things that appeared in the conversation: "a Bash run printed...", "the Write tool reported File created successfully", "pytest showed 22 passed." It did not, and cannot, open `duration.py` and run it itself. **Limits**: the task is small and Sonnet 5.5 finishes it in one turn, so the evaluator never got to do what `/goal` is really for, which is pushing a Claude that wants to stop early back to work on long jobs such as migrations or clearing a backlog. Each group ran only three times. So this data supports one claim only: **on a task finished in one turn, the `/goal` evaluator will not add requirements you left out of the condition.** The Outcomes grader can check with tools, but whether it would catch group D's mistakes still depends on whether the rubric names them; I did not test that, and it is an inference from the design. --- ## Risk: memory is a long-lived injection channel As quoted above, the docs warn that one injection into a read-write store can plant malicious content that every later session trusts. Dreaming adds another layer to that risk. **What follows is my inference; the docs do not discuss it**: a dream reads complete transcripts from up to 100 sessions. If one of them read an injected web page or tool output, the dream may treat the injected content as "a pattern that recurs across sessions" and write it into the new store, phrased even more convincingly than the original. Defenses: - **Split stores by trust level.** Give Agents that handle untrusted input only read-only reference stores; attach writable stores only to Agents that never touch external input. This follows the same idea as Meta's [Rule of Two](https://ai.meta.com/blog/practical-ai-agent-security/): of "processing untrusted input," "accessing sensitive data or systems," and "changing state or communicating externally," a session should hold at most two; when it needs all three, it should not run autonomously and needs at least human approval or another reliable means of validation. - **Review dream output before adopting it.** Dreaming does not overwrite the original by default, and the review step is your line of defense. Where untrusted input is involved, do not configure automatic adoption. - **Choose which sessions feed a dream.** Sessions that read a lot of external content should either be left out or be paired with `instructions` that limit extraction to a specific kind of information. - **Filter candidates by provenance, not just by prompt.** OpenClaw's Dreaming makes this a structural rule: every candidate memory carries a provenance label, and candidates labeled untrusted or system-generated are removed before the consolidation prompt is built, so the model never sees them. When you build your own, recording which session and what kind of input each memory came from stops injection earlier than review after the fact. --- ## Rebuilding on your own stack: two contracts Outcomes and Dreaming are managed features, but the designs have nothing to do with Anthropic and work with any model and any language. Here they are as inputs, outputs, state, and invariants. ### Contract 1: a capped, independent review loop | | Content | |---|---| | Input | Task description (for the writer), rubric (for the grader), output location, max rounds | | State | Current round, each round's verdict | | Output | Passed / needs revision / cap reached / rubric does not apply / interrupted, plus an itemized explanation | | Invariants | Every grading round uses a fresh context; the grader gets only the rubric, the output, and tools, never the writer's reasoning; the loop always terminates | Illustrative pseudocode (Python style, not any SDK's real API): ```python def run_outcome(task, rubric, max_iterations=3): workspace = new_workspace() # outputs live here; writer and grader can both read them feedback = None for iteration in range(max_iterations): writer_turn(workspace, task, feedback) # writer: same session, continues with last round's feedback verdict = grade_in_fresh_context(rubric, workspace) # grader: new context, only rubric and output if verdict.result in ("satisfied", "failed"): return verdict feedback = verdict.explanation # itemized feedback goes back to the writer verbatim writer_turn(workspace, task, feedback, final=True) # one wrap-up turn, no more grading return Verdict("max_iterations_reached", feedback) ``` The key is `grade_in_fresh_context`: it must not reuse the writer's conversation history. It is a brand-new request whose system prompt holds only the rubric, and it reads the output through tools. The same holds in Go, TypeScript, or with another provider's model. If you use a workflow engine like Temporal (which I covered in [why we moved from Celery to Temporal](https://zhuoqidev.com/en/posts/why-temporal-not-celery/)), it becomes a capped loop workflow with the writer and grader as separate activities, and rounds and verdicts land in workflow history for free. ### Contract 2: memory consolidation that never edits the original | | Content | |---|---| | Input | A snapshot of current memory, some session transcripts, a focus direction | | Output | A new memory | | Invariants | Never modify the input; the new memory can be diffed item by item against the old; there is a review gate before adoption; every memory can be traced back to the session it came from | Illustrative pseudocode (Anthropic has not published the pipeline's internals; this only expresses the contract): ```python def consolidate(store, transcripts, focus=None): draft = clone(store) # clone first; the docs say the official job also clones the input store for t in transcripts: draft = synthesize(draft, t, focus) # merge duplicates, replace stale entries, add cross-session patterns return draft # a human reviews diff(store, draft) before swapping it in ``` In my [LLM memory research](https://zhuoqidev.com/en/posts/llm-memory-research/) I broke Agent memory into a loop: experience, writing, storage, retrieval, context assembly, action feedback, and finally consolidation, revision, and forgetting. Dreaming pulls the "consolidation and revision" step out on its own and turns it into an offline, reviewable batch job. --- ## My manual version: "Dreaming" with a Git repository My own Agent work memory is a private Git repository, and in hindsight it is almost a manual Dreaming: | Managed Agents | My work memory repository | |---|---| | Memory store: many small files | A `memory/` directory: one fact per file with frontmatter, plus an index file loaded automatically every session and capped at 200 lines | | An immutable version per write | Git commit history | | Dream input: past sessions | Daily journals: raw material recording "what happened" | | Dream: merge duplicates, replace stale entries, add patterns | Periodic cleanup: merge duplicate entries, delete ones proven wrong, promote key lessons buried in journals into standalone memories | | Review the new store before use | Read `git diff` before committing | The differences are just as clear: - **I edit in place; it writes a separate copy.** I rely on Git to roll back; it never touches the original by design, which is safer. The idea worth borrowing is to reorganize on a branch and merge only after reviewing the diff, which amounts to "leave the original untouched." - **It reads raw session transcripts; I read only journals.** Journals are already filtered by me, so patterns I missed within a single session are invisible to my cleanup. - **My review relies on a rule.** I gave my Agent a rule: never write "merged" or "deployed" before doing it, and treat machine evidence such as PR state, CI, and deploy logs as the source of truth. That is the same idea as an Outcomes rubric: **make the checker produce evidence instead of trusting the worker's claim that it is done.** --- ## Seven takeaways 1. **Outcomes fixes self-grading bias.** The point is not "one more Agent"; it is a grader whose context is isolated from the writer and who holds only the criteria and the output. 2. **The rubric decides everything.** Make it more specific than the task, and make every item force the grader to produce evidence, or it will approve anything. 3. **The loop has to converge.** If every run hits the cap with the same kind of issue, the writer cannot fix it, and more rounds will not help. 4. **Dreaming fixes memory entropy.** It reads many sessions offline and writes a separate, reorganized store, leaving the original untouched for review before use. 5. **Memory is a long-lived injection channel.** Default read-write stores, plus a job that reads raw transcripts, both call for splitting by trust level and keeping a review gate. 6. **Both can be rebuilt without Anthropic.** One is a capped, independent review loop; the other is consolidation that never edits the original. Both hold for any model and any language. 7. **Claude Code only has `/goal`, and its evaluator only reads the conversation.** In the test, what worked was writing the criteria down; with a loose condition it still said "met." auto-dream is still in a rollout, and accounts outside it cannot turn it on with a setting. **The rule: never let the worker grade its own work, and never let memory only grow.** --- ## Sources - [Define outcomes (official docs)](https://platform.claude.com/docs/en/managed-agents/define-outcomes): the define-outcome event, three grading events, five results, rubric guidance - [Dreams (official docs)](https://platform.claude.com/docs/en/managed-agents/dreams): dream inputs and outputs, statuses, limits of `instructions`, billing and limits - [Using agent memory (official docs)](https://platform.claude.com/docs/en/managed-agents/memory): mounting, versions, limits, read-only access and the injection warning, beta headers - [Outcomes: agents that verify their own work (Claude Cookbook, 2026-05-03)](https://platform.claude.com/cookbook/managed-agents-cma-verify-with-outcome-grader): the three-round EV fast-charging brief and rubric principles - [New in Claude Managed Agents: dreaming, outcomes, and multiagent orchestration (Anthropic blog)](https://claude.com/blog/new-in-claude-managed-agents): release status, +10 / +8.4% / +10.1%, Harvey ~6x - [Memory for Claude Managed Agents (Anthropic blog, 2026-04-23)](https://claude.com/blog/claude-managed-agents-memory): memory stores in public beta, Rakuten figures - [ZDNET: Your Claude agents can 'dream' now (2026-05-06)](https://www.zdnet.com/article/your-claude-agents-can-dream-now-how-anthropics-new-feature-works/), [SiliconANGLE (2026-05-06)](https://siliconangle.com/2026/05/06/anthropic-letting-claude-agents-dream-dont-sleep-job/), [Simon Willison's Code w/ Claude 2026 live blog](https://simonwillison.net/2026/May/6/code-w-claude-2026/): release date and third-party coverage - [Meta: Agents Rule of Two: A Practical Approach to AI Agent Security (2025-10-31)](https://ai.meta.com/blog/practical-ai-agent-security/) - [Keep Claude working toward a goal (Claude Code docs)](https://code.claude.com/docs/en/goal): how `/goal` evaluates, reading only the conversation and calling no tools, evaluator tokens usually negligible - [All settings (Claude Code docs)](https://code.claude.com/docs/en/settings-reference): checked on 2026-10-09, no `autoDreamEnabled` entry - [anthropics/claude-code#86209](https://github.com/anthropics/claude-code/issues/86209): `autoDreamEnabled` silently ignored while the rollout flag is off - [anthropics/claude-code#38461](https://github.com/anthropics/claude-code/issues/38461): users already saw "Auto-dream: on" on 2026-03-24 - [Claude Code CHANGELOG](https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md): checked on 2026-10-09, no mention of dream - [Layer5: The Claude Code Source Leak](https://layer5.io/blog/engineering/the-claude-code-source-leak-512000-lines-a-missing-npmignore-and-the-fastest-growing-repo-in-github-history/), [Soma: The Architecture of Forgetting (2026-04-03)](https://soma.gravicity.ai/blog/the-architecture-of-forgetting): third-party analyses of the 2026-03-31 source exposure and `autoDream` - [Dreaming (OpenClaw docs)](https://docs.openclaw.ai/concepts/dreaming): OpenClaw's three-phase consolidation, on by default, provenance filter for untrusted candidates - [Pricing (official docs)](https://platform.claude.com/docs/en/about-claude/pricing): Managed Agents billed by tokens plus session runtime ($0.08 per hour) - [Why do I have to pay separately to use the Claude API? (Claude Help Center)](https://support.claude.com/en/articles/9876003-i-have-a-paid-claude-subscription-pro-max-team-or-enterprise-plans-why-do-i-have-to-pay-separately-to-use-the-claude-api-and-console): a Claude subscription does not include API usage > Checked on 2026-10-06; checks on Claude Code 2.1.294 and the `/goal` test added on 2026-10-09. Outcomes and memory stores are in public beta and Dreaming is a research preview, so fields, limits, and supported models may change. All Outcomes, Dreaming, and memory store request examples come from the official documentation above, and I did not run them; the pseudocode expresses design contracts only and does not represent Anthropic's internal implementation. --- title: "Why Codex Does Not Give the Model Every Tool: tool_search, BM25, and Model Replacement" description: "A plain-language, source-grounded explanation of why Codex loads tools on demand, how BM25 ranks the best few, and how the same design transfers to Python, Go, and other models." url: "https://zhuoqidev.com/en/posts/codex-first-request-tools/" language: "en" author: "Liu ZhuoQi" date: "2026-08-08" updated: "2026-08-09" series: "Agent Architecture Deep Dives" tags: ["Codex","OpenAI","Tool Calling","Tool Search","BM25","MCP","Code Mode","AI Agent"] translation: "https://zhuoqidev.com/posts/codex-first-request-tools/index.md" --- # Why Codex Does Not Give the Model Every Tool: tool_search, BM25, and Model Replacement > **TL;DR** Think of the model as an engineer with a small desk. If hundreds of tool manuals cover it, every request becomes expensive and the right manual is harder to find. Codex instead provides a short catalog and retrieves only the most relevant manuals. This article explains that design, BM25 ranking, reuse in Python or Go, and model replacement. This article is pinned to stable [`rust-v0.147.0`](https://github.com/openai/codex/releases/tag/rust-v0.147.0), released on August 7, 2026, at commit [`be6e8eac`](https://github.com/openai/codex/commit/be6e8eac029b183056b7e4402879f15d2c85f61b). I also rechecked `main` at commit [`646f7c0a`](https://github.com/openai/codex/commit/646f7c0a91b8e327d263335da68ae8ef212895ce) on August 9. Two conclusions matter up front: 1. **The tools visible on the first request are not a permanent list.** They change with the model, connection, runtime environment, and feature settings. 2. **`tool_search` and its BM25 ranking code are implemented, and ordinary direct mode can use them. The stable GPT-5.6 `exec`-centered path simply misses the search entrance.** The feature exists; one wire is not connected. ## Who This Article Is For The reader only needs to know that an LLM can call an outside tool. **Rust and prior Codex source reading are not prerequisites.** Read Rust iterator chains as “filter a list → transform each item → collect the result.” Two reading routes are available: - **Architecture route:** read the first principles, airport diagram, Tool Search runtime sequence, cross-language transfer, and model replacement. - **Source route:** continue through pseudocode, Rust correspondence, the end-to-end query, and the source index. ## Start With Four Plain Statements Erase the Rust function names, OpenAI protocol names, and GPT-5.6 model labels for a moment. A tool-using Agent still cannot escape four facts: 1. **A model can use only tools it can see.** Working execution code is useless to the model until its tool manual enters the request. 2. **The model's desk is finite, and every word costs time and money.** More manuals also create more chances to choose the wrong one. 3. **Search first reduces the choice.** Ranking a few relevant tools out of hundreds is easier than reading every manual at once. 4. **The Agent shell and model can be separated.** If both sides agree on requests, tool calls, and returned results, the language, search method, and model can change. The minimum architecture in this article is therefore one chain: ```mermaid flowchart TD A["limited desk space"] --> B["every tool manual cannot stay open"] --> C["keep a few common tools and a catalog"] --> D["retrieve the best few complete manuals"] --> E["model calls a tool"] --> F["the program executes it
and returns the result"] classDef cause fill:#fff3bf,stroke:#f08c00 classDef out fill:#d3f9d8,stroke:#2f9e44 class A cause class F out ``` Once that chain is clear, it becomes easy to decide what must stay and what can be removed: | What appears in the source | Must understand? | Why | | --- | --- | --- | | Whether a tool is visible now, visible after search, or never visible | Yes | It determines what the model can choose | | `searchable words → ranked IDs → complete manual` | Yes | This design transfers to any Agent | | How the model connection sends data and what it supports | Yes | It determines whether model replacement is safe | | Rust iterators, concrete type names, and paths | Not initially | They are project implementation, not architectural law | | The `bm25` crate and array-index trick | Replaceable | Python, Go, or a search service can implement the same interface | | The current GPT-5.6 Tool list | Version snapshot only | Model, environment, and Feature changes recompute it | ## Only Four Things Need Names | Plain name | Meaning | Source name when needed | | --- | --- | --- | | Tool manual | A tool's name, purpose, and required parameters | Schema / Tool Spec | | Tool catalog | Every tool the program currently knows, not necessarily every tool shown to the model | Registry | | Execution code | The program that does the work after the model names a tool | Runtime / Handler | | Model connection | The adapter that sends Codex requests in a format the model service understands | Provider | The source names appear later only for verification. They are not vocabulary the reader must memorize. ## Why Does Codex Put Many Tools Behind `exec`? This Mode does not mean the user can only write code. It controls **how the model sees and calls Tools**: | Tool Mode | Tool layout visible to the model | | --- | --- | | Direct | Most Tools are independent top-level entrypoints | | Code Mode | Some Tools remain direct while others are orchestrated as `tools.xxx()` inside `exec` | | Code Mode Only | Most ordinary Tools stop occupying separate top-level slots; the model primarily uses `exec`, while `wait` and a small control surface remain direct | In the pinned release, the GPT-5.6 Sol model catalog already contains `"tool_mode": "code_mode_only"`. **Codex selects it automatically with that model; there is no additional UI switch to turn on.** For models without a built-in selector, the source also retains an experimental feature override: ```toml [features] code_mode_only = true ``` The CLI equivalent is `codex --enable code_mode_only`. This also enables Code Mode as a dependency, but public configuration marks the feature as under development. Codex warns that forcing it on a model that does not advertise support may degrade performance, so the override is better suited to source testing than arbitrary production use. It is also unrelated to `--code-mode-host`, which only selects a remote execution host for Code Mode. The diagram can now be read without treating an internal selector as a familiar product switch. It shows both the deferred-discovery design and the stable Code Mode Only first-turn state. The dashed path and barrier mean that `tool_search` is implemented but not exposed to the model in this specific first-turn layout. ![Airport metaphor for Codex top-level and exec-nested tools, with tool_search implemented but not first-turn exposed in stable Code Mode Only](https://zhuoqidev.com/_astro/first-request-airport-v2-en.peuSoPLN_1khLSJ.webp "This is a versioned conceptual diagram, not a timeless tool list. Solid paths are visible entrypoints; the dashed tool_search path is the on-demand design, while the barrier marks its missing first-turn exposure in rust-v0.147.0 GPT-5.6 Code Mode Only.") ## What Can the Model See at First? Separate the Public API from Codex's Internal Framing First, correct the misleading shorthand: **when an application calls the [public OpenAI Responses API](https://developers.openai.com/api/docs/guides/tools) directly, tools still belong in the top-level `tools` field, including with GPT-5.6. Seeing `additional_tools` in Codex source is not a reason to rewrite a public API request.** Two different questions were being mixed together: 1. **Which tools can the model use?** That depends on the tool definitions the service ultimately presents to the model. 2. **How do those definitions travel over the network?** That depends on the request contract agreed upon by the client and service. The model does not inspect an HTTP packet and reason about JSON key spelling. The service first turns the request into a callable interface for the model. `tools` and `additional_tools` are therefore better understood as **two envelope formats** that can carry the same tool manual. The distinction affects client–server compatibility, not the model's underlying reasoning ability. ### Comparison One: Calling the Public Responses API Directly This is a complete minimal request. It uses one `weather` namespace so every piece can be compared with the second request: ```bash curl https://api.openai.com/v1/responses \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5.6", "instructions": "Call a tool for live weather; do not guess.", "input": [ { "role": "user", "content": "What is the temperature in Paris now?" } ], "tools": [ { "type": "namespace", "name": "weather", "description": "Look up live weather.", "tools": [ { "type": "function", "name": "get_current_weather", "description": "Get the current temperature for a city.", "parameters": { "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"], "additionalProperties": false }, "strict": true } ] } ], "tool_choice": "auto" }' ``` There are three sibling inputs: `instructions` states how to behave, `input` carries the user's request, and `tools` declares what is callable. The tool is neither a prose list hidden in a system prompt nor part of the user message. ### Comparison Two: The Request Codex Sends After Selecting Responses Lite The stable Codex model catalog selects `use_responses_lite` for GPT-5.6 Sol. Codex still builds a Responses-shaped body, but moves two things: top-level `tools` becomes the first `input` item, and top-level `instructions` becomes the second item. The following deliberately reuses the illustrative `weather` namespace so the fields can be compared one by one. Field positions and fixed values come from source; values in angle brackets are generated by Codex for each turn: ```http POST Authorization: Bearer Content-Type: application/json x-openai-internal-codex-responses-lite: true { "model": "gpt-5.6-sol", "input": [ { "type": "additional_tools", "role": "developer", "tools": [ { "type": "namespace", "name": "weather", "description": "Look up live weather.", "tools": [ { "type": "function", "name": "get_current_weather", "description": "Get the current temperature for a city.", "parameters": { "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"], "additionalProperties": false }, "strict": true } ] } ] }, { "type": "message", "role": "developer", "content": [ { "type": "input_text", "text": "Call a tool for live weather; do not guess." } ] }, { "type": "message", "role": "user", "content": [ { "type": "input_text", "text": "What is the temperature in Paris now?" } ] } ], "tool_choice": "auto", "parallel_tool_calls": false, "reasoning": { "effort": "low", "context": "all_turns" }, "store": false, "stream": true, "include": ["reasoning.encrypted_content"], "prompt_cache_key": "", "text": { "verbosity": "low" }, "client_metadata": { "x-codex-turn-metadata": "" } } ``` The most revealing detail is not merely another field name. The header literally says **`internal-codex-responses-lite`**. [Codex's source test](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/tests/suite/responses_lite.rs#L28-L139) also asserts that top-level `tools` and `instructions` are absent, that `input[0]` is `additional_tools`, and that a developer instruction follows it. This is an internal transport contract between Codex and a provider that supports it, not a new public Responses API format that ordinary developers should adopt. | Exact comparison | Public Responses API | Codex Responses Lite | | --- | --- | --- | | Intended caller | Ordinary API applications | Codex and providers that explicitly support Lite | | Tool location | Top-level `tools` | `input[0].tools` in an item whose type is `additional_tools` | | Instruction location | Top-level `instructions` | The following developer message | | Top-level `tools` present? | Yes | No; source sets it to `None` and serialization omits it | | Public and portable? | Yes | No; the request header explicitly marks it Codex internal | | Can the model ultimately see the tools? | Yes | Only when the service supports this internal contract | The precise earlier distinction should therefore have been: **one Codex client has two request encodings, and model metadata selects the internal Lite encoding for some models**—not “GPT-5.6 replaced the public API format.” In the Lite path, first-turn visibility still has two layers: entrypoints such as `exec` and `wait` are directly callable, while tools documented inside the `exec` guide are invoked as `tools.xxx(...)`. Think of the public API and Lite as boarding passes printed by two airlines, not two different airports. `exec` is then a connecting pass that already lists the counters available beyond its gate. ## What GPT-5.6 Sol/Terra Can Call Directly at First With the normal App/CLI defaults, an available execution environment, and no enterprise policy disabling them, the core top-level surface is: | First-turn entrypoint | Capability | Why it stays top-level | | --- | --- | --- | | `exec` | Runs JavaScript that can orchestrate multiple nested tools in one cell and return a compact result | The main Code Mode gateway; it reduces model–tool round trips | | `wait` | Waits on a long-running `exec` cell that previously yielded | Long work can continue without one blocking tool call | | `request_user_input` | Asks the user a short question in collaboration modes where it is allowed | Human control flow should not be buried in generated orchestration code | | `collaboration` | A namespace containing `spawn_agent`, `send_message`, `followup_task`, `wait_agent`, `interrupt_agent`, and `list_agents` | GPT-5.6 Sol/Terra select Multi-Agent V2, whose defaults keep this control surface directly callable outside Code Mode | `collaboration` is one namespace object but six callable functions. Luna still selects Multi-Agent V1, so its first-turn shape differs. Collaboration can also disappear when an agent depth limit is reached. ## What Lives Inside `exec`? `exec` is not merely another name for a shell. It is closer to a transit pass for the workstations behind the gate. In source, `add_core_tool_sources()` collects shell tools, MCP resource tools, utilities, and collaboration tools, then the planner adds extension, dynamic, and hosted tools. | Category | Common tools | Gate | | --- | --- | --- | | Command execution | `shell_command` is common on Windows; Unified Exec uses `exec_command` + `write_stdin` | Requires an execution environment; exact shape depends on model and feature selection | | File changes and inspection | `apply_patch`, `view_image` | The model and environment must support the corresponding tool | | Work orchestration | `update_plan` | Enabled by default, but configurable | | MCP resources | `list_mcp_resources`, `list_mcp_resource_templates`, `read_mcp_resource` | At least one MCP server must be connected | | App extensions | `web.run`, `image_gen.imagegen` | Provider, authentication, modalities, network mode, and feature gates must all pass | | Plugin installation | `request_plugin_install`, sometimes a candidate-list tool | Apps, Plugins, and ToolSuggest must be enabled and candidates must exist | | Other extensions | Goal, automation, and App Server dynamic tools | The relevant extension or client registration must exist | Code that can execute a tool may already exist in the repository without that tool appearing on every first request. Codex still has to **add it to the catalog → decide whether to show it now, after search, or never → decide whether it belongs inside `exec` → serialize the final request**. Missing environments, disabled features, provider limitations, plan gates, and name collisions can remove a tool along the way. ## Is `tool_search` on the First Request? By design, `tool_search` is registered only when both conditions hold: - The model advertises `supports_search_tool` and the provider supports namespace tools. - The registry contains at least one Deferred Tool with `search_info`. In `rust-v0.147.0`, MCP tools are registered into the runtime but receive Deferred exposure whenever search is available. Their full schemas do not have to occupy the first-turn context. A `tool_search` call returns matching `LoadableToolSpec` entries for the next model call, with a default limit of eight. There is, however, an important detail in the latest stable source: **the GPT-5.6 `code_mode_only` path does not currently convert `ToolSpec::ToolSearch` into an `exec`-nested definition**. `register_code_mode_executors()` explicitly continues past `ToolSearch`, while Code Mode Only hides ordinary direct tools other than `exec` and `wait`. The practical result is: - In Direct mode, `tool_search` can be a first-turn top-level tool whenever deferred tools exist. - In the stable GPT-5.6 Code Mode Only path, `tool_search` **does not enter** the callable first-turn surface even if its BM25 handler has been registered. Without an exposed Tool Spec, the Agent has no invocation entrypoint. - Public issue [#32101](https://github.com/openai/codex/issues/32101) documents this bridge gap. As of August 9, 2026, commit [`646f7c0`](https://github.com/openai/codex/commit/646f7c0a91b8e327d263335da68ae8ef212895ce) on `main` still contains the skip branch. That distinction matters: **present in the registry is not the same as visible in the request, and intended to be discoverable is not the same as wired through every Tool Mode today.** ## What One Tool Search Actually Looks Like Set aside the Code Mode Only bridge gap for a moment and examine a Direct/Deferred path in which `tool_search` **is exposed to the model**. It is not a retrieval job that automatically runs every turn. It is one optional action in the model's toolbox. ### It Is Model-Chosen, Not an Automatic Per-Turn Fallback The Codex client contains no fallback loop that says, “if none of the explicit Tools fit, run BM25.” Tool choice is automatic; only after the model emits a structured `tool_search_call` does the Router dispatch it to the handler. The model may use an already-visible Tool first or search immediately from the task—it does not have to fail against each explicit Tool before searching. Imagine a developer entering a hardware store. A hammer and screwdriver are already on the shelf, and the clerk can search the stockroom. The clerk does not search the stockroom after every step of tightening a screw. A request such as “measure fiber loss,” however, justifies a catalog lookup. Once a searched Tool is loaded and remains in the active conversation context, later turns can call it **without searching for the same Tool again**. A new need, a changed catalog, a new task, or loss of the earlier search output from active context can require another search. The official API's Deferred Loading still gives the model minimal discovery information on turn one: Namespaces and MCP servers expose high-level names and descriptions, while an individually deferred Function retains its name and description but defers most parameters. In Codex's client-side BM25 path, the model first sees the `tool_search` entry and its searchable-source guidance; the complete `LoadableToolSpec` arrives only after a hit. Both designs follow the same principle: **show the catalog before moving the warehouse.** ### Should a New Agent Use This Difference? Start from first principles. An Agent only needs four links to work: **the tools you own → the definitions the model receives → the format in which the model requests a call → the code that executes it and returns a result**. `tools` versus `additional_tools` is only transport packaging for the second link. It should not leak into business tool implementations. | Your situation | What to do | What not to do | | --- | --- | --- | | Calling the public OpenAI Responses API | Keep using top-level `tools`; add `tool_search` and deferred tools there when needed | Do not construct `additional_tools` yourself | | Only a small tool set | Send complete definitions directly and keep the design simple | Do not add search merely to imitate Codex | | Hundreds of tools, all known when creating the request | Use hosted `tool_search` so the OpenAI service searches and loads them | You do not need to copy Codex's local BM25 first | | Tools vary by tenant, project, or permission | Use client-executed `tool_search`; let the model request discovery, run BM25/vector/hybrid retrieval in your application, and return `tool_search_output` | Do not send every tenant's entire inventory | | Forking Codex with a provider that explicitly supports Responses Lite | Preserve the Codex provider adapter and let it produce `additional_tools` plus the internal header | Do not couple business code directly to this internal format | | Connecting DeepSeek, GLM, Kimi, or a local model | Keep one internal tool representation and write a thin adapter for each model protocol | Do not assume another provider understands OpenAI's internal `additional_tools` item | For a new large-tool Agent that calls the public Responses API, this official pattern—not the Lite body—is the part you can use directly: ```python from openai import OpenAI client = OpenAI() response = client.responses.create( model="gpt-5.6", input="List open orders for customer CUST-12345.", tools=[ { "type": "namespace", "name": "crm", "description": "Customer profile and order tools.", "tools": [ { "type": "function", "name": "list_open_orders", "description": "List open orders by customer ID.", "defer_loading": True, "parameters": { "type": "object", "properties": { "customer_id": {"type": "string"} }, "required": ["customer_id"], "additionalProperties": False, }, "strict": True, } ], }, {"type": "tool_search"}, ], parallel_tool_calls=False, ) ``` This is the reusable design: show the model the `crm` catalog first, then load the full `list_open_orders` parameters only when needed. Whether the outer transport uses top-level `tools`, a Lite input item, or another model vendor's format belongs in the provider adapter at the edge. ### Where Results Appear and How Long They Remain Usable A client-executed search has this sequence: ```text model: tool_search_call(query="calendar") Codex: BM25 Top-K → LoadableToolSpec context tail: tool_search_output { call_id, status, tools: [...] } model: function_call(name="create_event", ...) later turns: create_event remains callable ``` `tool_search_output` is a dedicated Responses input item—not system text, user text, or a retroactive edit of the initial `additional_tools`. Codex records the Call/Output pair in conversation history. When the next request carries those history items, the model can treat the complete schemas in `tools` as loaded candidates. The official design injects loaded tools at the end of model context, which also preserves a stable prefix and improves the opportunity for Prompt Cache hits. “Callable in future turns” needs a boundary: it means within the same active conversation while that output remains in the context sent to the model. It is not a process-wide permanent installation. A new conversation, context-compaction policy, or catalog update can require discovery again. Think of placing a stockroom manual into this work order, not welding the machine permanently onto the bench. ### Why Does the Stable Catalog Select Code Mode Only for GPT-5.6? The source proves a configuration fact: GPT-5.6 Sol, Terra, and Luna in the stable model catalog all declare `tool_mode: "code_mode_only"` and Responses Lite; the adjacent GPT-5.5 entry does not. The code treats this as **model capability metadata and a protocol compatibility contract**. There is no comment or design document saying, “5.6 is smarter, therefore enable it.” API-level `tool_search` is not exclusive to 5.6 either: the official guide lists support for GPT-5.4 and later. The following is an explicitly labeled engineering inference, not a source quotation. Code Mode Only asks the model to write valid JavaScript inside `exec`, follow Tool Schemas, manage asynchronous work and failures, and filter or compress results within one cell. Stronger coding, reasoning, and tool-use training are clearly enabling conditions. Activation also depends on targeted post-training, evaluation thresholds, provider protocol support, and rollout policy. **“A stronger model makes this viable” is reasonable; “raw strength is the only reason” is not established.** ## What Shrinking Hundreds of Manuals to a Few Actually Saves Suppose an Agent connects N Tools and each complete Schema averages S tokens. Sending everything directly makes the tool portion of one request approximately: ```text Full cost ≈ N × S ``` Deferred loading keeps only a search entrance and short catalog resident, then loads K hits: ```text Deferred cost ≈ catalog_summary + search_tool + K × S ``` When `K ≪ N`, the saving is not only tokens. The model sees fewer similar names and parameter combinations, making final Tool selection easier. Strictly speaking, `tool_search` does not increase intelligence inside the model weights. It improves effective Agent intelligence by **reducing noise, presenting the right information, and splitting one large decision into two smaller decisions.** This is not free in every case. Search adds a Model → Tool → Model round trip and can miss a relevant Tool. If the Agent has a dozen Tools and uses most of them every turn, direct exposure may be faster. If it has hundreds or thousands and needs only a few per task, Deferred + Top-K becomes much more attractive. That design judgment transfers without Codex or Rust. ## BM25 Does Not Abandon Keywords; It Ranks Them “Why BM25 instead of keyword matching?” contains a small misconception. **BM25 is still lexical keyword retrieval.** It is not an embedding model, and it does not inherently know that “book a meeting” and “create a calendar event” are synonyms. The difference is that a crude matcher usually answers only hit or miss; BM25 also answers which hit deserves first place. ![Library-card metaphor comparing boolean keyword matching with BM25 Top-K ranking over tool metadata](https://zhuoqidev.com/_astro/bm25-library-v2-en.CQ4EywXb_Z1g0rOF.webp "A keyword filter dumps every matching card into a pile. BM25 ranks by term rarity, frequency saturation, and description length. Top-5 is illustrative for readability; Codex defaults to a limit of eight. The footer also makes the lexical—not semantic—boundary explicit.") First, keep this conceptual shorthand in mind: ```text score(D, Q) = Σ IDF(t) × TF_saturation(t, D) × length_normalization(D) ``` That is not the full equation executed by the library. `bm25` 2.3.2 uses the standard form: ```text score(D, Q) = Σ IDF(qᵢ) × f(qᵢ,D) × (k₁ + 1) ───────────────────────────────────────── f(qᵢ,D) + k₁ × (1 - b + b × |D| / avgdl) ``` Here `f(qᵢ,D)` is the term frequency in one tool card, `|D|` is that card's length, and `avgdl` is the corpus-wide average length. Codex calls `with_documents(...).build()` without overriding parameters, so the crate defaults apply: `k₁ = 1.2` and `b = 0.75`; fitting the document batch establishes `avgdl`. Codex decides which cards reach the librarian, while the crate tokenizes, counts, scores, and ranks them. In library terms: - **Rare terms weigh more (IDF).** Almost every card may contain `get`, so it has little discriminating power. If only a few contain `calendar`, that term is much more useful. - **Repetition saturates.** Writing `calendar` three times can strengthen the signal, but it does not mechanically make the card three times better. Keyword stuffing cannot dominate indefinitely. - **Length is normalized.** A long description has more chances to contain query words. BM25 corrects for that so “wrote the most” does not automatically mean “most relevant.” - **The output is already Top-K.** `tool_search` wants eight ordered candidates by default, not a bag of unordered boolean hits. The source does not index only tool names. For ordinary tools, search text includes both the raw name and an underscore-to-space form—`create_event` and `create event`—plus namespace metadata, descriptions, parameter names, and parameter descriptions. MCP tools add canonical and callable names, server, title, connector, plugin display names, and input-schema property names. This matters because the default tokenizer splits on Unicode word boundaries, lowercases, removes English stop words, and applies English stemming; the space-separated form makes `create` and `event` independent lexical signals. ## Source Deep Dive (Optional): How Codex Connects BM25 The formula explains how documents are scored, but an implementation must still answer three questions: where the documents come from, how a BM25 document ID leads back to a tool, and how a search hit becomes a callable schema on the next turn. In the stable release, that path runs through `spec_plan.rs`, the `ToolSearchHandler`, and the search-text builders in `codex-tools`. The first important clarification is that **Codex does not reimplement the BM25 equation itself**. `codex-rs/Cargo.toml` depends on [`bm25 = "2.3.2"`](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/Cargo.toml#L294). That in-memory library handles tokenization, document frequency, average document length, and scoring; Codex supplies the corpus and maps ranked results back into its tool types. You can understand the implementation without Rust by reading it as four lines of pseudocode: ```text cards = deferred_tools.map(search_text + loadable_schema) engine = BM25.build(cards.search_text) ids = engine.search(query, limit = 8) return merge(cards[id].loadable_schema) ``` The Rust below is the strongly typed form of those four operations. Read `.filter()`, `.map()`, and `.collect()` as ordinary list filtering, transformation, and collection. ### Step 1: Collect only tools that should appear after search `append_tool_search_executor()` filters the registry for Deferred Tools and asks each runtime for its `search_info`. This is condensed code with the real types and method names preserved: ```rust let search_infos = registry .entries() .filter(|tool| tool.exposure.is_deferred()) .filter_map(|tool| tool.runtime.search_info()) .collect::>(); registry.register_trusted( tool_search_handler_cache.get_or_build(search_infos, source_listing), ); ``` Each entry binds the text used for retrieval to the definition that should be returned on a hit: ```rust pub struct ToolSearchEntry { pub search_text: String, pub output: LoadableToolSpec, } ``` `search_text` contains the names, descriptions, parameters, namespace metadata, and other fields described above. `output` is the Function or Namespace schema that can be loaded into the next model request. Think of a library card whose front contains searchable terms while its back carries the retrieval slip for the actual book. ### Step 2: Give every card a numeric retrieval number [`ToolSearchHandler::new()`](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/tools/handlers/tool_search.rs#L76-L110) enumerates the cards and uses each entry's position in `search_infos` as its BM25 document ID: ```rust let documents = search_infos .iter() .map(|info| info.entry.search_text.clone()) .enumerate() .map(|(idx, text)| Document::new(idx, text)) .collect(); let search_engine = SearchEngineBuilder::::with_documents( Language::English, documents, ) .build(); ``` That `usize` is the bridge back. The BM25 engine only needs to return ranked IDs such as `7, 2, 11`; Codex can then look up `search_infos[7]`, `search_infos[2]`, and `search_infos[11]`. There is no database key or second mapping table—the array position is the retrieval number. The full corpus is indexed in memory with `Language::English`. ### Step 3: Rank the IDs and recover complete tool manuals On invocation, the handler trims the query and rejects both an empty query and `limit = 0`. If the caller omits the limit, `TOOL_SEARCH_DEFAULT_LIMIT` supplies eight. The central retrieval code is short: ```rust let results = self.search_engine .search(query, limit) .into_iter() .map(|result| result.document.id) .filter_map(|id| self.search_infos.get(id)) .map(|info| &info.entry); coalesce_loadable_tool_specs( results.map(|entry| entry.output.clone()), ) ``` Two details are easy to miss: 1. Codex does not forward the numeric BM25 score to the model; it uses the order already produced by the retrieval library. 2. `coalesce_loadable_tool_specs()` merges multiple hits from the same Namespace, avoiding several duplicate Namespace wrappers. The whole pipeline is therefore: ```mermaid flowchart TD A["Deferred Runtime"] --> B["ToolSearchInfo
search_text + LoadableToolSpec"] --> C["BM25 Document#lt;usize#gt;"] --> D["ranked Top-K document IDs"] --> E["recover LoadableToolSpec"] --> F["coalesce Namespaces"] --> G["include in the next model request"] classDef out fill:#d3f9d8,stroke:#2f9e44 class G out ``` ![Complete Codex tool_search data flow from Deferred Tool through ToolSearchInfo, Document usize, the bm25 crate, and tool_search_output](https://zhuoqidev.com/_astro/tool-search-data-flow-v1-en.BedGVwN3_Zyy1bE.webp "The source boundary is explicit: steps 1–3 and 5–6 are Codex glue code; step 4 is the external bm25 2.3.2 crate. The IDs [7, 2, 11] are labeled as illustrative—not measured ranks or scores.") ### One Query, End to End Suppose the query is `calendar`. One indexed card might contain the following shortened text: ```text create_event create event Create a calendar event title start_time end_time ``` The default English tokenizer applies Unicode normalization, lowercasing, stop-word removal, and stemming to both query and card. If BM25 places this card in Top-K, the handler uses its document ID to recover the associated `LoadableToolSpec`. “Matching text” is not itself the invocation: **the match yields a card number, and that number retrieves the complete schema.** Next, [`ToolSearchOutput::to_response_item()`](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/tools/context.rs#L149-L185) turns the result into a dedicated Responses input item. The source unit test verifies this wire shape; the example keeps an empty parameter object to make the path easy to see: ```json { "type": "tool_search_output", "call_id": "search-1", "status": "completed", "execution": "client", "tools": [{ "type": "function", "name": "create_event", "description": "", "strict": false, "defer_loading": true, "parameters": {"type": "object", "properties": {}} }] } ``` This is neither ordinary text output nor a late mutation of the first request's `additional_tools` array. It enters conversation history as `tool_search_output` and travels with the next Responses request, allowing the provider and model to treat the entries in `tools` as search-loaded candidates. The unit test proves the output framing, not an actual BM25 ranking; ranking remains the job of `SearchEngine::search()` described above. There is one final performance detail. [`ToolSearchHandlerCache`](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/tools/handlers/tool_search.rs#L34-L74) compares the new `search_infos` and source-listing mode with the cached handler. If the tool world is unchanged, Codex reuses the existing `Arc` and in-memory index; it rebuilds only when MCP, plugin, or dynamic-tool metadata changes. In library terms, yesterday's card catalog remains useful until new books arrive. To follow the exact source path, read [`append_tool_search_executor()`](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/tools/spec_plan.rs#L1165-L1185), [`ToolSearchHandler`](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/tools/handlers/tool_search.rs#L76-L197), and the [`ToolSearchEntry` / `search_text` builders](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/tools/src/tool_search.rs#L12-L150) in that order. ## Python and Go Can Use the Same Design The lesson above should not be “Codex has a Rust-specific trick.” It should reveal three language-independent interfaces: ```text build(cards: [{id, search_text, schema}]) -> index search(index, query, top_k) -> ranked_ids load(cards, ranked_ids) -> callable_schemas ``` Codex stores cards in `Vec`, ranks them with a Rust `bm25` crate, and recovers schemas by array index. Another language only changes the container and library: | Environment | Tool cards | BM25 ranking | Load after a hit | | --- | --- | --- | --- | | Rust / Codex | `Vec` | `bm25` crate | `search_infos[id]` | | Python Agent | list / dataclass / dict | Any BM25 library or search service | `cards[id]["schema"]` | | Go Agent | `[]ToolCard` struct | Any Go BM25 implementation or search service | `cards[id].Schema` | | Multi-service system | Database or Tool Registry | Separate retrieval service | Fetch Schema by Tool ID | The reusable part is the data contract, not the source syntax: 1. Generate high-quality `search_text` for every Tool, including its name, action, object, parameters, and risk terms. 2. Return ranked Top-K IDs from BM25, not an unordered boolean set of “contains keyword” matches. 3. Resolve IDs into complete, validated schemas and write them into subsequent Agent state. 4. Measure recall, false retrievals, eventual call success, and added round-trip latency before choosing K or adding vector recall. A business Agent written entirely in Python or Go can therefore reuse this design. The BM25 equation, card inputs, and Top-K output do not change with the programming language; Rust is simply Codex's implementation vehicle. ## Why BM25 Is a Sensible Engineering Choice Here The source establishes what Codex uses; it does not contain a design memo claiming the following exact rationale. As an engineering inference from the corpus and execution path, BM25 fits tool discovery well: 1. **Tool metadata is short, structured, and terminology-heavy.** API names and parameters such as `calendar`, `pull_request`, `spawn_agent`, and `issue_number` are strong lexical signals. 2. **The index is rebuilt locally.** The handler constructs a Rust `bm25` engine with `Language::English`; there is no embedding request and no vector database to operate. 3. **First-turn latency and availability matter.** The tool set changes with MCP servers, plugins, dynamic tools, and policy. BM25 is quick to rebuild, cheap to query, deterministic, and explainable for this small dynamic corpus. 4. **The job is schema reduction, not open-world question answering.** Retrieval narrows dozens or hundreds of cards to Top-K; a capable model still makes the final tool choice. 5. **Exact technical distinctions are safety-relevant.** `delete_issue` and `get_issue` are semantically related but operationally very different. Lexical evidence from names and parameters matters. A substring filter lacks robust ranking. Embeddings improve paraphrase and cross-language recall, but introduce another model or service dependency, index-update work, and semantic false positives. For short schemas, BM25 is a pragmatic middle ground. ## The Boundaries Are Just as Clear The current engine uses `Language::English` and does not add vector recall, a synonym layer, an explicit exact-name boost, or a reranker. These queries are therefore outside its sweet spot: - A Chinese query meaning “schedule a meeting” when the tool only says `create_calendar_event`. - “Review code” when the tool only uses `pull_request_review`. - Two tools whose descriptions share boilerplate while the important business distinction is implied rather than named. That is why tool `name`, `description`, and parameter documentation are operational metadata, not cosmetic copy. BM25 is the librarian, but it can read only what is printed on the cards. If tool scale, multilingual use, or paraphrase diversity grows substantially, a sensible upgrade is **BM25 recall + exact-name boost + lightweight reranking**, not necessarily an immediate jump to pure vector retrieval. ## Can BM25 Search Memory Too? **Yes, but it fits best as the lexical-recall leg—not as an entire long-term memory system.** First, a source-level correction: this stable Codex release does not reuse the BM25 implementation above for Memory search. `ext/memories` exposes substring queries, optionally normalizes whitespace or separators, and organizes matches by path and line number. It neither computes BM25 relevance nor ranks by semantic similarity. BM25 is still useful for Memory because error codes, function names, file paths, project codenames, people, and exact decision language are powerful lexical anchors. For “how did we handle `EADDRINUSE` last time?”, BM25 is often more stable than vectors alone and ranks results more usefully than a substring filter. Memory is harder than a collection of Tool cards, however. The same event may be paraphrased; a newer conclusion may supersede an older one; retrieval must account for whose memory, which project, when it happened, and how trustworthy it is. BM25 does not model those relationships: | Memory need | BM25 alone | | --- | --- | | Exact names, paths, error codes, and APIs | Strong | | Paraphrases and cross-language expressions | Weak | | User, project, and time scope | Needs metadata filters | | Conflicting conclusions, trust, and importance | Needs additional ranking and governance | A more reliable production pipeline is: ```mermaid flowchart TD A["metadata filters for user / project / time"] --> B1["BM25 lexical recall"] & B2["embedding semantic recall"] B1 & B2 --> C["fuse both candidate sets
(for example, RRF)"] C --> D["weight recency, importance, trust, and usage"] --> E["lightweight reranking"] --> F["return source spans and provenance"] classDef out fill:#d3f9d8,stroke:#2f9e44 class F out ``` For a small local Memory dominated by technical logs, starting with BM25 is entirely reasonable; it adds meaningful ranking over the current substring list. As paraphrase, multilingual use, and contradictory memories grow, hybrid retrieval becomes more valuable. This is a general architecture recommendation—**not a claim that Codex already implements this pipeline.** ## Open-Source Codex Can Change Models, but It Takes More Than a New Name The precise claim is not “turn GPT-5.6 into DeepSeek, GLM, or Kimi.” It is: **keep the Codex Agent Runtime and replace the model and Provider behind it.** Core components such as Codex CLI, SDK, and App Server are open source, while model transport is isolated behind the Provider layer. The sandbox, approvals, Tool Registry, MCP, conversation history, and execution loop can therefore remain reusable. The minimum control loop is independent of a model brand: ```mermaid flowchart TD A["user task + conversation state + Tool Schemas"] --> B["Model Provider"] B --> C["Tool Call / Agent Text"] C --> D["Codex Runtime executes, approves, records"] D -->|"next model request"| B classDef swap fill:#e5dbff,stroke:#6741d9 class B swap ``` Model replacement is not only a one-line `model = "..."` edit. At least three contracts must align: | Adaptation layer | What must align | Symptom when it does not | | --- | --- | --- | | Transport | `base_url`, authentication, headers, streaming, retries | Connection, 401, or broken-stream failures | | Wire protocol | Responses input items, Tool Call/Output, stream events, and errors | Text works but the Tool loop breaks | | Capability | Context size, structured calls, parallel Tools, reasoning fields, Tool Mode, and Search | Requests run but Agent quality or features disappear | The official configuration supports a custom `model_provider`. If a business gateway already exposes a compatible Responses endpoint, the shortest path looks like: ```toml model = "your-business-model" model_provider = "business_gateway" [model_providers.business_gateway] name = "Business model gateway" base_url = "https://llm-gateway.example.com/v1" env_key = "BUSINESS_LLM_API_KEY" wire_api = "responses" ``` One pinned-version restriction is essential: the [`WireApi`](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/model-provider-info/src/lib.rs#L49-L79) in `rust-v0.147.0` accepts only `responses` and explicitly rejects `wire_api = "chat"`. Therefore, when integrating DeepSeek, GLM, Kimi, or another business model: - If the server fully supports Responses, streaming events, and Tool Calling, start with a custom Provider configuration. - If it exposes only a Chat-Completions-style API, use a gateway that translates Responses bidirectionally into the vendor protocol, or modify the open-source Provider/Client adapter. - “The endpoint returns text” is not proof of Agent compatibility; evaluate real Tool Calls, Tool Outputs, long context, recovery, and parallel calls. Another subtle source behavior matters: an unknown model slug receives [`fallback model metadata`](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/models-manager/src/model_info.rs#L137-L180). Those conservative defaults disable `supports_search_tool`, Responses Lite, Code Mode Only, and parallel Tool Calls. A basic Direct Tool loop may work first, but GPT-5.6-specific optimizations **do not transfer automatically**. Restoring them requires extending model metadata or capability configuration and proving through evals that the replacement model handles those protocol actions reliably. The fastest low-risk migration order for a business team is: 1. Make the replacement model close a text-plus-small-Direct-Tool loop through a Responses adapter. 2. Validate parameter schemas, Tool Call/Output pairing, stream interruption, and context compaction. 3. Compare success rate, cost, and latency on business tasks—not only chat quality. 4. Enable parallel Tools, Deferred Search, and Code Mode incrementally. That is the real reuse value of an open-source Agent framework: retain a mature execution and governance foundation, replace model transport and capability boundaries, then add business Tools, policy, Memory, and evals in layers you control. ## Seven Takeaways First, the initial request does not “send every tool to the model.” Exposure, Tool Mode, provider capabilities, environment state, and feature gates jointly compute the interface. Second, the public Responses API puts tools in top-level `tools`; only Codex's internal Responses Lite moves them into an `additional_tools` input item. Both are structured data rather than system/user prose. Third, `tool_search` is a model-chosen action, not an automatic per-turn fallback. Its structured output enters history, so loaded Tools remain callable in the same active conversation. Fourth, Code Mode Only demands stronger coding and orchestration capabilities, but the stable source only establishes model compatibility metadata; it does not prove that “strength” is the sole reason GPT-5.6 receives it. Fifth, BM25 is ranked keyword retrieval. It fits Tool Schemas and exact lexical Memory recall; a complete Memory system benefits from metadata, BM25, vector recall, and reranking together. Sixth, BM25 and the `search_text → ranked IDs → Schema` flow form a language-independent contract that can be reproduced in Rust, Python, Go, or a separate retrieval service. Seventh, open-source Codex permits Model Provider replacement, but Transport, the Responses wire protocol, and model capabilities must align together. Replacing a model does not automatically inherit GPT-5.6 Tool Search or Code Mode metadata. ## Source Index - [GPT-5.6 Sol capabilities and Tool Mode](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/models-manager/models.json#L4-L22) - [GPT-5.6 Terra Code Mode Only metadata](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/models-manager/models.json#L119-L137) - [GPT-5.6 Luna Code Mode Only metadata](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/models-manager/models.json#L232-L250) - [The adjacent GPT-5.5 entry does not select Code Mode Only](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/models-manager/models.json#L341-L359) - [Official OpenAI Tool Search guide: deferred loading, call sequence, and reuse in future turns](https://developers.openai.com/api/docs/guides/tools-tool-search) - [Official OpenAI Programmatic Tool Calling guide](https://developers.openai.com/api/docs/guides/tools-programmatic-tool-calling) - [Official Codex custom Model Provider configuration](https://learn.chatgpt.com/docs/config-file/config-advanced#custom-model-providers) - [Codex open-source component boundaries](https://learn.chatgpt.com/docs/open-source) - [Tool registry, exposure, and model-visible spec planning](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/tools/spec_plan.rs#L319-L486) - [Conditional registration of core tools](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/tools/spec_plan.rs#L818-L1118) - [Responses Lite `additional_tools` request framing](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/client.rs#L849-L885) - [Responses Lite test: internal header, omitted top-level fields, and input-item order](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/tests/suite/responses_lite.rs#L28-L139) - [`tool_search` BM25 engine and Top-K output](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/tools/handlers/tool_search.rs#L76-L169) - [How names, descriptions, and schemas become tool-search text](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/tools/src/tool_search.rs#L23-L150) - [How a `tool_search` result becomes a dedicated input item for the next request](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/tools/context.rs#L149-L185) - [How `tool_search_call` / `tool_search_output` remain in conversation history](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/context_manager/history.rs#L350-L386) - [Direct / Deferred registration of MCP tools](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/core/src/mcp_tool_exposure.rs#L17-L89) - [BM25 crate version: 2.3.2](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/Cargo.toml#L294) - [`bm25` 2.3.2 equation and default parameters](https://github.com/Michael-JB/bm25/blob/8ef726045b41702e148d8996d344f3500844fde1/src/embedder.rs#L146-L205) - [`bm25` 2.3.2 tokenizer splitting, normalization, stop words, and stemming](https://github.com/Michael-JB/bm25/blob/8ef726045b41702e148d8996d344f3500844fde1/src/default_tokenizer.rs#L263-L289) - [Current substring-query semantics of the Codex Memory Tool](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/ext/memories/src/tools/search.rs#L28-L62) - [Codex Memory local search and result ordering](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/ext/memories/src/local/search.rs#L17-L88) - [The stable release accepts only the Responses Wire API and rejects old `chat` configuration](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/model-provider-info/src/lib.rs#L49-L79) - [How an unknown model receives conservative fallback capability metadata](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/models-manager/src/model_info.rs#L137-L180) > Source checked on August 9, 2026. The stable release is pinned to `be6e8eac`; the same-day `main` check is pinned to `646f7c0a`; the BM25 crate is pinned to `v2.3.2` commit `8ef72604`. Tool exposure and model catalogs continue to evolve; debug a specific environment against its actual request and exact commit. > Image-generation note: the two language editions contain six technical figures in total, all generated or edited through Codex's built-in `image_gen.imagegen`. In the pinned source, that extension hard-codes [`IMAGE_MODEL` to `gpt-image-2`](https://github.com/openai/codex/blob/be6e8eac029b183056b7e4402879f15d2c85f61b/codex-rs/ext/image-generation/src/tool.rs#L53-L58) and uses it to construct the Images request. Mode state, Top-K annotations, illustrative IDs, and bilingual labels were manually checked before publication. --- title: "Five Codex Harness Designs Worth Copying After Reading the Source" description: "A source-grounded analysis of selective parallelism, Unified Exec, ExecCell, progressive Skills, persistent Goal continuation, and embedding the runtime through App Server." url: "https://zhuoqidev.com/en/posts/codex-agent-design/" language: "en" author: "Liu ZhuoQi" date: "2026-08-02" updated: "2026-08-03" series: "Agent Architecture Deep Dives" tags: ["Codex","OpenAI","AI Agent","Agent Harness","Tool Calling","Agent Runtime","Skills","Goal","App Server"] translation: "https://zhuoqidev.com/posts/codex-agent-design/index.md" --- # Five Codex Harness Designs Worth Copying After Reading the Source > **TL;DR** Think of Codex as a small construction crew. The model is the site lead deciding what should happen next. The agent harness is everything around that lead: dispatch desk, access control, job records, and the progress board. The source is valuable not merely because the lead can issue commands, but because the surrounding system keeps work safe, recoverable, and understandable to the customer. Many agent tutorials reduce the loop to this: ```python while True: response = model(messages, tools) if response.tool_calls: messages += execute(response.tool_calls) else: return response.text ``` The loop is valid, but it omits the hard product questions. Can several work orders start together? Who keeps custody of a test that runs for ten minutes? Should every status check interrupt the user? Could “still inspecting” be mistaken for delivery? Can the system find the right manual when the user does not know its name? When the user explicitly says “do not stop,” how does the job survive multiple shifts? For this article I read the official [openai/codex](https://github.com/openai/codex) Rust implementation at commit [`bb5054f`](https://github.com/openai/codex/commit/bb5054fe47abe73ecbbd454751066a28c89f4bb9). The component names, branches, and constants below come from that source rather than guesses from the UI. The product will continue to evolve, but the code already answers a durable engineering question: **what should an enterprise agent harness take off the model's shoulders?** ## Start Here: What Actually Happens Inside One Agent Turn? Treat one complete restaurant visit as a Thread—the tab for the entire case. Starter, main course, and dessert can be separate Turns. Each Turn is the complete cycle from “the user requests this round of work” to “the result of this round is actually delivered.” ![Restaurant-order metaphor explaining context assembly, model decisions, tool calls, tool results, verification, Final, and turn completed inside one Codex Agent Turn](https://zhuoqidev.com/_astro/agent-turn-anatomy-v1-en-4k.DAKLVs-n_Z1c2f1J.webp "One Turn resembles an order-to-delivery round: it may dispatch several tool work orders, or it may need no tool at all.") Following the diagram, a Turn usually contains six parts: 1. The user states the request while the runtime assembles conversation history, environment rules, available tools, and Skill/Goal state. 2. The model chooses a next action: progress commentary, a plan update, a Tool Call, or a direct Final. 3. Before a tool runs, the harness checks permission, isolation, parallel admission, and waiting rules, then dispatches the appropriate station. 4. Tool Results return evidence to the model. If work remains, the model decides again and may call more tools. One Turn can therefore contain zero, one, or many Tool Calls. 5. The user may steer an active Turn. Commentary is visible progress, but it does not close the lifecycle. 6. After verifying requirements and evidence, the model produces Final and the runtime emits `turn/completed`. Only then is this Turn closed. If a Goal remains active, thread idle may automatically begin another Turn. You do not need to memorize the Rust names that follow. First place each term in the picture—as context, work order, station, returned evidence, progress update, or delivery—and the source becomes much easier to follow. ## 1. Parallel Work Uses a Library-Style Access Gate Imagine a library. Many readers can enter the reading room together because they are only looking at books. If a librarian must move shelves and renumber the collection, everyone else briefly waits; otherwise a book could move while somebody is using it. Codex uses the same idea. One model response may contain several tool calls—several work orders—but the runtime does not blindly `spawn`, or start, all of them. [`ToolCallRuntime`](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/core/src/tools/parallel.rs) is the attendant at the door. `Arc>` is the shared access gate. The Rust spelling is unimportant; it offers two kinds of admission: - “Reader admission”: proven parallel-safe stations share the room and work together. - “Librarian admission”: a station that may affect shared state gets the only key, making everyone else wait. - A new handler—the station doing the work—has no parallel pass by default; safety must be declared explicitly. - `shell_command`, `exec_command`, and `write_stdin` are source examples that explicitly request the parallel pass. ![Codex ToolCallRuntime uses capability declarations and an RwLock to create selective concurrency while preserving original call order for model delivery](https://zhuoqidev.com/_astro/source-parallel-gate-v1-en-4k.DxKwacWP_Z8CBHJ.webp "Read the cyan lane as a shared reading room and the amber lane as a temporary closure while the librarian rearranges shelves.") The kitchen may finish ticket three before ticket one, but the waiter still returns results to the model in original ticket order. Tests explicitly verify this stable delivery. The timers along the bottom also have ordinary meanings. `dispatch duration` is queueing and assignment time. `handler duration` is time spent doing the actual work. `total duration` covers the entire trip from receipt to completion. A `CancellationToken` is a site-wide stop-work announcement; some stations are allowed to clean up before reporting `aborted by user`. The transferable design is not specifically Rust's `RwLock`. It is the contract: 1. Parallelism is a declared tool capability, not a privilege inferred from confidence. 2. The default is conservative; only proven-safe handlers opt in. 3. Execution may be asynchronous, while model context remains stable, ordered, and replayable. A prompt that says “parallelize when possible” cannot provide those guarantees. The model identifies potentially independent work; the harness owns the actual boundary. ## 2. Long Commands Work Like Taking a Number at a Service Desk At a public service counter, a quick request can finish while you wait. A request needing thirty minutes is first registered, then you receive a claim number. You can leave the counter and check later without losing the case or blocking everybody behind you. Codex [`unified_exec`](https://github.com/openai/codex/tree/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/core/src/unified_exec) turns long commands into that kind of registered case rather than a one-shot `command → stdout` exchange: 1. `ToolOrchestrator` is the intake desk: it checks approval and chooses an isolated work area, or sandbox. 2. A PTY or plain pipes are the communication line used to receive output or send characters. 3. `ProcessStore` is the case ledger. **Codex registers the process before yielding the counter**, so an interrupted turn cannot make the background job disappear. 4. A quick job returns its result. A long one returns a `session_id/process_id`—the claim number. 5. `write_stdin` means returning with that number. It can add input or ask “is it done?” with an empty poll. One case accepts one interaction at a time; different cases can be checked together. The desk also has a sensible checking schedule rather than asking every second. The source defines an initial observation window of 250–30,000ms, with a ten-second floor on Windows. A write waits at most thirty seconds; a status-only poll can quietly wait 5–300 seconds. If output is enormous, the model does not receive the entire paper roll. It gets about 10,000 tokens by default under a 1MiB cap. A head-tail buffer works like an incident summary that preserves “how it began” and “how it ended” while clearly marking the omitted middle. ![Codex Unified Exec lifecycle from approval and sandboxing through PTY, ProcessStore, yield, write_stdin recovery, and bounded output](https://zhuoqidev.com/_astro/source-unified-exec-lifecycle-v1-en-4k.DReIVS2M_8Jzrj.webp "Read ProcessStore as the case ledger, session_id as the claim number, and write_stdin as returning to check or supply more information.") Observability still does not mean handing the user every line from the printer. Codex TUI [`ExecCell`](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/tui/src/exec_cell/model.rs) behaves more like a parcel-tracking app: many scanner events can collapse into one understandable “in transit” card. The source groups only viewing operations—`Read`, `ListFiles`, and `Search`—under `Exploring`. Other commands stay separate. A `call_id` is the parcel tracking number: a completion event must return to its original card. An event without a matching number remains standalone rather than being attached to the wrong job and hiding unfinished work. ![Codex TUI parses command events into Read, List, Search, or Run, then uses call_id routing and bounded previews to render ExecCell](https://zhuoqidev.com/_astro/source-exec-cell-compression-v1-en-4k.BKRd1_OT_Z1uqU4v.webp "Like a parcel app collapsing many scans into “in transit,” the default is readable while every checkpoint remains available on expansion.") Consecutive reads become one row with unique filenames, repeated wait records may be suppressed, and live preview is capped at 50 lines and 1MiB with beginning and ending retained. Users see where the agent looked and what state it reached; they expand the full transcript only for diagnosis. Commentary follows the same rule. “Plumbing is complete; tiling has started” is a construction update, not the handover of the building. App Server [`AgentMessage.phase`](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/app-server-protocol/src/protocol/v2/item.rs) separates progress from final answer, and `turn/completed` confirms that one round is actually closed. **Text can reassure the user without forcing the agent to stop.** ## 3. Skills Work Like a Menu Plus an Operating Manual A restaurant menu tells you what dishes exist and what they are like; it does not print every complete recipe. The chef opens a recipe only after a dish is selected. Codex uses the same split. `SkillCatalog` is the menu. `SKILL.md` is the operating manual. The [`skills` extension](https://github.com/openai/codex/tree/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/ext/skills/src) combines Host, Executor, Orchestrator, and Plugin sources, then posts only names, descriptions, and manual locations on the model's noticeboard—called WorldState in the source. The noticeboard is finite. Long addresses receive aliases, long descriptions are shortened, and an overfull catalog explicitly says `N additional skills omitted`. A WorldState diff resembles posting only a changed notice instead of replacing the entire board every morning. The cheap-selector experiments in source currently measure in shadow; they do not secretly alter the model's menu. ![Codex Skills flow from multi-source discovery and bounded catalog rendering to model selection, progressive SKILL.md loading, and explicit or implicit invocation](https://zhuoqidev.com/_astro/source-skills-progressive-context-v1-en-4k.DZbej1gq_1vP4ld.webp "Treat the upper-left catalog as the menu and the lower-right workflow as the kitchen: choose cheaply first, then open the full manual.") [`catalog_prompt.rs`](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/ext/skills/src/catalog_prompt.rs) tells the model: if the user names a manual **or the task clearly matches a menu description**, use it for this turn. Read `SKILL.md` completely, then retrieve only the references, scripts, or templates that manual says are relevant. An explicit Skill selection delivers the complete manual at its exact address. If the model opens a `SKILL.md` itself or runs one of its scripts, [`invocation_utils.rs`](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/core-skills/src/invocation_utils.rs) works like a library checkout scanner: it recognizes which manual was actually used, deduplicates that implicit invocation within the turn, and records it. Detection does not mean every manual was preloaded. This is useful enterprise context engineering: users state outcomes, a low-cost directory helps the model discover the workflow, and domain rules consume context only when needed. The boundary remains explicit: **a Skill can change how an authorized task is performed; it does not expand permission scope.** ## 4. Goal Is a Project Order That Survives Shift Changes Imagine a repair project passed across three shifts. If the objective exists only on the first technician's sticky note, the next shift may conclude, “I fixed one part, so I am done.” A durable project order instead records the objective, budget, time spent, and acceptance criteria in one central file. “Finish the migration and do not stop before verification passes” asks for that kind of handoff. Leaving it only in chat history makes it vulnerable to long execution, retries, and compaction. The Codex [`goal` extension](https://github.com/openai/codex/tree/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/ext/goal/src) creates a persisted `ThreadGoal` project order containing objective, status, budget, usage, and time. `create_goal` opens the order, `get_goal` checks it, and `update_goal` records a terminal state. The rules prevent every small request from becoming an endless project: - Create a Goal only when the user or system/developer instructions explicitly request one; do not infer it from ordinary work. - Set `token_budget` only when explicitly requested. - Through `update_goal`, the model can write only `complete` or `blocked`. - Complete requires requirement-by-requirement evidence that the full objective is achieved. - Blocked requires the same blocker across at least three consecutive Goal turns and no remaining path to meaningful progress. ![Codex Goal flow from semantic admission through tools and persisted ThreadGoal state to continue_if_idle, accounting, and terminal evidence gates](https://zhuoqidev.com/_astro/source-goal-continuation-v1-en-4k.K6taJzne_2qKHUE.webp "Read ThreadGoal as the central project order and each Turn as one shift. Ending a shift does not erase the project.") The model does not continue by repeatedly remembering the phrase “do not stop.” When one shift ends and the thread becomes idle, [`on_thread_idle`](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/ext/goal/src/extension.rs) invokes `continue_if_idle()`. Think of it as the shift supervisor's checklist: is the project enabled, not deferred, still attached to a live worksite, and still active? If yes, the system hands the full objective, remaining budget, and acceptance checklist to the next shift, then `try_start_turn_if_idle()` opens another Goal turn. Final is therefore an end-of-shift report, not automatic project completion. Tool completion, turn stop, abort, and token events update the ledger. Reaching the budget resembles exhausting prepaid labor: the system sets `budget_limited`, stops new substantive work, and requests a status and next step rather than fake acceptance. A mutation lock is the rule that only one clerk edits the master ledger at a time, preventing accounting, outside changes, and automatic handoff from overwriting one another. This is the clean boundary between model intelligence and harness design. The model interprets an explicit persistent objective and constructs its structured form; the runtime ensures that state, continuation, metering, and termination evidence do not depend on model memory alone. ## 5. Enterprise Adoption Starts With “Who Decides, Who Enforces?” The implementation can be summarized as a division of responsibility: | Question | Model owns | Harness owns | | --- | --- | --- | | Which work may be independent? | Judge semantic dependencies | Issue parallel passes, close the gate when necessary, return stable result order | | When should long work be checked? | Choose meaningful checkpoints | Keep process custody, issue claim numbers, support cancellation, bound output | | What should the user see? | Write concise progress | Fold events into readable cards while retaining the full record | | When should domain rules load? | Select a Skill from catalog meaning | Bound the menu and supply the complete manual after selection | | Should work persist until an outcome? | Recognize explicit Goal intent | Preserve the project order, hand off shifts, meter budget, verify acceptance | To embed the same runtime in an IDE or enterprise product, think of [Codex App Server](https://learn.chatgpt.com/docs/app-server) as a standardized front desk. Your product need not rebuild the entire workshop; it submits work through standard forms and receives progress, approval requests, and results. Technically those forms use bidirectional JSON-RPC and expose Threads (whole cases), Turns (rounds of work), Items (events or artifacts), Skills, Goals, and approvals. ```mermaid flowchart TD A["initialize"] --> B1["thread/start"] & B2["thread/resume"] B1 & B2 --> C["turn/start"] C --> D["consume item/*, plan, diff,
approval, and delta events"] D --> E["close the active UI lifecycle
only on turn/completed"] classDef out fill:#d3f9d8,stroke:#2f9e44 class E out ``` Clients can use `skills/list` to build a capability picker or send a structured `skill` item in `turn/start`. They can manage Goals through `thread/goal/*` and correct active work with `turn/steer`. The remote WebSocket transport remains experimental; production clients should pin the Codex version and generate TypeScript types or JSON Schema from that exact build. The durable conclusion is simple: > Let the model make semantic judgments. Turn concurrency, time, state, permission, observability, and termination evidence into protocols the harness can enforce. That will improve reliability and user experience more than merely adding tools. Before shipping, an enterprise agent team should be able to answer: Where is concurrency admitted? Who persists long processes? How is duplicate polling suppressed? What separates progress from Final? How do domain rules enter context only when needed? How does “do not stop” survive turns? Which evidence supports completion? ## Source Index - [Selective parallelism: parallel.rs](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/core/src/tools/parallel.rs), [router.rs](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/core/src/tools/router.rs), and [registry.rs](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/core/src/tools/registry.rs) - [Unified Exec](https://github.com/openai/codex/tree/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/core/src/unified_exec) - [TUI ExecCell](https://github.com/openai/codex/tree/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/tui/src/exec_cell) and [command_lifecycle.rs](https://github.com/openai/codex/blob/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/tui/src/chatwidget/command_lifecycle.rs) - [Skills extension](https://github.com/openai/codex/tree/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/ext/skills/src) and [core-skills](https://github.com/openai/codex/tree/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/core-skills/src) - [Goal extension](https://github.com/openai/codex/tree/bb5054fe47abe73ecbbd454751066a28c89f4bb9/codex-rs/ext/goal/src) - [Codex App Server](https://learn.chatgpt.com/docs/app-server) > Source and documentation checked on 2026-08-03. This article explains the public implementation at the pinned commit, not a permanent API guarantee for every future release. --- title: "How Agents Remember You: Human Memory Science and a Code Audit of Six Open-Source Systems" description: "From Ebbinghaus, H.M., working memory, and engrams to Mem0, Letta, Graphiti, LangMem, Cognee, and MemoryOS: a history of memory paradigms and a code-level comparison of what open-source agent memory systems actually implement." url: "https://zhuoqidev.com/en/posts/llm-memory-research/" language: "en" author: "Liu ZhuoQi" date: "2026-07-30" updated: "2026-07-30" tags: ["AI Agent","LLM","Memory","Memory Systems","Cognitive Science","Open-Source Architecture"] translation: "https://zhuoqidev.com/posts/llm-memory-research/index.md" --- # How Agents Remember You: Human Memory Science and a Code Audit of Six Open-Source Systems Almost every agent project now claims to provide “long-term memory.” For one project, that means embedding chat history. For another, it means maintaining a user profile. A third lets the model edit Markdown files. A fourth builds a bitemporal knowledge graph. All four use the word *memory*, but they are not the same system and should not be placed on one undifferentiated leaderboard. ![From biological memory traces to an agent memory stack](https://zhuoqidev.com/_astro/agent-memory-cover-v3-4k.BrXnIPjb_1CQ2TV.webp) To decide whether a system genuinely remembers, I would rather ask three questions: 1. **After an experience, which state in the system actually changes?** 2. **Where does that state live, who may modify it, and when does it expire?** 3. **Before the next action, how is it brought back accurately and with the right permissions?** This article starts from those questions. The first half places the history of human memory science beside the evolution of agent memory. The second half reads the code behind Mem0, Letta, Graphiti, LangMem, Cognee, and MemoryOS, comparing their claims, actual data paths, system boundaries, and memory paradigms. > [!TIP] > **The short version:** mainstream agents have not acquired a single, brain-like “memory organ.” What works in engineering is a lifecycle: **experience → write gate → representation → storage → retrieval → context assembly → action and feedback → consolidation / revision / forgetting**. Open-source projects differ mainly in which parts of this loop they choose to own. The first diagram is not the component architecture of a particular product. It is the **shared coordinate system** for the rest of the article. Its question is not merely “where is data stored?” but “how does a past experience alter a future action?” First follow the seven-step loop in the center from experience to action. Then use the three carriers on the left to distinguish the current task, cross-session memory, and real-world state. The cards on the right explain each transformation, while the bottom row shows consolidation, revision, and forgetting over the system’s lifetime. This prevents databases, context, caches, and source-of-truth state from all being mislabeled as “memory.” ![A scientific systems map of Agent memory: seven loop stages, three carriers, and three governance outcomes](https://zhuoqidev.com/_astro/memory-loop-systems-map-v3-en-4k.BVmWXdEu_DbxIG.webp "Figure 1. This establishes the article’s working definition of a memory system. The central loop shows the online behavior path, the left column separates working memory, long-term memory, and world state, and the bottom row shows lifecycle governance. A database owns only step four; without write decisions, retrieval, context assembly, conflict handling, and feedback, more storage is merely more logging.") --- ## 1. Separate the Five Things Most Often Called “Memory” An LLM system contains at least five physically distinct state carriers. They differ in location, write speed, lifetime, governance, and retrieval semantics. This diagram exists to **disambiguate the vocabulary**. Read across to compare the five carriers, then down through writer, lifetime, strengths, and limitations. The goal is not to pick one universal winner. It is to prevent architectural category errors: treating a compute cache as durable memory, treating context as persistence, or copying real business state into a natural-language recollection that can go stale. ![Model weights, context, KV cache, external memory, and world state compared](https://zhuoqidev.com/_astro/memory-carriers-v4-en-4k.CjWWd3MU_2y4PO.webp "Figure 2. This answers where state actually lives. Five different things share the word memory, and one of the most dangerous design mistakes is treating two of them as interchangeable.") ### 1.1 Parametric memory: model weights Pretraining and fine-tuning write statistical regularities into parameters. This layer has enormous capacity and strong generalization, but writes are slow, precise deletion is difficult, and provenance is weak: the system usually cannot answer which experience produced a particular piece of knowledge. Weights are appropriate for language ability, general world knowledge, and stable skills. They are a poor fit for per-user updates after every conversation. Continually fine-tuning user preferences into weights is not only expensive; it also creates catastrophic-forgetting, tenant-isolation, deletion, and audit problems. ### 1.2 Working memory: the context window The current system prompt, conversation, tool results, scratchpad, and retrieved passages all live here. The model can attend to them directly, making context the strongest workspace available at inference time. But context does not persist across calls by itself. A longer window is only a larger desk for the current call. It does not automatically decide what deserves to survive, nor does it build a stable user model. ### 1.3 Compute cache: KV cache and prompt cache The KV cache stores attention keys and values that have already been computed. Prompt caching reuses prefill work for an identical prefix. Both reduce repeated computation, but neither decides what information matters or produces editable, retrievable memory records. Therefore: - A cache hit may mean nothing was “remembered”; the service merely avoided recomputation. - Cache expiry does not imply that long-term memory was lost. - Updating a memory inside the prompt prefix may itself cause a cache miss. **Caching is a performance mechanism. Memory is a state-governance mechanism.** ### 1.4 External long-term memory: files, SQL, vectors, and graphs This is where most agent memory engineering happens today. External stores can isolate users, preserve provenance, support deletion, and retrieve state into the next context. But “put it in a vector database” is not equivalent to “build a memory system.” Vector search provides approximate similarity. It does not inherently solve factual conflict, temporal truth, importance, authorization, bad writes, or forgetting. ### 1.5 Environmental memory: Git, CRM, calendars, and real world state Much information should never be copied into a natural-language memory record. Whether code was deployed, an invoice was paid, or a meeting was rescheduled should normally be queried from its source of truth. A reliable agent distinguishes: - **What should be recalled:** preferences, prior decisions, successful experience. - **What should be queried:** orders, permissions, inventory, calendars, code state. - **What should be recomputed:** prices, aggregates, and derived metrics. This is why “embed everything” often produces a system with plenty of information but unreliable facts. --- ## 2. How Human Memory Became a Systems Problem Comparing a vector database to the hippocampus or context to working memory can be pedagogically useful. It is not structural equivalence. The most important lesson from more than a century of memory research is precisely that **memory is neither one location nor an immutable file written once.** The historical diagram is not background decoration. It explains why this article rejects the model “memory = storage.” You do not need to memorize every date. Follow the three conceptual shifts at the bottom: from one warehouse, to separable systems, to a dynamic process reconstructed during retrieval. The later discussion of episodes, facts, procedures, consolidation, and revision follows directly from that progression. ![A history of human memory science, from the forgetting curve to engrams](https://zhuoqidev.com/_astro/human-memory-history-v4-en-4k.26qCxLze_Z26OiDU.webp "Figure 3. This explains the origin of the article’s memory paradigm. Research gradually replaced the idea of one storage location with multiple systems that jointly encode, consolidate, retrieve, and reconstruct.") ### 1885: Ebbinghaus made memory measurable Hermann Ebbinghaus repeatedly learned nonsense syllables and used the *savings method* to measure forgetting. Even when direct recall failed, relearning was faster. Memory moved from philosophical speculation to an experimental object that could be plotted and compared across intervals and repetitions. Many current agent-memory evaluations use a rougher measure than Ebbinghaus: final question accuracy. A useful evaluation should also ask: - How soon after writing is a memory available? - Does repeated successful retrieval stabilize it? - When a fact is superseded, does the old version still reappear? - When evidence is insufficient, can the system abstain? Primary source: [Ebbinghaus, *Memory: A Contribution to Experimental Psychology* (1885/1913)](https://psychclassics.yorku.ca/Ebbinghaus/) ### 1900: memory requires consolidation Georg Elias Müller and Alfons Pilzecker found that material learned immediately after a new item increased interference. They proposed that memory traces require time to stabilize, helping establish *consolidation* as a central concept. For agents, the lesson is not simply “run a nightly cron job.” It is to separate: - **Raw experience:** complete, traceable, and preferably append-only. - **Consolidated products:** profiles, facts, rules, and summaries that may be revised or overturned. If only the second layer survives, one faulty model summary can rewrite history. If only raw events survive, retrieval drowns in low-value detail. Primary source: [Müller & Pilzecker, *Experimentelle Beiträge zur Lehre vom Gedächtniss* (1900)](https://books.google.com/books?id=5RdCAQAAMAAJ) ### 1949: Hebb located persistence in changing connections Donald Hebb proposed cell assemblies and changes in connection efficiency driven by co-activation. The familiar phrase “fire together, wire together” is not a verbatim quotation, but it captures the direction: experience leaves a trace through network plasticity. This helps distinguish three changes in an agent system: - Putting an experience into context changes **activation state**. - Writing it to persistent storage changes **system state**. - Updating model weights changes **parameters**, a slower and less governable process. ### 1957: H.M. showed that memory is not one faculty Scoville and Milner reported that patient H.M. developed severe anterograde amnesia after bilateral medial temporal-lobe surgery, while short-term retention and some forms of skill learning were not impaired in the same way. The case broke the intuition that memory was a single capacity. The architectural lesson remains powerful: do not make one collection carry current task state, historical episodes, user facts, and executable skills at the same time. Primary paper: [Scoville & Milner, “Loss of Recent Memory after Bilateral Hippocampal Lesions” (1957)](https://pmc.ncbi.nlm.nih.gov/articles/PMC497229/) ### The 1970s: episodic, semantic, procedural, and working memory diverged Endel Tulving distinguished: - **Episodic memory:** what happened to me, where, and when. - **Semantic memory:** what I know independent of a particular episode. Baddeley and Hitch replaced a single short-term store with a multicomponent working-memory model. In parallel, the separation between skill learning and declarative knowledge helped establish procedural memory as another category. This classification remains more useful for agent architecture than “short-term versus long-term”: | Human category | Agent analogue | Typical storage | Typical read path | |---|---|---|---| | Working memory | Current goal, plan, intermediate state, tool output | Context / graph state / scratchpad | Direct injection at every step | | Episodic memory | Conversations, actions, failures, observations | Event log + temporal index | Joint retrieval by time, entity, and similarity | | Semantic memory | Preferences, stable facts, concepts, relations | Profile / KV / vector / knowledge graph | Exact key, semantic, or graph query | | Procedural memory | Prompts, rules, skills, successful trajectories | Files / version control / skill registry | Task routing or explicit mounting | **Short-term versus long-term describes lifetime. Episodic, semantic, and procedural describes content and function.** These dimensions are not substitutes. Primary sources: [Tulving, “Episodic and Semantic Memory” (1972)](https://cir.nii.ac.jp/crid/1574231874408386176?lang=en); [Baddeley & Hitch, “Working Memory” (1974)](https://doi.org/10.1016/S0079-7421%2808%2960452-1) ### 1971–2012: from spatial representation to manipulable engrams O'Keefe discovered hippocampal place cells. Later work on grid cells and related systems exposed neural mechanisms of spatial representation. In 2012, Liu, Ramirez, Tonegawa, and colleagues used optogenetics to reactivate hippocampal cells tagged during fear-memory formation and elicited behavior associated with memory recall. This did not reveal one address containing an entire memory. Modern engram research points instead to distributed, reactivatable cell assemblies whose content still depends on cross-region networks and retrieval conditions. Sources: [2014 Nobel Prize scientific background](https://www.nobelprize.org/prizes/medicine/2014/advanced-information/); [Liu et al., “Optogenetic stimulation of a hippocampal engram activates fear memory recall” (2012)](https://pmc.ncbi.nlm.nih.gov/articles/PMC3331914/) ### 2000: retrieval is not read-only Experiments by Nader, Schafe, and LeDoux showed that a consolidated fear memory becomes plastic after reactivation and again requires protein synthesis to stabilize. This result helped launch modern research on reconsolidation. For agent systems, the useful analogy is that **every recall can become an update.** If a user says, “I no longer drink coffee,” the system should not leave two contradictory preferences beside each other in a vector store. At minimum it should represent: ```text old_fact: user likes coffee validity: 2025-03 → 2026-07 new_fact: user avoids coffee source: conversation/event/... relation: new_fact supersedes old_fact ``` Primary paper: [Nader, Schafe & LeDoux, “Fear memories require protein synthesis in the amygdala for reconsolidation after retrieval” (2000)](https://pubmed.ncbi.nlm.nih.gov/10963596/) ### Four principles worth borrowing from memory science 1. **Memory is a collection of systems, not one vector store.** 2. **Consolidation transforms events into stable representations; it is not merely text compression.** 3. **Retrieval is reconstructive, so provenance and versions must survive.** 4. **Forgetting is not only failure; it also reduces interference, controls cost, and protects privacy.** The analogy must stop there. The hippocampus is not Redis. Vector similarity is not a complete model of associative recall. An LLM summary is not sleep-dependent consolidation. Neuroscience analogies should generate engineering questions, not replace evidence. --- ## 3. How Agent Memory Evolved The human-memory timeline explains why we ask these questions. The agent timeline explains why current systems have their present shape. The important feature is not the list of model names but the migration of the state boundary: from programs and network dynamics, to context, to retrieved external data, and finally to a dedicated memory layer that owns writing, time, permissions, and deletion. ![The evolution of agent memory from symbolic state and LSTM to memory engineering](https://zhuoqidev.com/_astro/agent-memory-history-v4-en-4k.BO9sZdZ1_ZQyBdS.webp "Figure 4. This locates the current engineering stage. Competition has shifted from “can state be preserved?” to “what gets written, when is it recalled, how is it revised, and who may delete it?”") ### Stage 1: state lived in programs Early symbolic AI and cognitive architectures already had working memory, production rules, and long-term knowledge. Programmers defined both state and representation. These systems addressed how a reasoning process maintains state, not natural-language personalization. ### Stage 2: neural networks learned to preserve and address state LSTM used gated recurrence to reduce long-range dependency problems. The 2014 Neural Turing Machine connected a network to a differentiable memory matrix with learned read and write heads. The goal was to learn algorithms such as copying, sorting, and associative recall end to end. Primary paper: [Neural Turing Machines (2014)](https://arxiv.org/abs/1410.5401) This line of work put memory inside model architecture, but training difficulty, scale, and weak governance limited its use as a general per-user agent memory layer. ### Stage 3: Transformer context became a universal workspace Transformers allowed every token position to interact directly with every other position. Prompting became a uniform interface: rules, examples, documents, and tool results could all be supplied at inference time without modifying weights. The cost was that every API call still began from a new context by default. “LLMs are stateless” is better stated as: **the model API makes no cross-call state guarantee on behalf of the application.** ### Stage 4: RAG connected non-parametric memory to generation RAG combined parametric generation with a retrievable non-parametric corpus. It was designed for knowledge-intensive tasks and updatable sources, not personal memory, but retrieve-then-generate quickly became the default read path for long-term agent memory. Primary paper: [Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020)](https://arxiv.org/abs/2005.11401) One boundary matters: **RAG is a read mechanism, not a complete memory system.** If data never enters through experience-driven writing, updating, conflict resolution, or forgetting, it is closer to an external knowledge base. ### Stage 5: 2023 combined writing, reflection, hierarchy, and skills Several 2023 systems filled different gaps: - **Generative Agents:** an event stream, recency/relevance/importance retrieval, and reflection that consolidates episodes into higher-level beliefs. - **Voyager:** successful code becomes a reusable skill library, emphasizing procedural memory. - **MemGPT:** an operating-system analogy treats context as a working set and lets the agent move information between memory tiers through tools. - **CoALA:** a cognitive architecture connecting working, episodic, semantic, and procedural memory to an agent decision loop. Primary papers: [Generative Agents](https://arxiv.org/abs/2304.03442) · [Voyager](https://arxiv.org/abs/2305.16291) · [MemGPT](https://arxiv.org/abs/2310.08560) · [CoALA](https://arxiv.org/abs/2309.02427) ### Stage 6: from 2024 onward, governance became the differentiator The dividing line is no longer whether a project supports vector search. It is: - Who decides to write? - Is memory represented as events, facts, documents, graph edges, prompts, or executable skills? - Are conflicts overwritten, coexisting, or invalidated over time? - Is provenance preserved? - Does consolidation happen in the background? - Can state be isolated by user, agent, run, and tenant? - Can users inspect, edit, export, and delete it? - Can operators observe memory failures? Those questions define the code audit below. --- ## 4. How to Read an Open-Source “Agent Memory” Project I did not rank projects by their home-page benchmark. Scores depend on the base model, answer prompt, judge model, retrieval budget, and data cleaning. A high memory-QA score says little about permissions, deletion, stability, cost, or whether the architecture fits production. This audit is anchored to repository states visible on 2026-07-30 and examines seven dimensions: 1. **Write path:** trigger, gating, deduplication, and structured extraction. 2. **Representation:** events, facts, profiles, graphs, prompts, or skills. 3. **Storage abstraction:** files, SQL, vectors, graphs, and replaceability. 4. **Read path:** exact search, vector search, BM25, graph traversal, reranking. 5. **Time and conflict:** overwrite, invalidation, versioning, or bitemporal modeling. 6. **System boundary:** library, engine, toolkit, or full runtime with API and tenancy. 7. **Loop completeness:** consolidation, feedback, forgetting, deletion, and observability. The next diagram is a **selection map**, not a logo wall or an overall score. Read each row from left to right: public claim, observed code path, system boundary, and memory paradigm. That makes it possible to distinguish an SDK, runtime, temporal graph engine, framework toolkit, knowledge pipeline, and research implementation before committing to the detailed audit. ![System boundaries and memory paradigms across six open-source agent-memory projects](https://zhuoqidev.com/_astro/open-source-memory-architectures-v4-en-4k.BSp9slA5_hm6iB.webp "Figure 5. This shortens the project-selection path rather than naming an overall winner. A full runtime is heavier; a small toolkit is easier to embed. The important question is whether the boundary matches the desired memory paradigm.") ### Summary: claims, code paths, and memory paradigms | Project | Public positioning | Observed core code path | Closest memory paradigm | Architecture type | Main boundary | |---|---|---|---|---|---| | **Mem0** | Universal memory layer, personalization, cross-session learning | History + vector recall → LLM incremental fact extraction → batch embeddings → vector store; SQLite for messages/history | Primarily semantic facts with user/agent scopes | Pluggable memory SDK | Runtime, task state, and full governance sit outside the core | | **Letta** | Stateful, self-improving agents with advanced memory | AgentState + memory blocks + messages/passages + context calculator + agent loop/tools | Working + episodic + semantic; model-managed | Full stateful agent runtime | Adopting it often means adopting its runtime model | | **Graphiti** | Real-time temporal context graph and historical truth | Episode → entity/fact extraction → bitemporal edges → semantic/BM25/graph hybrid search | Temporal semantic memory with episodic provenance | Temporal graph engine | User, session, and agent services are separate | | **LangMem** | Continuous learning, hot-path tools, background memory | Manage/search tools + background manager + LangGraph BaseStore + prompt optimizer | Semantic/episodic templates + procedural memory | Framework toolkit | Persistence, deployment, and permissions inherit from LangGraph or custom code | | **Cognee** | Turn data into AI memory and replace traditional RAG | Add → cognify pipeline → graph/vector/relational storage → search/memify | Enterprise semantic memory and knowledge graph | Knowledge pipeline / infrastructure | Personal conversation memory is not the sole center | | **MemoryOS** | OS-style short-, mid-, and long-term hierarchy | Short-term QA queue → mid-term segment/heat → profile and knowledge extraction → JSON/embedding retrieval | Hierarchical episodic-to-semantic consolidation | Research reference implementation | Production tenancy, transactions, and governance need additional work | --- ## 5. Six Systems: What Exists Between Marketing and Code ### 5.1 Mem0: a fact-distillation and retrieval pipeline, not a brain Mem0 has a clear role: add a unified long-term memory API to an existing application. Its surface centers on `add / search / get / update / delete`, with providers for LLMs, embeddings, vector stores, and rerankers. The current OSS Python v3 write path is visible in [`mem0/memory/main.py`](https://github.com/mem0ai/mem0/blob/9c2d6222ce86bf6a73ae7ca97464a8e1a55ab3ca/mem0/memory/main.py): 1. Establish scope from `user_id / agent_id / run_id`. 2. Read recent messages from SQLite. 3. Recall existing memories from the vector store using the current conversation. 4. Give old memories, new messages, and recent context to an LLM for incremental extraction. 5. Batch-embed the extracted memory texts. 6. Write them back and record history. The core operation is not raw chat storage. It is **LLM-driven distillation of conversation into shorter retrievable facts**. **Where the claim holds:** - Low integration cost. - Mature provider abstraction. - Practical scope, metadata, history, async, and reranking interfaces. - A strong fit for preferences, identity facts, and previous decisions. **Where the claim can mislead:** - “Universal” does not mean optimal for every memory type. - The default center is semantic facts, not full working memory or a skill system. - LLM extraction can omit, misattribute, or overgeneralize at write time. - Vector similarity does not answer complex historical-truth questions by itself. > [!NOTE] > **Paradigm:** an external memory layer centered on semantic memory. ### 5.2 Letta: memory as the state model of an agent runtime Letta grew out of MemGPT. Its largest difference from Mem0 is not retrieval quality but system boundary. [`AgentState`](https://github.com/letta-ai/letta/blob/b76da9092518cbaa2d09042e52fdcbde69243e18/letta/schemas/agent.py), [`Memory`](https://github.com/letta-ai/letta/blob/b76da9092518cbaa2d09042e52fdcbde69243e18/letta/schemas/memory.py), [`Passage`](https://github.com/letta-ai/letta/blob/b76da9092518cbaa2d09042e52fdcbde69243e18/letta/schemas/passage.py), and [`agent_loop.py`](https://github.com/letta-ai/letta/blob/b76da9092518cbaa2d09042e52fdcbde69243e18/letta/agents/agent_loop.py) show that: - Memory blocks are part of agent state. - Messages, passages, tools, model configuration, and agent identity persist together. - A context-window calculator decides which state enters each turn. - The agent may modify its own memory through tools. - Server, API, ORM, and multi-agent groups live inside one runtime model. **Where the claim holds:** - It is genuinely a stateful agent platform, not a vector wrapper. - Memory, agent loop, tools, and context budgeting are integrated. - It fits long-running agents that actively maintain their own state. **The trade-off:** - You adopt an agent runtime, not only a memory library. - Model-managed writing expands the surface for prompt injection, bad writes, and permission errors. - A complete runtime can be more system than a narrow application needs. > [!NOTE] > **Paradigm:** OS-style hierarchical and model-managed memory spanning working, episodic, and semantic state; tools, files, and skills carry more of the procedural layer. ### 5.3 Graphiti: time and provenance are the product, not merely “a graph” Many knowledge-graph projects store `subject - predicate - object`. Graphiti differentiates itself through episode provenance and bitemporal relations. [`graphiti_core/edges.py`](https://github.com/getzep/graphiti/blob/2645dee20fd71797a61e1c6177a93cccd5584574/graphiti_core/edges.py) and [`graphiti.py`](https://github.com/getzep/graphiti/blob/2645dee20fd71797a61e1c6177a93cccd5584574/graphiti_core/graphiti.py) show: - Episodes preserve source input and provenance. - Entity nodes represent people, objects, organizations, and concepts. - Entity edges represent facts and relations. - `valid_at / invalid_at` describe when a fact is true in the world. - `created_at / expired_at` describe when the system learned and invalidated it. - Search recipes combine semantic search, BM25, graph traversal, and reranking. The model can therefore answer different questions: - What is true now? - What was true in March 2025? - When did the system learn that it changed? - Which episode produced this edge? **Where the claim holds:** - Time and provenance exist in the data model, not only in a prompt instruction. - The design is valuable for changing relations, multihop queries, and auditability. - Graph backends and search recipes have explicit abstractions. **Boundary:** - Open-source Graphiti is an engine, not a complete user/session/agent product. - Graph construction still relies on LLM extraction, so schema and model quality directly affect write correctness. > [!NOTE] > **Paradigm:** temporal semantic memory with episodic provenance. ### 5.4 LangMem: composable primitives rather than a memory server LangMem packages common memory operations into composable tools: - `manage_memory` and `search_memory` in the hot path. - A background manager for extraction, merging, and updates. - Profile and collection forms of semantic memory. - Procedural memory through prompt optimization from successful and failed trajectories. - Persistence through LangGraph `BaseStore`. The core implementation is visible in [`knowledge/extraction.py`](https://github.com/langchain-ai/langmem/blob/56d85939d80bb731bd5e237567148d817d7bfd16/src/langmem/knowledge/extraction.py) and [`prompts/optimization.py`](https://github.com/langchain-ai/langmem/blob/56d85939d80bb731bd5e237567148d817d7bfd16/src/langmem/prompts/optimization.py). **Where the claim holds:** - It supports both in-the-loop and background writing. - Procedural memory has a real prompt optimizer behind it. - Projects already using LangGraph get high composability. **Boundary:** - It is not an independent production database, user system, or agent server. - `InMemoryStore` examples disappear on restart; production needs Postgres or another durable `BaseStore`. - Consistency, permissions, deletion, and observability depend on the surrounding platform or custom implementation. > [!NOTE] > **Paradigm:** memory primitives centered on semantic and procedural memory. ### 5.5 Cognee: knowledge infrastructure rather than preference memory Cognee describes its product as converting raw data into AI memory. The mature code path resembles an ECL knowledge pipeline: ```mermaid flowchart TD A["add"] --> B["classify / chunk"] --> C["cognify
LLM entity and relation extraction"] --> D["graph + vector + relational storage"] --> E["search / memify"] classDef out fill:#d3f9d8,stroke:#2f9e44 class E out ``` [`cognify.py`](https://github.com/topoteretes/cognee/blob/88aa09b4e3289e3dbf12c0c090080920816e2fb7/cognee/api/v1/cognify/cognify.py) orchestrates the pipeline. The storage layer exposes graph and vector interfaces, while upper layers include datasets, users, roles, and ACLs. **Where the claim holds:** - Data ingestion, pipelines, and graph/vector/relational adapters are substantial. - It supports multiple search types, ontology work, and multi-tenant permissions. - It is attractive for durable knowledge built from documents, code, and enterprise data. **What to calibrate:** - Its strongest paradigm is semantic knowledge infrastructure. - A full cognify pipeline may be excessive for “remember that this user dislikes cilantro.” - It is more appropriate than a chat-memory SDK when sources are heterogeneous, relations matter, and access control is central. > [!NOTE] > **Paradigm:** graph-structured semantic or organizational knowledge memory. ### 5.6 MemoryOS: the clearest cognitive analogy, still a research-oriented implementation MemoryOS makes short-, mid-, and long-term tiers explicit: - Short-term memory stores recent question-answer pairs. - Capacity pressure migrates content into mid-term session segments. - Segments carry heat. - High heat triggers LLM updates to the user profile, user knowledge, and assistant knowledge. - A retriever searches mid-term pages and long-term knowledge before assembling the generation prompt. The path is readable in [`memoryos-pypi/memoryos.py`](https://github.com/BAI-LAB/MemoryOS/blob/587ed7755c7aed179965792830ff1b5ad9a6fa92/memoryos-pypi/memoryos.py). **Where the claim holds:** - Hierarchy, migration, heat, and consolidation are explicitly implemented. - It is useful for reproducing experiments on episodic-to-semantic consolidation. - The code is direct enough for researchers to modify strategies. **Code-level reality:** - The default implementation relies heavily on local JSON, SentenceTransformer, and LLM calls. - Similar modules remain across `memoryos-pypi`, `memoryos-playground`, `memoryos-chromadb`, and `memoryos-mcp`. - Transactions, concurrency, tenant isolation, unified schemas, migrations, monitoring, and fine-grained deletion require application work. That does not make the project “bad.” It means the deliverable is a research reference, not the same product category as a full platform. > [!NOTE] > **Paradigm:** hierarchical episodic memory consolidated into profiles and semantic knowledge. --- ## 6. A Fair Comparison Uses Capability Surfaces, Not One Score | Capability | Mem0 | Letta | Graphiti | LangMem | Cognee | MemoryOS | |---|---|---|---|---|---|---| | Current task state | External runtime | **Core capability** | Not central | LangGraph | Not central | Partially covered by short-term tier | | Episodic events | Application may retain them; core favors distilled facts | Messages / passages | **Episodes are first-class** | Schema-based extraction | Can ingest | **Core short/mid-term layer** | | Semantic facts / profile | **Core capability** | Memory blocks | Entity/fact graph | **Core capability** | **Core capability** | Long-term layer | | Procedural memory | Agent/procedural paths exist but are not the center | Tools / files / skills | Not central | **Prompt optimizer** | Extensible through rules / memify | Not central | | Temporal conflict | Extraction and metadata policy | Agent/application policy | **Native bitemporal model** | Schema/manager policy | Temporal search depends on model | Profile merging and heat migration | | Replaceable storage | **Strong** | Platform persistence model | Replaceable graph backend | Replaceable `BaseStore` | **Strong graph/vector/relational adapters** | Multiple distributions | | Full agent runtime | No | **Yes** | No | No | No | Research runtime with generation | | Natural use | Add memory APIs to an existing app | Build a long-lived stateful agent | Add time-aware graphs for changing relations | Compose memory policy in LangGraph | Build organizational knowledge memory | Research and reproduce experiments | Bold indicates where a project concentrates complexity, not universal superiority. ### Why benchmarks cannot replace architecture Benchmarks such as LoCoMo and LongMemEval are valuable, but they mostly test whether an answer uses conversation history. Production systems also face: - **Bad writes:** an LLM stores an inference as a user fact. - **Memory inflation:** every turn produces repeated low-value records. - **Stale resurrection:** an expired but semantically similar fact ranks highly. - **Cross-user leakage:** a scope or filter is missing. - **Memory poisoning:** external content persuades an agent to persist malicious instructions. - **Incomplete deletion:** summaries, vectors, graph edges, and caches survive source deletion. - **Weak explainability:** the answer cannot identify which memory influenced it. - **Runaway cost:** extraction, embeddings, reranking, and graph construction accumulate every turn. A system can win LoCoMo and still be unsuitable for healthcare, finance, or multi-tenant SaaS. --- ## 7. Choosing a Memory Paradigm ### Scenario A: coding agents, personal tools, and a few hundred stable rules Start with: ```text Markdown / JSON + explicit namespaces + Git history + BM25 or simple full-text search ``` Files are readable, diffable, and reviewable. File memory is not an outdated vector database. For small datasets with exact terminology and rules that must load deterministically, it is often more reliable. Add embeddings only when cross-language paraphrase or thousands of records make lexical retrieval insufficient. ### Scenario B: chat assistants, support, and light personalization Start with a Mem0- or LangMem-style path: ```mermaid flowchart TD A["conversation"] --> B["write gate"] --> C["fact / profile extraction"] --> D["user-scoped store"] --> E["semantic retrieval"] classDef out fill:#d3f9d8,stroke:#2f9e44 class E out ``` The vector database is not the first design choice. Decide: - What must never be written? - Can the user inspect and delete it? - How do new preferences supersede old ones? - When uncertain, should the system preserve only the raw episode? ### Scenario C: long-lived autonomy and model-managed state Use a Letta-style runtime when agent identity, memory blocks, message persistence, context budgeting, tool permissions, and the agent loop must work as one system. ### Scenario D: changing facts and historical-state questions Use a Graphiti-style temporal graph when questions include: - Which contract version applied previously? - When did a person move from Team A to Team B? - Which version of a fact was known when a decision was made? Increasing top-k from 5 to 20 does not solve temporal truth. ### Scenario E: enterprise documents, heterogeneous sources, relations, and permissions Use a Cognee-style knowledge pipeline, or build a graph + vector + SQL layer on an existing data platform. Here memory means continuously updated, searchable, permissioned organizational knowledge—not merely conversational recall. ### Scenario F: research on hierarchy, heat, consolidation, and forgetting MemoryOS is a readable experimental baseline. A paper-oriented reference implementation should not be treated as a high-concurrency multi-tenant service without substantial engineering. --- ## 8. A Production-Ready Agent Memory Layer The earlier figures define concepts and compare projects. This one is the **implementation blueprint**. Read it from top to bottom: the top row is the online read/write path for one request; the middle row separates persistence by memory type; the bottom row covers source lineage, consolidation, conflict revision, and deletion. It is not a mandatory component list. Its job is to make sure a production design assigns every critical lifecycle responsibility. ![A production agent-memory layer spanning writing, typed stores, retrieval, filtering, and maintenance](https://zhuoqidev.com/_astro/production-memory-blueprint-v4-en-4k.BGZWsnSw_ZeWnAX.webp "Figure 6. This turns the article’s conclusions into an implementation checklist. Production memory is not one vector store; it is an entire layer from raw events and write gating through recall, permission filtering, context assembly, and background governance.") ### 8.1 Separate raw events from derived memory ```mermaid flowchart LR EL["event_log
immutable, auditable"] --> e1["conversation"] & e2["tool_result"] & e3["user_correction"] & e4["environment_observation"] DM["derived_memory
mutable, invalidatable"] --> d1["profile_fact"] & d2["episodic_summary"] & d3["entity_relation"] & d4["procedure"] & d5["policy"] classDef raw fill:#d0ebff,stroke:#1971c2 classDef derived fill:#e5dbff,stroke:#6741d9 class EL raw class DM derived ``` Every derived record should retain `source_event_ids`. When source data is deleted, the system can identify which summaries, embeddings, and graph edges must be rebuilt or revoked. ### 8.2 Put the write gate before embedding At minimum, the gate decides: - Is this relevant to a future task? - Is it an explicit fact or a model inference? - Does it contain sensitive data? - Did the user authorize persistence? - Does it already exist? - Should it become an episode, fact, relation, procedure, or policy? **Every stored memory is a tax on every future retrieval.** ### 8.3 Use different keys and retrieval for different memory types | Type | Recommended key | Primary retrieval | |---|---|---| | Profile fact | `tenant/user/fact_type` | Exact key + version | | Episode | `tenant/user/time/event_id` | Temporal filter + hybrid search | | Relation | Entity IDs + relation type + validity | Graph query + time | | Procedure | Task signature + version | Routing + semantic recall | | Policy | Scope + priority + version | Deterministic mounting | One embedding collection is not a substitute for schema design. ### 8.4 Make time and provenance first-class fields A minimal record should include: ```yaml id: tenant_id: subject_id: memory_type: content: source_event_ids: confidence: valid_from: valid_to: created_at: expired_at: supersedes: access_scope: ``` Use a Graphiti-style bitemporal model when facts change frequently. For simpler facts, at least retain `valid_from / valid_to / supersedes`. ### 8.5 Read through candidate generation, filtering, and assembly A robust read path looks like: ```mermaid flowchart TD A["query"] --> B["scope / ACL filter"] --> C["exact + BM25 + vector + graph candidates"] --> D["recency / importance / validity rerank"] --> E["contradiction check"] --> F["token-budget packing"] --> G["provenance-preserving context"] classDef out fill:#d3f9d8,stroke:#2f9e44 class G out ``` Similarity is only one signal. ### 8.6 Run consolidation and forgetting in the background Background jobs can: - Cluster similar episodes. - Extract stable facts. - Update profiles. - Generate procedures. - Mark superseded facts. - Decay low-value material. - Apply TTL and user deletion. - Rebuild affected indexes. The online path should perform only essential fast writes rather than paying the full LLM cost on every turn. ### 8.7 Evaluate task outcomes, not only memory QA Track at least: - **Write precision:** how many records were genuinely worth keeping? - **Stale recall rate:** how many retrieved records were no longer valid? - **Provenance coverage:** how many memory-influenced answers identify source events? - **Cross-tenant leakage:** this must be zero. - **Deletion completeness:** does derived state remain after deletion? - **Task success delta:** did memory improve actual completion? - **Token, latency, and cost:** what is the marginal cost of one useful memory? --- ## 9. Where Agent Memory Is Heading No single “most brain-like” project is likely to dominate soon. A more plausible convergence has three layers: 1. **Runtime:** current task, agent identity, tool permissions, and context. 2. **Memory service:** events, facts, relations, skills, time, provenance, and deletion. 3. **Model:** longer context, stronger test-time learning, and possibly architectural memory modules. Today most agents remember in exactly one way: drop a sentence into a vector database and fetch a few that mean something similar (`vector_db.search(text)`). That is an assistant who can only flip through an old notebook looking for similar sentences. The interface that eventually settles will look more like a dependable secretary, one who can do at least five things: | What the secretary does | Roughly the interface | In plain words | |---|---|---| | **Note it down** | `remember(event, policy)` | Something happened; decide by the rules whether it is worth keeping and for how long, instead of copying everything into the notebook | | **Recall it** | `recall(query, scope, time, budget)` | Look things up only for the right person, project, and period, and only as much as fits — don't hand the model someone else's, stale, or far too many notes | | **Correct it** | `revise(memory, evidence)` | When new evidence arrives, update the old memory: "stopped drinking coffee" replaces "likes coffee" instead of sitting next to it | | **Forget it** | `forget(subject, reason)` | When a user says "forget my address", delete it along with the summaries and indexes derived from it, and record why | | **Cite the source** | `explain(memory_id)` | Answer "why do you remember that?": which conversation it came from, when, and whether it still holds | **Today's memory can only "find something similar". Tomorrow's has to note down, recall, correct, forget, and cite its source.** Human memory science spent more than a century moving from “where is memory stored?” to “how do multiple systems reconstruct the past during retrieval?” Agent memory engineering is undergoing the same conceptual upgrade: > [!TIP] > **The useful question is no longer whether an agent has memory. It is what change the agent preserves, why it preserves it, when it recalls it, how it revises it, and who has the authority to make it forget.** --- ## Primary Sources and Pinned Code Entrypoints ### Human memory science - [Ebbinghaus — *Memory: A Contribution to Experimental Psychology*](https://psychclassics.yorku.ca/Ebbinghaus/) - [Müller & Pilzecker — *Experimentelle Beiträge zur Lehre vom Gedächtniss*](https://books.google.com/books?id=5RdCAQAAMAAJ) - [Scoville & Milner — the H.M. case](https://pmc.ncbi.nlm.nih.gov/articles/PMC497229/) - [Baddeley & Hitch — Working Memory](https://doi.org/10.1016/S0079-7421%2808%2960452-1) - [2014 Nobel Prize — place cells and grid cells](https://www.nobelprize.org/prizes/medicine/2014/advanced-information/) - [Nader, Schafe & LeDoux — Reconsolidation](https://pubmed.ncbi.nlm.nih.gov/10963596/) - [Liu et al. — Optogenetic activation of a hippocampal engram](https://pmc.ncbi.nlm.nih.gov/articles/PMC3331914/) ### Agent-memory papers - [Neural Turing Machines](https://arxiv.org/abs/1410.5401) - [Retrieval-Augmented Generation](https://arxiv.org/abs/2005.11401) - [Generative Agents](https://arxiv.org/abs/2304.03442) - [Voyager](https://arxiv.org/abs/2305.16291) - [CoALA](https://arxiv.org/abs/2309.02427) - [MemGPT](https://arxiv.org/abs/2310.08560) - [Lost in the Middle](https://arxiv.org/abs/2307.03172) ### Pinned code-audit entrypoints - [Mem0 `memory/main.py` @ `9c2d622`](https://github.com/mem0ai/mem0/blob/9c2d6222ce86bf6a73ae7ca97464a8e1a55ab3ca/mem0/memory/main.py) - [Letta `schemas/memory.py` @ `b76da90`](https://github.com/letta-ai/letta/blob/b76da9092518cbaa2d09042e52fdcbde69243e18/letta/schemas/memory.py) - [Graphiti `graphiti_core/edges.py` @ `2645dee`](https://github.com/getzep/graphiti/blob/2645dee20fd71797a61e1c6177a93cccd5584574/graphiti_core/edges.py) - [LangMem `knowledge/extraction.py` @ `56d8593`](https://github.com/langchain-ai/langmem/blob/56d85939d80bb731bd5e237567148d817d7bfd16/src/langmem/knowledge/extraction.py) - [Cognee `cognify.py` @ `88aa09b`](https://github.com/topoteretes/cognee/blob/88aa09b4e3289e3dbf12c0c090080920816e2fb7/cognee/api/v1/cognify/cognify.py) - [MemoryOS `memoryos.py` @ `587ed77`](https://github.com/BAI-LAB/MemoryOS/blob/587ed7755c7aed179965792830ff1b5ad9a6fa92/memoryos-pypi/memoryos.py) --- *Audit date: 2026-07-30. Open-source repositories change quickly, so architectural claims link to pinned commits. Project marketing is used only to describe self-positioning, not as evidence about implementation.* --- title: "How to Choose an LLM Inference Engine — A 2026 Map from Local Single-GPU to PD Disaggregation" description: "Aliyun's four-engine table — Ollama / vLLM / SGLang / HF Pipeline — is no longer enough in 2026. This piece re-maps inference engines into three tiers (Local → High-Performance Serving → Distributed/Disaggregated), covering 8 mainstream engines plus three new trends — PD disaggregation, speculative decoding, and FP4 quantization — with a decision matrix and a decision tree." url: "https://zhuoqidev.com/en/posts/llm-inference-engine-selection/" language: "en" author: "Liu ZhuoQi" date: "2026-07-19" updated: "2026-07-30" tags: ["LLM","Inference Engine","vLLM","SGLang","Model Serving","Inference Optimization","Selection Guide","Research"] translation: "https://zhuoqidev.com/posts/llm-inference-engine-selection/index.md" --- # How to Choose an LLM Inference Engine — A 2026 Map from Local Single-GPU to PD Disaggregation > **TL;DR** This article updates that map to mid-2026. It won't tell you which product to buy — it gives you a tiered framework, a decision matrix, and a decision tree so you can narrow the field to 1–2 candidates yourself. Aliyun's CAP has a piece on picking an inference engine that narrows the field to four: **Ollama, vLLM, SGLang, and Hugging Face Pipeline**. In 2024, that framing was fine. By 2026, it's missing half the map. NVIDIA's TensorRT-LLM has completed its "PyTorch-ification," SGLang became famous as the first open-source project to reproduce DeepSeek's large-scale deployment, Hugging Face slapped a "maintenance mode" banner on TGI and told you to switch to vLLM — and the real throughline of the entire 2025 inference landscape can be summed up in one word: **disaggregate**. The first diagram prevents the most common comparison error: Ollama, KTransformers, and vLLM do not solve the same layer of the problem. Classify the deployment as local, hybrid offload, or high-performance serving before comparing throughput, model formats, and hardware support within that layer. ![A three-layer map of LLM inference engines in 2026](https://zhuoqidev.com/_astro/three-layer-inference-map-bilingual-v3-4k.D2QRcI0K_QGqDD.webp "Three-layer inference map: L1 optimizes for running anywhere and simple setup; L1.5 trades host RAM for VRAM; L2 optimizes for concurrency, throughput, and multi-GPU serving. This is a classification framework, not an overall ranking.") --- ## Why "Choosing an Inference Engine" Is a Real Problem in 2026 Three years ago this wasn't worth agonizing over. Back then, getting a 7B model running on a GPU with a reasonably smooth token stream was already a pass. That changed, for three reasons: 1. **Model capabilities are converging; deployment forms are diverging.** On the open side — from Llama and Qwen to DeepSeek-V3/R1 and Kimi K2 — the capability gap is shrinking. What actually separates teams now is "can you run it efficiently, cheaply, and reliably." The same DeepSeek-R1: one person barely runs it on a single 4090 for a prototype, another hits near-official throughput on 96 H100s at one-fifth the official API cost. The difference is the engine and the deployment architecture. 2. **Large-scale MoE has pushed inference from "single-GPU engineering" to "distributed-systems engineering."** DeepSeek-V3 is 671B total / 37B active; Kimi K2 is 1T total. These models don't fit on one machine — you *must* do expert parallelism (EP) across nodes, and the engine you pick decides whether you have to wrestle with all-to-all communication and expert load balancing. 3. **For the same model, throughput/cost between engines can differ by several times.** Not "a little faster" — "same GPU, several times more users served." That goes straight onto your cloud bill. So selection is no longer "grab whatever runs." It's **first figure out which tier of the map you're on.** --- ## The One-Table Overview: Eight Engines at a Glance Conclusion first. Here's where the mainstream 2026 engines sit, side by side: | Engine | Backed by | Core mechanism in one line | Latest version (as of 2026-07) | Typical positioning | |---|---|---|---|---| | **vLLM** | Community / Red Hat | PagedAttention + fully async V1 engine | v0.25.x (2026-07) | The general-purpose open-source standard | | **SGLang** | LMSYS / Community | RadixAttention prefix-tree KV reuse | v0.5.x | Large-scale MoE / DeepSeek deployment | | **TensorRT-LLM** | NVIDIA | Compiled kernels + In-flight Batching | v1.2.1 (2026-04) | Peak performance on NVIDIA cards | | **LMDeploy** | Shanghai AI Lab | TurboMind high-performance C++ engine | v0.13.x | Chinese models / mixed-precision quant | | **Ollama** | Ollama | llama.cpp/GGML + in-house multimodal engine | v0.30.x | Local / desktop one-command runs | | **llama.cpp** | ggml-org | GGUF + runs on CPU/GPU alike | Rolling | Edge / CPU / quantization | | **HF TGI** | Hugging Face | Early Rust serving gateway | v3.3.7 (**maintenance mode**) | ⚠️ Officially advised to migrate away | | **KTransformers** | Tsinghua kvcache-ai | CPU/GPU heterogeneous, experts on CPU | Rolling | Running huge MoE on a single GPU | See the ⚠️ on the HF row? That's one of the most important shifts of 2025 — more on it below. --- ## Think in Tiers: Inference Engines Actually Split Into Three Layers Laying all eight engines flat in one table is a trap — they aren't solving the same problem. **Choosing between Ollama and SGLang is like comparing a sedan and a container truck on fuel economy.** The right move is to tier them first: ``` L1 Local / Single-Machine — one user; what matters is "runs at all, installs easily" Ollama · llama.cpp · LM Studio · HF Pipeline (prototype) L2 High-Performance Serving — one service, many users; throughput / latency / VRAM efficiency win vLLM · SGLang · TensorRT-LLM · LMDeploy · (TGI has exited) L3 Distributed / Disaggregated — one data center, one big model; architecture matters more than the engine PD disaggregation · NVIDIA Dynamo · Mooncake · large-scale Expert Parallelism (EP) ``` **Locate your tier first, then pick within it.** Most people's agonizing comes from taking an L1 need to L2 to shop — or the reverse. --- ## L1 Local / Single-Machine: Runs at All, Installs Easily The core need here isn't throughput — it's **"one person, one machine, ideally one command to get the model running."** Privacy-sensitive, offline, edge, and rapid prototyping all live here. ### Ollama / llama.cpp: The Face and the Kernel These two belong together — they're two sides of the same thing: - **llama.cpp** is the kernel — C/C++, the **GGUF** quantization format, runs on CPU / GPU / Apple Silicon alike. The bedrock of edge quantized inference. - **Ollama** is the face — a product layer on top of llama.cpp for "one-command launch + model management." `ollama run qwen3` and you're done. There's a **2025 change many older articles haven't caught up on**: > **Ollama is moving from "llama.cpp wrapper" to its own in-house engine.** In May 2025 it shipped a new multimodal engine, making vision models (gemma3, qwen2.5vl, mistral-small3.1, etc.) first-class citizens, talking directly to the GGML tensor library and gradually decoupling from llama.cpp ([Ollama blog](https://ollama.com/blog/multimodal-models)). By 0.30, it added an MLX engine on Apple Silicon alongside the llama.cpp one, enabled Vulkan by default (extending to AMD/Intel GPUs), and gained ~+20% on NVIDIA ([Ollama blog](https://ollama.com/blog)). So if you still think "Ollama is just a llama.cpp launcher," it's time to update. **When to pick them:** local dev, desktop apps, private data that can't leave the intranet, edge deployment, quickly checking whether a model "works." GGUF is the universal exchange format for this tier. **When not to:** high-concurrency online services. These target single-machine / single-user experience, not data-center throughput — a positioning *orthogonal* to L2 engines, not a "who's stronger" question. ### HF Pipeline vs. TGI: One's a Prototyping Toy, One Has Exited Aliyun's original lists "Hugging Face Pipeline" as a candidate. Two things need untangling here, because **they get conflated constantly and both have traps**: - **`transformers.pipeline`** is the high-level "call a model in a few lines" API — great for teaching, prototypes, small offline batches. But it **was never a production serving solution** — no continuous batching, no PagedAttention; it collapses under concurrency. As a candidate, it's a "prototype-stage" candidate only. - **TGI (Text Generation Inference)** was HF's actual serving product. But — > [!CAUTION] > **One of the most important shifts of 2025: Hugging Face put a "maintenance mode" banner on TGI with its own hands.** The GitHub README states plainly *"text-generation-inference is now in maintenance mode,"* accepts only minor bug fixes, and **recommends users move to vLLM, SGLang, or local llama.cpp / MLX** ([TGI GitHub](https://github.com/huggingface/text-generation-inference)). HF strategically ceded serving to vLLM/SGLang and went back to focusing on the transformers model-architecture standard. **Takeaway:** if someone still recommends TGI for production in 2026, their information stopped in 2024. Prototype with `pipeline`; for production, look at L2. --- ## L2 High-Performance Serving: One Service, Many Users This is the tier most "going live" teams actually shop in. The core needs become **throughput, latency (TTFT / TPOT), and VRAM efficiency** — same GPU, several times more requests. Two foundational techniques worth remembering first: - **Continuous Batching**: don't wait for a batch to finish — at every generation step, dynamically admit new requests and retire completed ones, squeezing the GPU dry. Standard in every modern serving engine. - **PagedAttention / RadixAttention**: two KV-cache management philosophies, the signature moves of vLLM and SGLang respectively. Detailed below. ### vLLM: The Open-Source Standard, V1 Rewrite in 2025 **Core mechanism**: **PagedAttention** slices the KV cache into fixed-size "pages" and, like an OS managing virtual memory, maps logical positions to non-contiguous physical blocks — eliminating fragmentation and boosting concurrency. **The big 2025 event was the V1 architecture rewrite**: shipped as an alpha with v0.7.0 (2025-01), claiming **up to 1.7× throughput** over the old architecture, with all optimizations on by default, an isolated execution loop, and zero-overhead prefix caching ([vLLM blog](https://vllm.ai/blog/2025-01-27-v1-alpha-release)); **V1 became the default engine** from v0.8.0 on ([Red Hat measurements](https://developers.redhat.com/articles/2025/04/28/performance-boosts-vllm-081-switching-v1-engine)). The V1 scheduler can **mix prefill and decode within the same step**, and bundles chunked prefill, prefix caching, speculative decoding, and PD disaggregation. **Speculative decoding, quantization, PD disaggregation — all present**: built-in n-gram / EAGLE / Medusa / MTP speculative decoding; quantization spanning FP8 / INT8 / AWQ / GPTQ / GGUF up to MXFP4 / NVFP4 on Blackwell; PD disaggregation via KV Connector to external KV services like Mooncake / LMCache. **One-line positioning**: **the "default option" with the widest model coverage, richest ecosystem, and single-GPU-to-multi-node reach.** When you can't decide, starting with vLLM is rarely wrong. ### SGLang: The Star of Large-Scale MoE / DeepSeek Deployment **Core mechanism**: **RadixAttention** uses a radix tree (prefix tree) + LRU to automatically reuse shared-prefix KV cache across requests, making prefix caching a first-class citizen — a natural fit for Agents, multi-turn dialogue, and few-shot loads where **prefixes are highly repetitive**. **The battle that made SGLang famous**: in 2025-05 it became the **first open-source project to reproduce DeepSeek's official-grade large-scale deployment** — 12 nodes × 8×H100 = **96 H100s**, at 2000-token input hitting 52.3K input tok/s and 22.3K output tok/s per node, **5× faster than naive tensor parallelism**, at roughly **$0.20 / million output tokens (about 1/5 of DeepSeek's official Chat API price)**, within 5–6% of the official profile ([LMSYS blog](https://www.lmsys.org/blog/2025-05-05-large-scale-ep/)). Behind it is a whole combo: DeepEP, DeepGEMM, EPLB, DP Attention, Two-Batch Overlap. On speculative decoding, SGLang running MTP on DeepSeek-V3 gets up to +60% output throughput, ~1.8× tokens/s ([LMSYS blog](https://www.lmsys.org/blog/2025-07-17-mtp/)). **One-line positioning**: **the top pick for huge MoE data-center deployment and long-shared-prefix (Agent / multi-turn) workloads.** By 2026, SGLang is the most aggressive open-source engine for "whole-cluster PD + EP engineering" on DeepSeek-class models — it long ago stopped being "one of four" and became the number-one player in that niche. ### TensorRT-LLM: NVIDIA's Own Engine, "PyTorch-ified" **Core mechanism**: compile the model into a TensorRT engine for kernel-level optimization, paired with In-flight Batching (NVIDIA's name for continuous batching). **The historical baggage and the 1.0 pivot**: TensorRT-LLM's old reputation was "compiled, blazing fast, but painful to use." **v1.0.0 (2025-09-24 — note 2025, not 2024)** is the turning point — the PyTorch backend became default and stable, the high-level LLM API stabilized, usability clearly moved toward vLLM/SGLang, and it added NVFP4, speculative-decoding enhancements, and Wide-EP (large-scale expert parallelism) ([v1.0.0 Release](https://github.com/NVIDIA/TensorRT-LLM/releases/tag/v1.0.0)). > ⚠️ **Read marketing numbers critically**: figures floating around like "10,000+ tok/s on H100 FP8, 4× faster than native PyTorch" mostly come from third-party blogs and aren't stably reproduced in NVIDIA's first-party materials. Cite with care. **One-line positioning**: **pure NVIDIA hardware (especially Hopper / Blackwell), chasing peak throughput/latency, going enterprise (Triton / NIM).** The price is NVIDIA-ecosystem lock-in and lots of closed-source kernels. ### LMDeploy / TurboMind: Chinese Force, Strong on Mixed-Precision and KV Quant **Core mechanism**: Shanghai AI Lab's in-house **TurboMind** high-performance C++/CUDA engine, built around blocked KV cache, online KV-cache quantization (INT4/INT8), AWQ, and automatic prefix caching. **Highlight numbers**: TurboMind's mixed-precision inference cuts latency by up to 61% and raises throughput by up to 156% versus existing mixed-precision frameworks, reaching 1.5× vLLM running gpt-oss on H800 ([arXiv:2508.15601](https://arxiv.org/abs/2508.15601)). For DeepSeek it integrates FlashMLA, DeepGEMM, DeepEP, and lands PD disaggregation via DLSlime + Mooncake. **One-line positioning**: **production deployment of Chinese models (InternLM / Qwen / DeepSeek), teams with strong KV-quant and low-latency needs.** Often beats vLLM on mid-low-batch latency and quantized inference, at the cost of a smaller ecosystem and model coverage. --- ## L1.5 Special Case: KTransformers — Trading Memory for "It Runs At All" This engine deserves its own slot because it solves one very specific but very painful problem: **not enough VRAM, but I want to run the full 671B MoE locally anyway.** **Core mechanism**: exploit MoE's "sparse activation" for CPU/GPU heterogeneity — offload the sparsely-activated expert weights to the CPU (big RAM + AMX instructions) and keep dense compute like attention on the GPU (high bandwidth). **Signature result**: running the **full DeepSeek-R1/V3 671B** on **a single 24GB GPU (RTX 4090D) + large RAM**, with prefill up to 286 tok/s and generation ~14 tok/s ([KTransformers tutorial](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md)). In 2025-10 its CPU kernels were merged into SGLang, taking "heterogeneous offload" from toy to serviceable. **One-line positioning**: **VRAM-constrained research / low-cost private deployment that still wants to run frontier big MoE.** It doesn't chase high-concurrency throughput — it chases "can it run at all." The poster child for "run frontier models on consumer hardware." --- ## Model Format × Engine: First, What Format Are Your Weights? Before picking an engine, check what format your model is in — this alone cuts half the candidates: | Format | Notes | Main engines | |---|---|---| | **GGUF** | Universal edge quant container (Q4_K_M, etc.), CPU-friendly | llama.cpp / Ollama | | **Safetensors** | Fast to load, resistant to malicious code, HF-ecosystem mainstream | vLLM / SGLang / TRT-LLM / LMDeploy | | **PyTorch (.pt/.pth)** | Raw state_dict weights | vLLM / HF Pipeline | ### Quantization: 2026 Has Entered the 4-bit (FP4) Era Aliyun's original stops at FP8 / GGUF for quantization — this part needs an upgrade. The 2025 storyline for quant is **8-bit → 4-bit**: | Quant format | Notes | Main engines | |---|---|---| | **FP8 (E4M3)** | Native since Hopper/Ada; weights + activations + KV cache | vLLM / TRT-LLM / SGLang / LMDeploy | | **AWQ** | Activation-aware 4-bit weight quant, good accuracy | vLLM / LMDeploy / SGLang | | **GPTQ** | Classic 4-bit post-training quant | vLLM / TRT-LLM | | **GGUF (Q4_K_M…)** | Universal edge quant container | llama.cpp / Ollama | | **MXFP4** | One power-of-two scale per 32 values (micro-scaling) | gpt-oss native / vLLM / LMDeploy / TRT-LLM | | **NVFP4** | One FP8 scale per 16 values, finer | Blackwell native / TRT-LLM / vLLM | The landmark event is **OpenAI's gpt-oss (2025) shipping directly in MXFP4** — the MoE expert weights (90%+ of parameters) are quantization-aware-trained, letting the 120B fit on a single 80GB card ([NVIDIA blog](https://developer.nvidia.com/blog/fine-tuning-gpt-oss-for-accuracy-and-performance-with-quantization-aware-training/)). NVIDIA claims NVFP4's KV cache is ~50% smaller than FP8 with <1% accuracy loss and 1.6× BF16 throughput ([NVIDIA Research](https://research.nvidia.com/labs/eai/blogs/pushing-intelligence-to-4-bit/)). > **A rule of thumb**: **format and quantization define your candidate pool — clear this gate before talking engines.** GGUF in hand? Don't dream of SGLang. Want NVFP4? You need a Blackwell card plus TRT-LLM/vLLM. --- ## The Three New Trends Aliyun's Original Skipped This is where this article adds the most over the original. The real 2025 progress in inference engines lives entirely in these three things. The next diagram explains the trend most often compressed into a single phrase: **Prefill–Decode disaggregation**. Prefill and Decode serve one request but are constrained by different objectives—throughput and tail latency—so separate resource pools can scale and schedule them independently. ![A distributed inference architecture that separates Prefill and Decode](https://zhuoqidev.com/_astro/prefill-decode-disaggregation-bilingual-v2-4k.SyrOKq7J_Z6TTYC.webp "Prefill–Decode disaggregation: a compute-heavy Prefill pool processes long prompts and produces KV Cache; a latency-sensitive Decode pool takes over token-by-token generation. KV transfer connects the stages while their autoscaling goals remain independent.") ### Trend 1: PD Disaggregation — The Architectural Throughline of 2025 The "disaggregate" from earlier mostly means this. **Why split**: one inference has two phases with opposite characteristics — - **Prefill**: computes the KV for the whole prompt — **compute-bound**, eats FLOPs. - **Decode**: generates one token per step — **memory-bound**, eats memory bandwidth. Cramming both onto the same card makes them interfere, and it's hard to satisfy time-to-first-token (TTFT) and time-per-output-token (TPOT) at once. **Split them onto separate GPU pools and each can scale independently and pick its own optimal parallelism** — that's Prefill-Decode disaggregation (disaggregated serving). Two representative works: - **Mooncake** (the serving platform behind Moonshot's Kimi): a **KVCache-centric** disaggregated architecture that not only splits prefill/decode but organizes idle CPU / DRAM / SSD across the cluster into a **global KV cache pool**. The paper won **FAST 2025 Best Paper**, showing up to +525% throughput in simulated long-context scenarios ([arXiv:2407.00079](https://arxiv.org/abs/2407.00079)). It's now widely integrated by vLLM, SGLang, TensorRT-LLM, and Dynamo. - **NVIDIA Dynamo** (see Trend 3) productizes PD disaggregation directly. > **There's a mental shift behind this**: **the KV cache is going from "a thing in one machine's VRAM" to "a cluster-level storage layer."** Mooncake and LMCache sink KV down to CPU/DRAM/SSD, enabling cross-instance, near-GPU prefix caching. That step takes inference squarely into distributed-systems territory. ### Trend 2: Speculative Decoding — From Paper to Standard Feature **One-line principle**: a lightweight "draft model" quickly guesses the next several tokens, then the big model **verifies the whole string in a single forward pass** — every correct guess skips a step of one-at-a-time generation. In 2025 it went from "a trick in papers" to "flip a switch in vLLM/SGLang": - **EAGLE-3** ([arXiv:2503.01840](https://arxiv.org/abs/2503.01840)): drops feature prediction in favor of direct token prediction + multi-layer feature fusion, **up to 6.5× speedup**, and found a scaling law — "more training data, higher speedup." - **MTP (Multi-Token Prediction)**: DeepSeek-V3 ships with an MTP head that naturally serves as a draft. SGLang measures ~1.8× tokens/s at 85–90% acceptance; vLLM auto-detects the MTP head. - **Medusa**: the 2024 landmark (multiple decoding heads + tree attention), largely surpassed by EAGLE-3 in 2025 but still a built-in option. **What it means for selection**: if your engine (vLLM / SGLang / TRT-LLM) supports speculative decoding and you haven't turned it on — that's a nearly free 1.4–2× speedup worth trying first. ### Trend 3: NVIDIA Dynamo — Productizing Disaggregated Inference **The key positioning point**: Dynamo **is not yet another inference engine — it's an orchestration layer above the engines.** Announced at GTC 2025, it's data-center-grade, open-source, purpose-built for distributed inference orchestration. It packages the first two trends into a product, with four components: 1. **Planner**: dynamically switches between "disaggregated vs. aggregated" and scales GPUs up/down by SLO and load. 2. **Smart Router**: tracks the whole cluster's KV cache via a radix tree and routes a request to the card that will hit the cache. 3. **KV Cache Manager**: sinks cold KV to CPU / SSD / object storage. 4. **NIXL**: a high-throughput transfer library across heterogeneous memory / storage. **Performance**: DeepSeek-R1 671B gets 30× throughput on GB200 NVL72; Llama 70B doubles throughput on Hopper. Crucially, **it accepts vLLM / SGLang / TensorRT-LLM as backends** ([NVIDIA blog](https://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/)). > **The signal is clear**: NVIDIA uses Dynamo to standardize "PD disaggregation + smart routing + KV tiering," and **doesn't bind to its own engine** — effectively conceding that vLLM/SGLang are a done deal and retreating to add value at the orchestration and hardware layers. The open engines' standing is thereby stamped. --- ## The Decision Matrix Putting the L2 engines that you actually pick "one of two / one of three" from side by side, scored by dimension: | Dimension | vLLM | SGLang | TensorRT-LLM | LMDeploy | |---|---|---|---|---| | **Model coverage** | Widest | Wide | Medium (NVIDIA-first) | Medium (China-friendly) | | **Ease of onboarding** | Low | Medium | Medium (better after 1.0) | Medium | | **General high-concurrency throughput** | Strong | Strong | Strongest (NV cards) | Strong | | **Huge MoE / EP deployment** | Strong | Strongest | Strong | Strong | | **Prefix reuse (Agent/multi-turn)** | Good | Best (RadixAttention) | Good | Good | | **KV quant / low latency** | Good | Good | Strong | Strongest (TurboMind) | | **Hardware lock-in** | None (incl. TPU) | None | Hard NVIDIA lock | Mostly NVIDIA | | **Ecosystem / community** | Most active | Active | NVIDIA official | China community | No column is all "best" — **and that's exactly what "neutral" means: there's no one-size-fits-all engine, only the one that matches your workload.** --- ## The Practical Decision Tree Everything above, compressed into one tree: ```mermaid flowchart LR %% layout: tree Q(["Which tier are you on?"]) Q --> L1["Local / single-machine
privacy / prototype"] Q --> L2["High-performance online service
one service, many users"] Q --> L3["Data-center scale
one big model eats the whole cluster"] L1 --> C1["Just want one command, daily use"] --> E1["Ollama"] L1 --> C2["Squeezing CPU / edge
extreme quant"] --> E2["llama.cpp"] L1 --> C3["A few lines to validate a model"] --> E3["HF transformers.pipeline"] L1 --> C4["VRAM too small but must
run full 671B MoE"] --> E4["KTransformers"] L2 --> C5["Unsure / want the widest
model coverage"] --> E5["vLLM (default to it)"] L2 --> C6["Large MoE / DeepSeek
heavy multi-turn prefix"] --> E6["SGLang"] L2 --> C7["Pure NVIDIA + peak perf
+ enterprise"] --> E7["TensorRT-LLM"] L2 --> C8["Chinese models + strong
KV quant / low latency"] --> E8["LMDeploy"] L3 --> C9["Don't just stare at the engine —
add an orchestration layer"] --> E9["NVIDIA Dynamo
PD disaggregation
+ Mooncake KV pool + EP
backend is still
vLLM / SGLang / TRT-LLM"] classDef q fill:#fff3bf,stroke:#f08c00 classDef tier fill:#d0ebff,stroke:#1971c2 classDef engine fill:#d3f9d8,stroke:#2f9e44 class Q q class L1,L2,L3 tier class E1,E2,E3,E4,E5,E6,E7,E8,E9 engine ``` ⚠️ Still considering TGI for production? — Don't. It's officially in maintenance mode; reroute to vLLM/SGLang. --- ## Scenario Summary: One-Line Answers | If you are… | You'll most likely pick | |---|---| | An individual dev running models locally, prototyping | **Ollama** (+ GGUF) | | Squeezing edge / CPU / Apple Silicon | **llama.cpp** | | Teaching, small offline batches, validating in a few lines | **HF `pipeline`** (not for production) | | VRAM-constrained but want the full big MoE | **KTransformers** | | Going live, unsure, want a solid default | **vLLM** | | Running DeepSeek / MoE / Agent multi-turn at scale | **SGLang** | | Pure NVIDIA, chasing peak perf, want the enterprise stack | **TensorRT-LLM** | | Chinese models, heavy KV quant / low latency | **LMDeploy** | | One big model eating the whole data center | **Dynamo** (orchestration) + one of the above (backend) | --- ## Closing Aliyun's article isn't wrong — it just **stopped at the "run a model on one machine" tier.** The industry's center of gravity in 2025 has moved from "which engine runs the model" to "what architecture spreads one big model efficiently across an entire cluster." If I keep only one sentence, it's this: > **Choosing an inference engine is, at heart, choosing "which tier you're on."** Locate yourself in one of the three — local / serving / distributed — and the candidates auto-narrow to one or two; the rest of the difference, hand it to your own workload to benchmark. **There is no universally optimal engine — only the number that ran on your own traffic.** Three years ago, getting the model running won. Today, *how* you run it is the real dividing line. --- *The latest-progress data here (vLLM V1, SGLang reproducing DeepSeek, TensorRT-LLM 1.0, Mooncake, Dynamo, EAGLE-3, NVFP4, etc.) comes from multi-source cross-validation of official blogs, GitHub releases, and arXiv papers; key figures are linked inline. Vendor self-reported numbers (e.g., TGI's long-context self-benchmark, third-party TRT-LLM throughput) are flagged "read critically" in the text. As of 2026-07.* --- title: "OpenClaw in Practice: One File Path Eliminated 84% of Tool Calls — A Cron Job Debugging Story" description: "OpenClaw's daily-ai-news cron job kept timing out. The root cause wasn't a weak model, bloated prompt, or upstream API failure — it was a missing absolute path in the SKILL.md, causing the Agent to spend 15 exec calls searching for a tool's location every run. Message count ballooned from 54 to 165, tool results from 204KB to 1.1MB. This is the full debugging story." url: "https://zhuoqidev.com/en/posts/openclaw-cron-skill-optimization/" language: "en" author: "Liu ZhuoQi" date: "2026-06-20" updated: "2026-07-30" series: "OpenClaw Production Notes" tags: ["OpenClaw","AI Agent","Cron Job","SKILL.md","Prompt Engineering","Performance"] translation: "https://zhuoqidev.com/posts/openclaw-cron-skill-optimization/index.md" --- # OpenClaw in Practice: One File Path Eliminated 84% of Tool Calls — A Cron Job Debugging Story > [!NOTE] > This is the third post in the [OpenClaw Production Notes](https://zhuoqidev.com/series/openclaw-production-notes/) series. [The first](https://zhuoqidev.com/posts/openclaw-pitfalls/) covered compaction silently swallowing replies, [the second](https://zhuoqidev.com/posts/openclaw-memory-text-to-vector/) covered the memory system running fine without vector search and fixing it with NVIDIA's free API. This one is about a more mundane but higher-impact problem: **what happens when your AI Agent doesn't know where its tools are?** The before/after diagram establishes the complete causal chain. The model was not “stupid”; a vague instruction forced it to explore directories repeatedly. One absolute path collapsed the action space, reducing messages, searches, and execution calls at the same time. ![The OpenClaw cron job Outer Loop before and after an explicit path](https://zhuoqidev.com/_astro/outer-loop-path-before-after-bilingual-v1-4k.CE2cTKrE_1VLVCV.webp "One path changed the whole loop: messages fell 165→54, exec calls 44→7, and location searches 15→1. The optimization removes repeated tool discovery rather than merely making one tool call faster.") ## Symptom: Cron Job Keeps Timing Out OpenClaw runs a cron job at 21:05 daily — `daily-ai-news` — which scrapes AI news, generates a summary, creates a banner image, and posts to a Lark (Feishu) group chat. One day I noticed this job failing repeatedly. The logs showed: ``` Status: error Duration: 1200s+ (timeout) Tokens: in=2,800,000+ out=15,000+ ``` Twenty-minute timeout. 2.8 million input tokens consumed. For a daily newsletter task. ## Debugging: Reconstructing Agent Behavior from Logs OpenClaw's cron executions produce full conversation logs. I pulled the most recent failure and counted the key metrics: | Metric | Failed Run | |--------|-----------| | Total messages | 165 | | exec tool calls | 44 | | tool_result total size | 1,122 KB | | Runtime | 1200s+ (timeout) | 165 messages? 44 exec calls? A "scrape news → write summary → generate image → post to chat" pipeline shouldn't exceed 60 messages and 10 tool calls under normal conditions. Scrolling through the log line by line, I found the culprit. ## Root Cause: The Agent Didn't Know Where Its Tool Was The `daily-ai-news` SKILL.md (the skill's instruction file) contained this line: ```markdown Use create-img / gpt-image-2 to generate the banner image. ``` That's it. No path, no invocation syntax, no parameters. The Agent understood it needed to call `create-img`, but had no idea where it lived. So it started **searching**: ```bash # Attempt 1 find /home/openclaw -name "create-img" -type d # Didn't find what it wanted, keep going grep -r "create-img" /home/openclaw/.openclaw/ # Found some references, but not sure if they're executable ls -la /home/openclaw/.openclaw/plugin-skills/ # Didn't see it, try another directory ls -la /home/openclaw/.agents/skills/ # Found it! But not sure how to invoke it cat /home/openclaw/.agents/skills/create-img/SKILL.md # Read it, but still need to confirm the script path find /home/openclaw/.agents/skills/create-img -name "*.py" # Found the script, but which Python environment? which python3 python3 --version # Found Python, but where's the virtual environment? ls /home/openclaw/workspace/.venv*/ # Found the venv, now assembling the command... ``` **15 exec calls**, just to locate a tool and figure out how to call it. Worse, these search operations produced massive `tool_result` payloads — every `find`, `grep`, and `ls` output got stuffed into the conversation context. 44 exec calls accumulated 1.1MB of tool results, blowing up the context window, triggering compaction, and further slowing execution. ### The Vicious Cycle This is a textbook **context bloat feedback loop**: ```mermaid flowchart TD A["SKILL.md missing path"] --> B["Agent searches for tool location
15 exec calls"] B --> C["Search results fill context
+900KB"] C --> D["Triggers compaction
takes time + may lose info"] D --> E["Compaction discards the path Agent just found"] E -->|"searches again (another 15 exec calls)"| B E -->|"context bloats again"| F["Timeout"] classDef cause fill:#fff3bf,stroke:#f08c00 classDef bad fill:#ffe3e3,stroke:#e03131 class A cause class F bad ``` That's right — **compaction discarded exactly the tool path the Agent had painstakingly discovered**, so the next round started the search from scratch. This is where 165 messages came from: the Agent went through 2-3 complete cycles of "search → find → get compacted → search again." ## The Fix: One Absolute Path The fix was absurdly simple. Append this to the `ai-news-digest` SKILL.md, telling the Agent exactly how to invoke the tool: > **create-img invocation (use directly, do not search)** > > 1. Read the skill spec first: `/home/openclaw/.openclaw/plugin-skills/create-img/SKILL.md` > 2. Run the script (command below) > 3. After generation, upload to Lark with `lark-cli im images create --file ` to get the `image_key` ```bash source /home/openclaw/.bashrc /home/openclaw/workspace/.venv314/bin/python \ /home/openclaw/.openclaw/plugin-skills/create-img/scripts/omniroute_image_batch.py \ --prompt "" \ --size 1536x864 \ --quality high \ --format png \ --out-dir /home/openclaw/.openclaw/workspace/tmp/ai-news-banner ``` That's it. Give the Agent a **complete, copy-pasteable command** including: - Absolute path to the Python interpreter (in the virtual environment) - Absolute path to the script - All required parameters - Output directory ## Results First run after the fix: | Metric | Before | After | Change | |--------|--------|-------|--------| | Total messages | 165 | 54 | -67% | | exec tool calls | 44 | 7 | -84% | | tool_result size | 1,122 KB | 204 KB | -82% | | Runtime | 1200s+ (timeout) | 317s | From timeout to 5 minutes | Messages cut by two-thirds, exec calls cut by 84%, tool result volume cut by 82%. **One file path beat any algorithm optimization.** ## Why This Bug Was So Hard to Spot This bug had three characteristics that made it elusive: ### 1. It Didn't Fail Every Time If the Agent found the tool path on its first search round and didn't trigger compaction, the task completed — just slower. The timeout loop only kicked in when context happened to cross the compaction threshold during the search phase. This made it a **probabilistic timeout**, not a deterministic failure, and debugging instincts naturally drifted toward "maybe the upstream API is slow." ### 2. The Agent's Search Behavior Looked Reasonable Reading the logs, every single action was individually reasonable: didn't know the path → searched the filesystem → read files to confirm → checked the environment. This is a "smart but inefficient" behavior pattern. You wouldn't think the Agent did anything wrong — just that it was strangely slow. ### 3. Vague References in SKILL.md Don't Error Writing `use create-img to generate images` is perfectly valid syntax — SKILL.md has no type checking, no "unresolved reference" warning. The Agent won't say "I can't find create-img" and stop — it will go find it itself. This **silent degradation into search mode** is the hardest category of problems to catch. ## Deeper Lesson: Principles for Writing AI Agent Instructions This bug made me realize something fundamental: **the precision required for AI Agent instructions is far higher than for human documentation.** ### For Humans vs. For Agents When writing docs for humans, you can write "use create-img to generate images" — a human developer will ask a colleague, search the wiki, browse the directory tree. These exploration actions cost essentially nothing (no tokens, no context consumption). When writing instructions for an Agent, the same vague reference triggers exploration actions where **every step has a cost**: - Each `find` / `grep` consumes one tool call - Each tool result occupies context space - Context bloat triggers compaction - Compaction may discard previously found information **An Agent's "curiosity" has a price, measured in context space.** ### SKILL.md Writing Rules Based on this incident, I set rules for SKILL.md authoring: **1. All external tool references must include absolute paths.** Not "use create-img" but `/home/openclaw/.openclaw/plugin-skills/create-img/scripts/omniroute_image_batch.py`. Five extra lines in SKILL.md beats 15 exec calls at runtime. **2. Executable commands must be directly copy-pasteable.** Include interpreter paths, virtual environment activation, all parameters. The Agent should be able to paste the command straight into `exec` without any path resolution. **3. Explicitly write "do not search."** Alongside the full path, tell the Agent "use the following paths directly, do not search the filesystem." This seems redundant, but Agents sometimes "verify" paths you give them — one "do not search" line saves that verification step. **4. Hard-code environment dependencies.** Python virtual environment paths, Node.js versions, config files that need sourcing — all hard-coded. Don't assume the Agent knows the runtime environment. ### An Analogy This is like the difference between writing a README and writing a Dockerfile: - **README** (for humans): "Install Python 3.14, then pip install -r requirements.txt in a virtual environment" - **Dockerfile** (for machines): `RUN /usr/local/bin/python3.14 -m venv /app/.venv && /app/.venv/bin/pip install -r /app/requirements.txt` SKILL.md is an instruction set for Agent execution, not documentation for human reading. Its precision requirements are closer to a Dockerfile than a README. ## The Bigger Picture: Where Outer Loop Performance Bottlenecks Live Looking back at the three posts in this series, a clear pattern emerges: | Problem | Root Cause | Fix | |---------|-----------|-----| | Compaction silently swallows replies | Harness compression strategy doesn't protect critical info | Add safety margin + structured prompt | | Vector search broken unnoticed | Missing config causes silent degradation | Add embedding provider config | | Cron job timeout | SKILL.md missing path causes search bloat | Add one absolute path | The common thread: **none of these were model capability problems. All were Outer Loop configuration/instruction precision issues.** This validates the core thesis from the first post: in the harness era, **system quality bottlenecks are never in model inference — they're in everything outside the model** — context management, tool call specifications, instruction precision, degradation paths, monitoring and alerting. One file path eliminated 84% of tool calls. One embedding provider config revived vector search. One safety margin parameter prevented compaction from swallowing replies. None of these fixes required a stronger model — they required **better harness maintenance**. When your Agent underperforms, don't reach for a model upgrade first. Open the logs, count the tool calls, watch how context grows. The answer is probably in one line of your SKILL.md. --- > This is the third post in the OpenClaw Production Notes series. If you're also running AI Agent cron jobs, I hope the "one file path" lesson saves you some debugging time. The series will continue — the next topic might be tool call cost control and context budget management for Agents. --- title: "OpenClaw Memory in Practice: From 'Vector Search Is Down But Everything Still Works' to Zero-Cost NVIDIA Embeddings" description: "OpenClaw's vector retrieval silently failed — but BM25 text search kept the memory system running for two weeks unnoticed. When you discover 'it works without embeddings,' should you even bother fixing it? Here's how I used NVIDIA's free embedding API to complete the picture, and what I learned about when vector search actually matters." url: "https://zhuoqidev.com/en/posts/openclaw-memory-text-to-vector/" language: "en" author: "Liu ZhuoQi" date: "2026-06-20" updated: "2026-07-30" series: "OpenClaw Production Notes" tags: ["OpenClaw","AI Agent","Memory System","Embedding","NVIDIA","Vector Search","BM25"] translation: "https://zhuoqidev.com/posts/openclaw-memory-text-to-vector/index.md" --- # OpenClaw Memory in Practice: From 'Vector Search Is Down But Everything Still Works' to Zero-Cost NVIDIA Embeddings > [!NOTE] > This is the second post in the [OpenClaw Production Notes](https://zhuoqidev.com/series/openclaw-production-notes/) series. [The first](https://zhuoqidev.com/posts/openclaw-pitfalls/) covered compaction silently swallowing replies and defensive Outer Loop design. This one is about the memory system—specifically, a real experience that made me rethink whether vector search is worth the trouble. ## Background: How OpenClaw's Memory Retrieval Works OpenClaw uses a **hybrid retrieval system** — vector (embedding similarity) and BM25 (keyword matching) fused at a 7:3 weight ratio, then passed through PPO-adaptive five-dimensional reranking (recency 0.35 + frequency 0.25 + semantic 0.25 + saliency 0.15 + procedural on-demand). The diagram below places “it still worked” and “why repair it anyway?” on the same path. The upper half shows BM25 maintaining basic retrieval while the vector lane is unavailable. The lower half shows both result streams entering one hybrid ranker after Embedding is restored. ![OpenClaw's dual-engine BM25 and vector memory retrieval architecture](https://zhuoqidev.com/_astro/dual-engine-retrieval-bilingual-v1-4k.BEI1Rrbf_Z2mmGrT.webp "Dual-engine retrieval: BM25 does not depend on Embeddings and preserves basic availability during a vector-lane failure. Once vector retrieval returns, it improves recall for paraphrases and semantically related memories.") Looks great on paper. But after two weeks in production, I discovered something puzzling. ## Discovery: Vector Search Was Broken, and Nobody Noticed During a routine health check, I ran `openclaw memory status` and spotted an anomaly: ``` Vector store: unknown Embedding cache: enabled (0 entries) FTS: ready ``` **Vector store unknown, embedding cache empty** — vector retrieval wasn't working at all. But here's the strange part: the memory system was retrieving memories daily, Dreaming consolidation ran on schedule, and the Agent's response quality showed no obvious degradation. Look at the FTS (full-text search) line: **ready**. BM25 had been quietly carrying the entire load. The root cause: the `memorySearch` section in `openclaw.json` was missing the embedding provider configuration. On startup, embedding initialization failed silently, and the system degraded to pure BM25 mode. No error, no warning — it just used the 30%-weighted retrieval channel alone, and it held up. ### What This Means 903 memory chunks, 512 recall entries — the entire memory system ran on **pure text search** for two weeks, and nobody noticed anything was wrong. This forced me to reexamine a basic assumption: **how much does vector search actually matter for a coding Agent's memory?** ## Why BM25 Was Good Enough Here The memory content of a coding Agent has several characteristics that happen to be BM25's sweet spot: **1. Memory content is highly structured.** OpenClaw's memory files are Markdown with frontmatter (`name`, `type`, `description`), heading hierarchies, and precise technical terms. Not vague natural language, but things like `compaction safeguard`, `Gemini 429 quota exhaustion`, `fable-5 global ban`. BM25 handles this kind of semi-structured technical documentation quite well. **2. Queries and documents share the same vocabulary.** When someone asks "how to configure compaction," the memory literally contains the word "compaction." There's no semantic gap where the user says "apple" but the document says "fruit." In programming, terminology is highly standardized — `429` is `429`, `context overflow` is `context overflow`. This is exactly what BM25 excels at: **exact term matching**. **3. The memory corpus is in the thousands, not millions.** 99 files, 903 chunks — at this scale, BM25's precision and recall are both reasonable because the candidate set is small. Vector search's advantage shows at massive scale; at 1000 chunks, the edge is negligible. **4. PPO five-dimensional reranking compensates for retrieval quality.** Even if BM25's initial ranking is imprecise, the recency dimension (recently used memories rank higher) and frequency dimension (frequently recalled memories get boosted) push the right memories to the top. These dimensions don't depend on embeddings — they're pure usage statistics. One-line summary: **When terms are precise, data is in the thousands, and usage-based reranking is in play, BM25 alone is a perfectly adequate retrieval solution.** This echoes what I wrote in my [memory selection framework](https://zhuoqidev.com/posts/memory-choice-framework/): **don't use RAG unless you have to.** Vector search isn't the default — it's what data volume and semantic complexity force you into. ## So Why Fix It? If BM25 was enough, why did I ultimately add vector search back? Three reasons: **1. Cross-language retrieval.** My memory files mix Chinese and English — some incident records are in Chinese ("Gemini 资源池配额枯竭"), others in English ("unrestricted key enforcement"). BM25 can't do cross-language fuzzy matching: searching "quota exhausted" in English won't find the Chinese description of the same incident. Embedding spaces naturally align semantics across languages and synonyms. **2. Concept-level retrieval.** Ask "have we seen the bot not responding before?" — BM25 matches the literal words "not responding," but won't match `compaction silently swallowing replies` (which is exactly the answer). Vector search brings "bot not responding" and "silently swallowing replies" closer in semantic space. As memories accumulate, these concept-level associations become increasingly important. **3. Hybrid complementarity.** Vectors excel at semantic recall, BM25 excels at exact matching — the combination is strictly better than either alone. The 7:3 weight split means BM25 gives you a guaranteed floor of exact matches, while vectors add bonus semantic discovery. The first two weeks on pure BM25 were "good enough," but "good enough" and "good" are separated by exactly one vector index. ## Implementation: Zero-Cost Fix with NVIDIA's Free Embedding API The key constraint: **no additional cost.** OpenClaw is already burning model API credits; if embeddings also cost per-token (OpenAI's `text-embedding-3-small` runs $0.02 per million tokens), even the small cost for 903 chunks of continuous indexing and per-query embedding is unnecessary. NVIDIA's `nv-embed-v1` is a free embedding model available through the NVIDIA API Catalog, fully compatible with the OpenAI API format. It performs in the same tier as OpenAI's embedding models on MTEB benchmarks, outputs 4096 dimensions, and has solid multilingual support. ### Getting an API Key 1. Sign up at [NVIDIA API Catalog](https://build.nvidia.com/) 2. Navigate to the NV-Embed-V1 model page, click "Get API Key" 3. You get an `nvapi-` prefixed key — free to use, rate-limited but more than sufficient for Agent memory workloads ### Configuring OpenClaw Add a `memorySearch` section to the agent defaults in `openclaw.json`: ```json { "agents": { "defaults": { "memorySearch": { "enabled": true, "provider": "openai", "remote": { "baseUrl": "https://integrate.api.nvidia.com/v1", "apiKey": "nvapi-your-key-here" }, "model": "nvidia/nv-embed-v1" } } } } ``` A critical detail: `provider` is set to `"openai"`, not `"nvidia"`. NVIDIA API Catalog's endpoint is fully compatible with OpenAI's `/v1/embeddings` format — same request body structure, same response format. OpenClaw just needs to know "this is an OpenAI-compatible embedding endpoint," and `baseUrl` handles the routing to NVIDIA's servers. This is a testament to the OpenAI embedding API becoming the de facto standard: NVIDIA, Alibaba Tongyi, Jina, Cohere — they all implement the same interface. When choosing a provider, don't overthink it. As long as the API is compatible, `"openai"` is the universal adapter. ### Rebuilding the Index After updating the config, restart the Gateway and rebuild the memory index: ```bash openclaw memory reindex --deep ``` Output: ``` Indexed: 99/99 files · 903 chunks Vector dims: 4096 Embedding cache: enabled (779 entries) ``` All 99 files re-embedded and indexed into the vector store. The 4096 dimensions from nv-embed-v1 are higher than OpenAI's 1536 (text-embedding-3-small), theoretically providing a more granular semantic space. ### Verifying Hybrid Retrieval Testing with a semantically fuzzy query: ```bash openclaw memory search "what was that incident where the bot stopped responding" ``` Top-1 result: the compaction silently-swallowed-reply memory entry — impossible under pure BM25 (the memory contains neither "stopped" nor "responding," only "silently discarded" and "Compaction output empty summary"). Cross-language test: ```bash openclaw memory search "Gemini quota issue" ``` Matched the Chinese-language memory entry "Gemini 资源池整体配额枯竭" (Gemini resource pool quota exhaustion). BM25 cannot make this match. ## Before and After | Dimension | Before (BM25 only) | After (Vector + BM25 7:3) | |-----------|----------|----------| | Exact term matching | Works | Works | | Semantic/concept matching | No | Yes | | Cross-language retrieval | No | Yes | | Synonym matching | No | Yes | | Cost | $0 | $0 (NVIDIA free tier) | | Index size | 903 chunks | 903 chunks | | Embedding cache | 0 | 779 entries | | Retrieval latency | <10ms | <50ms (includes API call) | | System stability | Stable | Stable (BM25 still backstops) | The last row deserves emphasis: even if NVIDIA's API occasionally times out or becomes unavailable, the system automatically degrades back to pure BM25 — identical to the pre-fix state. **Vector search is a bonus, not a dependency.** ## Lessons Learned **1. Get it running first, optimize later.** Vector search was misconfigured for two weeks and nobody noticed. This tells you that the core value of a memory system isn't retrieval algorithm precision — it's whether memories are being written, managed, and surfaced when needed. BM25 handles 80% of retrieval scenarios; the remaining 20% of semantic matching is an optimization, not a requirement. **2. The right mental model for hybrid retrieval is "BM25 as floor, vectors as upside."** Don't flip it to "vectors as primary, BM25 as backup." In production, determinism (BM25's exact match will always find keyword-identical memories) matters more than probability (vector similarity might surface irrelevant content). In the 7:3 weight split, that 3 is the system's safety net. **3. OpenAI-compatible format is the de facto standard for embeddings.** Whether you use NVIDIA, Alibaba, Jina, or Cohere, the configuration pattern is identical: `provider: "openai"` + `baseUrl: "vendor endpoint"` + `model: "model name"`. No vendor-specific adapters needed. This also means: if NVIDIA's free tier disappears someday, switching to another free embedding service is a two-line config change. **4. Free ≠ inferior.** NVIDIA nv-embed-v1 is an MTEB-competitive model with 4096 dimensions and good multilingual support. It's free because NVIDIA wants to grow its API ecosystem — what you get is a genuinely competitive embedding model, not a crippled version. **5. "Is vector search worth it?" depends on your data characteristics.** If your memories are all precise technical terms, data is in the thousands, and queries use the same vocabulary as documents — BM25 is enough, vector search is optional. If your memories include cross-language content, natural language descriptions, and concept-level associations — vector search goes from "optional" to "necessary." My scenario was the latter (mixed Chinese/English, incident descriptions vs. technical terminology), so I added it back. --- > Next: [One File Path Eliminated 84% of Tool Calls — A Cron Job Debugging Story](https://zhuoqidev.com/posts/openclaw-cron-skill-optimization/). When your AI Agent spends 15 exec calls every run searching for a skill's file path, inflating message count from 54 to 165, and the root cause is a single missing absolute path in the SKILL.md. This matters more than any algorithm tuning. --- title: "Claude's Tool Calling Paradigm Shift: A Deep Dive into Programmatic Tool Calling and Dynamic Filtering" description: "Anthropic's Programmatic Tool Calling and Dynamic Filtering aren't just feature additions — they represent a paradigm shift in Agent architecture: from natural language orchestration to code-driven orchestration, from full context injection to on-demand filtering. Synthesizing multiple non-AI-written deep-dive articles, this post covers architecture, benchmarks, and production patterns." url: "https://zhuoqidev.com/en/posts/claude-programmatic-tool-calling-dynamic-filter/" language: "en" author: "Liu ZhuoQi" date: "2026-06-13" updated: "2026-07-30" series: "Agent Architecture Deep Dives" tags: ["Claude","AI Agent","Agent Architecture","Tool Calling","Context Engineering","Programmatic Tool Calling","Dynamic Filtering","Code Execution"] translation: "https://zhuoqidev.com/posts/claude-programmatic-tool-calling-dynamic-filter/index.md" --- # Claude's Tool Calling Paradigm Shift: A Deep Dive into Programmatic Tool Calling and Dynamic Filtering > **TL;DR** The important change is not “two more tool features.” It is the movement of multi-step orchestration into a code-execution environment, with only a compact result returning to model context. ## Background: The Cost Problem in Agent Tool Calling In traditional agent tool-calling, every tool invocation requires a full cycle of "model inference → tool execution → result return → model re-inference." This seemingly natural loop breaks down at scale in three ways: 1. **Context Pollution**: Every tool result is injected verbatim into the context window. Fetch expense reports for 20 employees, and 2,000+ line items enter context — even though you only need to know "which 3 people exceeded their budget." 2. **Inference Overhead**: Each tool call demands a full model inference pass. Five tools = five inference passes, each costing hundreds of milliseconds to seconds. 3. **Noise Degrades Accuracy**: When the context window is packed with intermediate results, the model must find signal in noise. [Context Rot research](https://arxiv.org/abs/2509) shows LLM performance on complex tasks drops 50-70% as context grows. As Florian Bruniaux puts it in the [Claude Code Architecture Guide](https://cc.bruniaux.com/guide/architecture/): **"The Outer Loop — everything outside the model: context management, tool invocation, verification, memory consolidation — increasingly determines system quality more than model inference itself."** Anthropic's suite of tool-use enhancements, released between November 2025 and February 2026, are fundamentally about solving Outer Loop efficiency. Among them, **Programmatic Tool Calling (PTC)** and **Dynamic Filtering** represent the deepest paradigm shift. The diagram below establishes the article's **cost model**. The left side repeatedly invokes the model and injects full tool results. The right side keeps orchestration, filtering, and aggregation inside the sandbox and returns only the compact answer. The benchmark and deployment discussions later in the article map back to this path difference. ![The Outer Loop difference between turn-by-turn tool calling and Programmatic Tool Calling](https://zhuoqidev.com/_astro/tool-use-outer-loop-bilingual-v1-4k.Db399Pjy_Z1vIzIP.webp "Tool-use Outer Loop: PTC does more than reduce API calls—it keeps intermediate data inside the container. Dynamic Filtering removes irrelevant external results before they consume context.") --- ## Programmatic Tool Calling: Code-Driven Orchestration ### The Core Paradigm Shift Michael Ridland, writing on the [Team 400 blog](https://team400.ai/blog/2026-03-claude-programmatic-tool-calling-ai-agents), pinpoints the traditional bottleneck: > "Every time an agent needs to call a tool — query a database, check an API, read a file — it has to do a full round trip back to the model. The model generates a tool call, your code executes it, the result goes back to the model, the model processes it, and maybe generates another tool call. Repeat." PTC's paradigm shift: **Instead of Claude requesting tools one at a time and having results return to its context, Claude writes Python code that orchestrates all tool calls internally. Only the final `stdout` enters the context window.** ```mermaid flowchart TB T["Traditional
3 tools = 3 inference passes
3× intermediate results in context"] --> t0["Prompt"] --> t1["Claude"] --> t2["Tool 1"] --> t3["Result 1"] --> t4["Claude"] --> t5["Tool 2"] --> t6["Result 2"] --> t7["Claude"] --> t8["Answer"] P["PTC
3 tools = 1 inference pass
only final output in context"] --> p0["Prompt"] --> p1["Claude"] --> p2["writes Python"] --> p3["code calls Tool 1, 2, 3"] --> p4["stdout"] --> p5["Claude"] --> p6["Answer"] classDef head fill:#fff3bf,stroke:#f08c00 classDef ctx fill:#ffe3e3,stroke:#e03131 classDef out fill:#d3f9d8,stroke:#2f9e44 class T,P head class t3,t6 ctx class p4 out ``` The concept is simple, but its implications are profound — **it moves orchestration logic from the model's "reasoning chain" into a "code execution environment."** Loops, conditionals, data transformations, and error handling all become explicit code rather than implicit model reasoning. Masaki Hirokawa of Claude Lab, in his [production guide](https://claudelab.net/en/articles/api-sdk/claude-api-programmatic-tool-calling-production-guide), summarizes: > "Claude excels at writing code. By letting it express orchestration logic in Python rather than through natural language tool invocations, you get more reliable, precise control flow." ### How It Works: Container-Based Orchestration PTC relies on the Code Execution tool (sandboxed container) to run: 1. **Mark tools as callable**: Use the `allowed_callers` field to specify which tools can be invoked from code 2. **Claude generates orchestration code**: Claude writes a Python script containing multi-step tool calls, data processing, and control flow 3. **Container executes and pauses**: When code calls a tool, the container pauses and the API returns a `tool_use` block 4. **You provide tool results**: Results go back to the code (not the model context), and execution continues 5. **Only final output reaches Claude**: All intermediate results are filtered; Claude only sees the final `stdout` ```json { "tools": [ {"type": "code_execution_20260120", "name": "code_execution"}, { "name": "query_database", "description": "Execute SQL. Returns rows as JSON: id (str), name (str), revenue (float).", "input_schema": {...}, "allowed_callers": ["code_execution_20260120"] } ] } ``` The critical field is `allowed_callers`, which has three possible values: | `allowed_callers` Value | Meaning | |---|---| | Omitted / `["direct"]` | Traditional calling only | | `["code_execution_20260120"]` | Callable from sandbox code only | | `["direct", "code_execution_20260120"]` | Both modes (**not recommended** — confuses Claude) | Bruniaux's architecture guide highlights an important safety note: **`allowed_callers` is not a hard security boundary.** It's a strong guidance (Claude is trained to respect it), but your client should still be prepared to handle a direct `tool_use` for any tool it defines. ### Container Lifecycle Containers have a 4.5-minute idle timeout and a 30-day maximum lifetime. Always check the `expires_at` field on every response. If the container expires, Claude treats the tool call as timed out and retries. ### Real Performance Data Claude Lab provides detailed benchmark comparisons: ``` Benchmark: 10-URL web research task Traditional Tool Use: Round trips: 10 (1 search + 9 page fetches) Inference passes: 11 Average latency: 45 seconds Token consumption: ~120,000 tokens PTC: Round trips: 2 (1 code generation + 1 result return) Inference passes: 2 Average latency: 8 seconds (parallel fetching) Token consumption: ~25,000 tokens ``` Anthropic's official data validates this: - Average input token reduction of **37%** on complex research tasks (43,588 → 27,297 tokens) - BrowseComp accuracy jumped from 42% to **71%** (PTC was the key unlock) - On a 75-tool project-management agent benchmark, billed tokens dropped ~**38%** with no accuracy loss Community analyst Shayan Tabe's independent analysis confirms the ~37% overall token reduction, though this number has not been officially endorsed by Anthropic. ### Production Patterns: Four PTC Programming Paradigms PTC's real power lies in four concrete programming patterns, all achievable in a single inference pass: #### 1. Batch Processing ```python regions = ["West", "East", "Central", "North", "South"] results = {} for region in regions: data = await query_database(f"SELECT * FROM sales WHERE region = '{region}'") results[region] = sum(row["revenue"] for row in data) top_region = max(results.items(), key=lambda x: x[1]) print(f"Top region: {top_region[0]} with ${top_region[1]:,}") ``` #### 2. Early Termination ```python for endpoint in ["us-east", "eu-west", "apac"]: status = await check_health(endpoint) if status == "healthy": print(f"Found healthy endpoint: {endpoint}") break # Don't check remaining endpoints ``` #### 3. Conditional Tool Selection ```python file_info = await get_file_info(path) if file_info["size"] < 10000: content = await read_full_file(path) else: content = await read_file_summary(path) print(content) ``` #### 4. Data Filtering ```python logs = await fetch_logs(server_id) errors = [log for log in logs if "ERROR" in log] print(f"Found {len(errors)} errors") for error in errors[-10:]: # Return only last 10 print(error) ``` Team 400's Michael Ridland captures the key insight: > "If 20 database queries return 5,000 rows total but only 3 employees are over budget, the model only sees those 3 employees. That's potentially a 100x reduction in tokens." ### When to Use PTC (and When Not To) The combined wisdom from Claude Lab and Anthropic's docs yields a clear decision matrix: **Strong fit:** - Processing large datasets where you only need aggregates or summaries - Multi-step workflows with 3+ dependent tool calls - Filtering, sorting, or transforming results before Claude sees them - Tasks where intermediate data shouldn't influence Claude's reasoning - Parallel operations across many items (checking 50 endpoints) **Weak fit:** - Strictly sequential workflows where each step depends on Claude reasoning over the previous result - A small number of tool calls with small responses - Tools requiring immediate user feedback between calls Anthropic's internal evaluation on τ²-bench reveals PTC's blind spot: in airline/retail/telecom domains where each turn makes only 1-2 sequential tool calls, PTC left scores unchanged and cost ~8% more. **Sequential single-call workflows do not benefit.** ### Constraints and Pitfalls PTC is not a silver bullet. Key constraints to consider during architecture: | Constraint | Detail | |---|---| | **No ZDR support** | Cannot be used where Zero Data Retention compliance is required | | **No MCP tools** | Tools provided by MCP connectors cannot be called programmatically | | **No `strict: true`** | Structured outputs with `strict: true` are incompatible with PTC | | **No forced tool choice** | Cannot use `tool_choice` to force programmatic calling of a specific tool | | **Container timeout** | Slow tool execution may cause container timeout and retry loops | | **Debugging complexity** | Logic lives in generated code — inspect code execution output to understand failures | Michael Ridland specifically warns about debugging: > "When something goes wrong in a traditional tool-calling flow, you can trace each step. With PTC, the logic lives in generated code. You need to inspect the code execution output to understand what happened. Build in logging." --- ## Dynamic Filtering: Context Slimming for Web Search ### The Problem Web search is the most token-intensive task. The traditional flow: initiate query → get search results → fetch full HTML from multiple websites → reason over everything in context. But the context pulled in from search is often mostly irrelevant — navigation bars, ads, footers, recommendations... Anthropic's [official blog post](https://claude.com/blog/improved-web-search-with-dynamic-filtering) describes the solution: > "Claude's web search and web fetch tools now automatically write and execute code to post-process query results. Instead of reasoning over full HTML files, Claude can dynamically filter the search results before loading them into context, keeping only what's relevant and discarding the rest." ### How It Works Dynamic Filtering is essentially PTC principles applied natively to web search — let Claude write Python to pre-process search results: ```mermaid flowchart TB Q["Query"] --> S["Search API"] --> H["10 raw HTML pages"] H -->|"Traditional"| A1["all enter context"] --> A2["Claude reasons"] H -->|"Dynamic Filtering"| B1["Claude writes code to extract key data"] --> B2["filtered summary enters context"] --> B3["Claude reasons"] classDef ctx fill:#ffe3e3,stroke:#e03131 classDef out fill:#d3f9d8,stroke:#2f9e44 class A1 ctx class B2 out ``` ### Benchmark Results Anthropic evaluated Dynamic Filtering on two rigorous benchmarks: **BrowseComp Dataset:** | Model | Without Filtering | With Filtering | Improvement | |---|---|---|---| | Sonnet 4.6 | 33.3% | 46.6% | **+13.3 pp** | | Opus 4.6 | 45.3% | 61.6% | **+16.3 pp** | **DeepSearchQA:** Tests whether an agent can systematically plan and execute multi-step searches without missing answers. | Model | Without Filtering | With Filtering | Improvement | |---|---|---|---| | Sonnet 4.6 (F1) | 52.6% | 59.4% | +6.8 pp | | Opus 4.6 (F1) | 69.8% | 77.3% | +7.5 pp | Overall, Dynamic Filtering improved accuracy by an average of **11%** while using **24% fewer input tokens**. Poe by Quora's internal team validated these findings: > "Opus 4.6 with dynamic filtering achieved the highest accuracy on our internal evals when tested against other frontier models." — Gareth Jones, Product and Research Lead > > "The model behaves like an actual researcher, writing Python to parse, filter, and cross-reference results rather than reasoning over raw HTML in context." ### Configuration ```json { "model": "claude-opus-4-6", "max_tokens": 4096, "tools": [ {"type": "web_search_20260209", "name": "web_search"}, {"type": "web_fetch_20260209", "name": "web_fetch"} ], "messages": [ { "role": "user", "content": "Search for the current prices of AAPL and GOOGL, then calculate which has a better P/E ratio." } ] } ``` Note: With `web_search_20260209` / `web_fetch_20260209`, Dynamic Filtering is **enabled by default** on Sonnet 4.6 and Opus 4.6. To disable it (e.g., for ZDR compliance), use `"allowed_callers": ["direct"]`. **Important:** The basic `web_search_20250305` version qualifies for ZDR, but the Dynamic Filtering version does not by default because it relies on internal code execution. --- ## The Trinity: PTC + Tool Search + Tool Use Examples Anthropic's Advanced Tool Use is actually three complementary features: | Feature | Problem Solved | When to Use | |---|---|---| | **Tool Search Tool** | Tool definitions bloating context | 50+ MCP tools | | **Programmatic Tool Calling** | Intermediate results polluting context | 3+ step tool workflows | | **Tool Use Examples** | Schema alone can't express usage patterns | Complex nested parameter tools | Bruniaux's architecture guide offers clear prioritization: > "Address your biggest bottleneck first: context bloated by tool definitions? → Tool Search. Large intermediate results? → PTC. Parameter errors? → Tool Use Examples." ### Tool Search: On-Demand Discovery A real-world example: connecting 5 MCP servers (GitHub 35 tools + Slack 11 tools + Sentry 5 tools + Grafana 5 tools + Splunk 2 tools) requires preloading 58 tool definitions into context, consuming ~**55K tokens**. With Tool Search enabled, only ~500 tokens of search tool stay resident, and 3-5 matched tools (~3K tokens) load on demand. **85% token overhead eliminated.** Internal benchmarks: - Opus 4 tool selection accuracy: 49% → **74%** - Opus 4.5: 79.5% → **88.1%** ### Tool Use Examples: Teaching Claude Your Tool Conventions JSON Schema defines structure, but not: when to include optional parameters, which combinations make sense, or what conventions your API expects. ```json { "name": "create_ticket", "input_schema": { /* ... */ }, "input_examples": [ { "title": "Login page returns 500 error", "priority": "critical", "labels": ["bug", "authentication", "production"], "reporter": { "id": "USR-12345", "name": "Jane Smith", "contact": {"email": "jane@acme.com", "phone": "+1-555-0123"} }, "due_date": "2026-06-13" }, { "title": "Add dark mode support", "labels": ["feature-request", "ui"], "reporter": {"id": "USR-67890", "name": "Alex Chen"} }, { "title": "Update API documentation" } ] } ``` Three examples teach Claude: - Dates use YYYY-MM-DD, user IDs follow USR-XXXXX, labels use kebab-case - Critical bugs require full contact info + escalation; feature requests have reporter but no escalation; internal tasks need only title Internal testing: complex parameter handling accuracy improved from **72% to 90%**. --- ## Architecture Decisions: When to Deploy, When to Bypass ### Understanding the Outer Loop Bruniaux's architecture guide presents a core philosophy — **"Less scaffolding, more model."** This is embodied in Claude Code: > "No intent classifier. No task router. No RAG pipeline. No DAG orchestrator. No planner/executor split. The model itself decides when to call tools, which tools to call, and when it's done." PTC and Dynamic Filtering extend this philosophy — they don't add new abstraction layers, they **remove unnecessary ones**: no need to pass intermediate results to the model and wait for it to "think" about the next step; just orchestrate directly in code. ### Model Compatibility PTC requires `code_execution_20260120`, currently supported on: - Claude Opus 4.5+ - Claude Sonnet 4.5+ - Claude Fable 5 / Mythos 5 Dynamic Filtering requires `web_search_20260209`, enabled by default on Opus 4.6 / Sonnet 4.6. ### Deployment Checklist Team 400 provides a practical four-step adoption path: 1. Enable the code execution tool in your API calls 2. Add `allowed_callers` to tools that should be callable from code 3. Test with prompts that require multiple tool invocations 4. Compare latency, token usage, and output quality against your traditional approach Bruniaux emphasizes one easy-to-miss pitfall: **never set `["direct", "code_execution_20260120"]` on the same tool.** Pick one — this gives Claude clearer guidance on the optimal usage pattern. --- ## Summary Programmatic Tool Calling and Dynamic Filtering represent a significant paradigm shift in Agent architecture: **from "everything goes through model inference" to "code-driven orchestration + on-demand filtering."** This isn't a concept Anthropic invented from scratch — it's the systematic productization of a principle validated repeatedly in practice: **let the model do what it does best (reasoning and decision-making), and let code do what code does best (loops, filtering, parallel execution).** | Dimension | Traditional Tool Calling | PTC / Dynamic Filtering | |---|---|---| | Orchestration logic location | Model's reasoning chain | Python code | | Intermediate results | All enter context | Filtered in code, never hit context | | Inference passes for N tools | N+1 | 1~2 | | Token consumption | Linear growth with tool count | Major reduction (37%~80%) | | Debugging approach | Step through each turn | Inspect code execution output logs | | Best for | Simple lookups, human-in-the-loop decisions | Batch processing, data aggregation, multi-step research | This article synthesizes the following sources (all human-authored, not AI-generated): - [Anthropic Engineering: Introducing advanced tool use](https://www.anthropic.com/engineering/advanced-tool-use) (Official blog, Nov 2025) - [Claude Blog: Improved Web Search with Dynamic Filtering](https://claude.com/blog/improved-web-search-with-dynamic-filtering) (Official blog, Feb 2026) - [Claude API Docs: Programmatic tool calling](https://platform.claude.com/docs/en/agents-and-tools/tool-use/programmatic-tool-calling) (Official documentation) - [How Claude Code Works: Architecture & Internals](https://cc.bruniaux.com/guide/architecture/) (Florian Bruniaux technical analysis, Feb 2026) - [Claude Lab: PTC Production Guide](https://claudelab.net/en/articles/api-sdk/claude-api-programmatic-tool-calling-production-guide) (Masaki Hirokawa, Mar 2026) - [Team 400: Why PTC Matters for AI Agent Performance](https://team400.ai/blog/2026-03-claude-programmatic-tool-calling-ai-agents) (Michael Ridland, Mar 2026) --- title: "OpenClaw in Production: When the Most Advanced Memory System Meets the Quietest Failure" description: "A full-chain production battle log: from startup failures and Feishu message silent drops to production stability — compaction safeguard, five-layer debugging, model-harness fit, and memory system comparison." url: "https://zhuoqidev.com/en/posts/openclaw-pitfalls/" language: "en" author: "Liu ZhuoQi" date: "2026-05-27" updated: "2026-07-30" series: "OpenClaw Production Notes" tags: ["OpenClaw","AI Agent","Feishu","Memory System","Compaction","Debugging"] translation: "https://zhuoqidev.com/posts/openclaw-pitfalls/index.md" --- # OpenClaw in Production: When the Most Advanced Memory System Meets the Quietest Failure ## Why OpenClaw > Each paradigm shift in coding agents changes who owns the **verification loop**. The prompt engineering era (Copilot, 2021): humans write prompts, AI completes, humans verify — verification entirely in human hands. The context engineering era (Cursor, 2023): AI generates code blocks, humans review diffs — humans still lead, but AI starts doing lightweight lint/fix self-checks. The harness engineering era (Claude Code / Codex CLI / OpenClaw, 2025): AI writes code, AI runs tests, AI reads errors, AI fixes — the verification loop shifts from human to harness. **Humans no longer review every output line by line; they design and maintain the rules of the verification system.** The entire Outer Loop — everything outside the model: context management, tool invocation, verification, memory consolidation — now matters more for system quality than the model's reasoning ability itself. OpenClaw is the most ambitious on the memory side of this trend, but the least battle-tested on harness stability. > > Within the same era, there's a critical fork: **Vibe Coding** (Bolt.new, Lovable, Replit) throws verification at the user — "generate and ship"; **Engineering Rigor** (Claude Code, Aider, OpenClaw) encodes verification into the harness — retry on test failure. The gap between them isn't the model; it's the Outer Loop design philosophy. Here's a mindmap of OpenClaw's architecture — every pitfall ahead ties back to these components: ```mermaid mindmap root((OpenClaw)) 📨 Message Ingress Feishu WeChat Work WebChat Telegram 🔌 Channels Layer Message routing allowlist Deduplication dedup Session binding envelope 🖥️ Gateway Daemon WebSocket Server :18789 Authentication device pairing Scheduled tasks CRON engine 🔄 Agent Loop Message ingestion agent RPC Context assembly bootstrap session skills Model inference multi-backend fallback Tool execution memory shell web Reply generation streaming delivery 🧠 Memory System MEMORY.md long-term memory Daily logs YYYY-MM-DD.md DREAMS.md consolidation Vector BM25 hybrid search 7:3 ``` OpenClaw is a harness-era autonomous coding agent, cross-model CLI, supporting DeepSeek / Anthropic / OpenAI backends. What drew me in was its **memory system** — the most ambitious among all production-available coding agents: - **PPO cognitive weight adaptation**: the only production tool that uses reinforcement learning to adjust memory retrieval weights. Five-dimensional retrieval signal weighting (recency 0.35 + frequency 0.25 + semantic 0.25 + saliency 0.15 + procedural on demand), with time-based decay - **Triple sleep consolidation**: Light Sleep (Jaccard dedup, zero LLM cost) → REM Sleep (confidence scoring) → Deep Sleep (three-condition promotion gate: score≥0.80 + merge≥3 + recall≥3), automatically hardening short-term experiences into long-term memory - **Vector + BM25 hybrid search** (7:3): more robust than pure vector retrieval But after three weeks of production, my conclusion is: **the more advanced the memory system, the deeper the pitfalls**. Here's a timeline of everything that actually broke. ## Pitfall 1: Startup Failure Trilogy First time launching OpenClaw Gateway on a VPS, three consecutive errors, each with a different root cause. **`gateway token missing`** — easiest to overlook. OpenClaw's Gateway mode requires a separate `OPENCLAW_GATEWAY_TOKEN` environment variable, not the Feishu App Secret. Miss one `Environment=` line in the systemd unit file and you get 401. **`No credentials for provider`** — the `keyRef` in `auth-profiles.json` must match the environment variable name exactly. One character off and you get this error, with no indication of which keyRef failed to match. You have to eyeball it. **`400 InvalidParameter` / model not found** — Omniroute model names must follow the `/` format. Writing `gpt-5.5` returns 400; you must write `codex/gpt-5.5`. Add OpenClaw's own provider prefix, and it becomes `omniroute/codex/gpt-5.5`. Three layers of prefix nesting — miss one and nothing works. ```bash # These three steps save the most time when debugging systemctl cat openclaw-gateway.service | grep Environment # check env vars python3 -c "import json; json.load(open('/home/openclaw/.openclaw/openclaw.json'))" # JSON syntax journalctl -u openclaw-gateway.service -n 50 --no-pager | grep -iE "error|unauth" # check errors ``` ## Pitfall 2: Where Are the Logs OpenClaw runtime logs go **entirely to journald**, not to files. `/tmp/openclaw/openclaw-*.log` only has a few lines from startup; all runtime errors live in journald. This cost me half an hour — tailing file logs and seeing nothing, before realizing: ```bash # This is the correct way to view logs journalctl -u openclaw-gateway.service -f ``` ## Pitfall 3 (The Worst): Compaction Silently Swallows Replies This was the most severe incident in three weeks. Before diving into the timeline, here's a flowchart of compaction's position in the Agent Loop and its failure path: ```mermaid flowchart TB MSG["👤 User message arrives"] LOOP["🔄 Agent Loop processing
Assemble context → Reason → Generate reply"] CHECK{"📏 Context ≤ model limit?"} DELIVER["✅ Reply delivered to Feishu"] COMPACT["🧹 Auto-Compaction triggered
Compress old messages into summary"] OVERFLOW{"⚠️ Compaction prompt also over limit?"} EMPTY["💀 Model outputs empty string
Conversation is empty"] LOST["❌ Reply silently discarded
User perception: bot unresponsive"] SAFEGUARD["🛡️ safeguard mode
reserveTokens + fallback
Compaction complete → normal delivery"] MSG --> LOOP --> CHECK CHECK -->|"✅ Under limit"| DELIVER CHECK -->|"❌ Overflow 209K > 200K"| COMPACT --> OVERFLOW OVERFLOW -->|"❌ No protection"| EMPTY --> LOST OVERFLOW -->|"🛡️ Fixed"| SAFEGUARD --> DELIVER style EMPTY fill:#ff6b6b,color:#fff style LOST fill:#ff6b6b,color:#fff style SAFEGUARD fill:#51cf66,color:#fff style DELIVER fill:#51cf66,color:#fff ``` The Mermaid diagram preserves the exact failure branch. The systems graphic below changes the viewpoint by separating **system state** from **user state**. The dangerous case is a job that appears complete and exception-free while the user still receives no final answer. ![A silent Compaction failure where the system succeeds but the user receives no reply](https://zhuoqidev.com/_astro/silent-compaction-failure-bilingual-v1-4k.BgI-w9P2_xnhrU.webp "Silent Compaction failure: creating a summary is not the same as achieving the user's goal. A correct loop preserves the pending question before compression, restores the task afterward, and verifies that the reply was actually delivered.") A normal user message — the Agent generated a complete reply — but the **user received nothing**. The bot appeared dead silent. ### Timeline - `01:43:40` User in Feishu news group: "失败请求占用到我们账户的请求资源的，麻烦方便的时候检修下程序中错误的请求参数" - `01:43:57` Agent generated a full text reply (session trajectory shows `message` event with assistant response) - `01:44:00` Auto-compaction triggered: session context had reached 209K tokens, exceeding the primary model's 200K limit - `01:44:00` **Reply vanished**: compaction output an empty summary `"Conversation is empty"`, session ended, reply never delivered to Feishu - User side: complete perception of "bot not responding" ### Root Cause Compaction is an **implicit middleware layer**. Normally it compresses context; but when the context itself already exceeds the model limit (209K > 200K), and the compaction prompt can't hold the full context either, **the model outputs an empty string**. OpenClaw has no protection against this by default — no error, no degradation, no notification, just silently discarding the already-generated reply. This is a textbook case of `middleware-semantic-leak`: the middleware layer, under extreme conditions, flipped from "compressing context" to "discarding replies" — a complete semantic inversion with zero signal. Looking at the source code reveals the full silent-failure chain. OpenClaw's `compaction.ts` defines an innocuous-looking constant: ```typescript // src/agents/compaction.ts const DEFAULT_SUMMARY_FALLBACK = "No prior history."; ``` When `summarizeChunks()` receives an empty message list, it returns this string directly — no error thrown, no warning, no fallback path. And here's the actual compaction prompt sent to the model — the system prompt has zero defensive instructions: ```text // packages/agent-core/src/harness/compaction/compaction.ts SUMMARIZATION_SYSTEM_PROMPT: "You are a context summarization assistant. Your task is to read a conversation between a user and an AI coding assistant, then produce a structured summary following the exact format specified. Do NOT continue the conversation. Do NOT respond to any questions in the conversation. ONLY output the structured summary." ``` The user prompt template: ```text // packages/agent-core/src/harness/compaction/compaction.ts SUMMARIZATION_PROMPT: " [The full conversation history is inserted here — at 209K tokens, that's tens of thousands of words of raw messages] The messages above are a conversation to summarize. Create a structured context checkpoint summary that another LLM will use to continue the work." ``` The critical design flaw lies in the `` tags: **the instructions ("summarize the conversation") come AFTER the conversation history**. When the history exceeds the model's context window, the model doesn't see "a bunch of conversation" + "please summarize". It sees a truncated `` opening, followed by nothing — no closing `` tag, no summarization instructions. The system prompt only says "you're a summarization assistant, output only structured summaries" — it never tells the model what to do if the conversation content is truncated. So the model faces: a truncated XML fragment + a system prompt that only says "output a summary". For most models' default behavior, this means an extremely short empty output — exactly the "Conversation is empty" seen in the trajectory. Even more fatal is `summarizeWithFallback()`'s final fallback path: when full summarization and partial summarization both fail (chunk overflow, model 5xx, token overflow), it falls back to: ```typescript // src/agents/compaction.ts — summarizeWithFallback final fallback return ( `Context contained ${messages.length} messages ` + `(${oversizedNotes.length} oversized). ` + `Summary unavailable due to size limits.` ); ``` That line is the source of the "Conversation is empty" you see in the session trajectory — **the compaction pipeline assumes "after summary failure, the original context is still usable"**, but the 209K > 200K edge case precisely breaks this assumption: the original context won't fit in the prompt, the summary model returns empty, and the only output is a natural language notification saying "summary unavailable" — which OpenClaw treats as a normal compaction result and continues execution, silently discarding the already-generated reply. Two compounding constants are worth noting: ```typescript // src/agents/compaction.ts const SAFETY_MARGIN = 1.2; // 20% buffer for estimateTokens inaccuracy const SUMMARIZATION_OVERHEAD_TOKENS = 4096; // prompt/system prompt overhead ``` `SAFETY_MARGIN` is a 20% buffer for token estimation — `estimateTokens()` uses `chars/4` heuristic, which systematically underestimates multi-byte characters, special tokens, and code tokens. This means "looks like 10K headroom" can actually already be hitting the wall. `SUMMARIZATION_OVERHEAD_TOKENS` is the fixed overhead for compaction prompt + system prompt + serialization wrappers — but it does NOT include the previous summary's size. **When the session has been refined through multiple rounds and the previous summary alone is tens of KB, this 4096 fixed reserve is insufficient**. And the confirmation that this is default behavior, not a misconfiguration, lies in the type definition: ```typescript // src/config/types.agent-defaults.ts export type AgentCompactionConfig = { mode?: AgentCompactionMode; // optional field — if unset, no safeguard reserveTokens?: number; keepRecentTokens?: number; // ... }; ``` `mode` is optional. Without explicitly configuring `"safeguard"`, neither `reserveTokens` nor `maxHistoryShare` protection is active. Production default behavior: compaction runs naked. ### Fix Two steps, both required: ```json // openclaw.json "compaction": { "mode": "safeguard", "reserveTokens": 20000, "keepRecentTokens": 16000 }, "model": { "fallbacks": ["deepseek/deepseek-v4-flash"] } ``` The `safeguard` mode reserves a token budget for the post-compaction summary, preventing it from filling the context window. And the fallback to `deepseek-v4-flash` (1M context window) ensures the compaction task itself never overflows, even if the primary model's window is full. ### General Lesson This isn't OpenClaw-specific. Any agent system with automatic context compression faces the same risk: **compaction's failure mode is silent**. If your agent suddenly stops replying, the third thing to check is compaction — look in the session trajectory for compaction events with empty content. ## Feishu Message Non-Response: The Five-Layer Debugging Method After this incident, I developed a debugging chain from outside-in, ordered by probability. Next time the bot doesn't respond, follow this: The diagram pairs each layer with its **observable evidence**. Debugging should not begin by blaming the model; prove how far the message traveled through Process → Connection → Routing → Execution → Delivery. ![A five-layer debugging stack for an unresponsive OpenClaw message](https://zhuoqidev.com/_astro/five-layer-debugging-bilingual-v1-4k.BtvxvRQB_Z1kNvGk.webp "Five-layer debugging: require one verifiable signal at every boundary—process health, connection event, inbound log, run trace, and outbound message ID. The first missing signal usually identifies the failure boundary.") | Layer | Checkpoint | How to check | |---|---|---| | L1 Feishu App Layer | Did the bot receive the message | `lark-cli im +chat-list --as bot` confirm group in allowlist | | L2 Event Ingestion Layer | Did the message reach Gateway | Check dedup records for recent entries | | **L3 Session Layer** | Did the Agent process the message | **Check the last event in session trajectory** — this is the most critical step | | L4 Model Invocation Layer | Did the model respond normally | journalctl grep 429/401/500/timeout | | L5 Delivery Layer | Did Feishu send succeed | Send a test message directly via CLI to verify permissions | When facing non-response, follow this decision tree — check the last 10 lines of the trajectory first. Eighty percent of issues are resolved at step three: ```mermaid flowchart TD PROBLEM["❓ Feishu group bot not responding"] L1["L1: Group permissions
lark-cli im +chat-list --as bot"] L1_OK{"Group in allowlist?
Bot in group?"} L1_FIX["Fix permissions / add group to allowlist"] L2["L2: Message arrival
Check dedup records"] L2_OK{"Dedup has recent entries?"} L2_FIX["Check Feishu event subscription URL
OPENCLAW_GATEWAY_TOKEN"] L3["🔑 L3: Session Trajectory
cat session/*.jsonl last 10 events"] L3_MSG{"What's the last event?"} L4["L4: Model invocation
journalctl grep 429/500/timeout"] L5["L5: Delivery
lark-cli im +send-message test"] L3_CTX["💀 compaction empty
→ context overflow
→ this incident, add safeguard immediately"] L3_MESSAGE["Reply was generated
→ issue in delivery layer
→ go to L5"] L3_ERROR["Has error event
→ go to L4 check model"] PROBLEM --> L1 L1 --> L1_OK L1_OK -->|"✓ OK"| L2 L1_OK -->|"✗ Issue"| L1_FIX L2 --> L2_OK L2_OK -->|"✓ Has records"| L3 L2_OK -->|"✗ No records"| L2_FIX L3 --> L3_MSG L3_MSG -->|"compaction empty"| L3_CTX L3_MSG -->|"assistant text"| L3_MESSAGE L3_MSG -->|"error"| L3_ERROR L3_ERROR --> L4 L3_MESSAGE --> L5 style L3 fill:#ffd43b,color:#333 style L3_CTX fill:#ff6b6b,color:#fff style L3_MESSAGE fill:#51cf66,color:#fff style L3_ERROR fill:#ff922b,color:#fff ``` ## Outer Loop: Why the Framework Matters More Than the Model [SWE-bench Pro](https://www.swebench.com/), the most credible enterprise-grade coding benchmark (maintained by Princeton, independently audited), provides compelling evidence: **the exact same model, Claude Opus 4.5, scores anywhere from 45.9% to 55.4% depending on the harness** — SEAL standardized scaffold 45.9%, Cursor 50.2%, Auggie 51.8%, Claude Code 55.4%. Same model, different harness: a 9.5-percentage-point gap. Anthropic's November 2025 engineering blog *Effective harnesses for long-running agents* confirms the same pattern: splitting "doing" from "evaluating" across separate agents with hard pass/fail gates allowed a three-agent harness to run for 6 hours and produce a complete working application, where a single agent crashed after 20 minutes. OpenClaw doesn't appear on any mainstream benchmark leaderboard — not because of model capability, but because its harness quality isn't yet sufficient for benchmark-level reliability. This isn't random. A converging industry insight in 2025: **"Structure around the model matters more than cleverness inside the model."** The model only does generation, and generation is cheap — what truly determines system quality are three things in the Outer Loop: | Outer Loop Component | What it does | What happens when it fails | |---|---|---| | **Verification** | Compiler, test suite, linter check output | AI-generated wrong code goes undetected | | **Context Engineering** | Compaction, pruning, memory flush | This article's incident — silent reply loss | | **Tool Definitions (ACI)** | Tool schema, parameter constraints, guardrails | Wrong parameters, wrong file paths, silent retries | These three layers are all **outside the model** — they're systems you deploy and maintain. The root cause of the compaction incident isn't that the model wasn't smart enough; it's that nobody told the harness "when compaction fails, don't discard the reply — raise an error." This also explains why Anthropic observed that "the most successful agent implementations rarely use complex frameworks" — a framework is someone else's Outer Loop, and you can't control its failure modes. OpenClaw's multi-backend support looks like an advantage, but each model has a different context window, reasoning profile, and API behavior — **every additional model adds another combination of compaction boundary conditions**, making the Outer Loop test matrix grow exponentially. This leads to a more fundamental structural question — the divergent paths in memory system design. ## OpenClaw vs Claude Code: Divergent Memory System Philosophy OpenClaw's most distinctive memory mechanism is **background Dreaming consolidation** — not just "write it down", but a three-stage automated pipeline: ```mermaid flowchart LR subgraph Short["📝 Short-term Memory"] SESSION["Session conversations"] RECALL["recall traces"] end subgraph LIGHT["💡 Light Sleep — Zero LLM"] SORT["Jaccard semantic dedup + sorting"] end subgraph REM["🌙 REM Sleep — Pattern Discovery"] REFLECT["Pattern recognition + reflection
Does NOT write MEMORY.md"] end subgraph DEEP["🧠 Deep Sleep — Three-condition promotion"] RANK["score ≥ 0.80"] MERGE["merge ≥ 3"] RECALL_THRESH["recall days ≥ 3"] end subgraph Long["📦 Long-term Memory"] MEM["MEMORY.md"] end Short --> LIGHT --> REM --> DEEP DEEP -->|"✓ All passed"| MEM DEEP -->|"✗ Not met"| LIGHT style DEEP fill:#b197fc,color:#fff style MEM fill:#51cf66,color:#fff ``` After running both systems for a while, here's a side-by-side comparison: | Dimension | OpenClaw | Claude Code | |---|---|---| | Retrieval mechanism | Vector+BM25 hybrid (7:3) | LLM semantic judgment (Sonnet) | | Weight evolution | **PPO adaptive** | Manual fixed classification | | Consolidation | Triple sleep | Three-gate four-stage AutoDream | | Model lock-in | Multi-model | Claude only | | Memory classification | By time (long-term distill / short-term daily) | By type (user/feedback/project/reference) | | Production maturity | Early stage | Validated by millions of agents | OpenClaw's memory architecture is more "academically correct" — PPO adaptive weights, triple sleep, hybrid retrieval, each layer close to the frontier of memory systems research. Claude Code looks "cruder": memory is four categories of Markdown files (user/feedback/project/reference), retrieval relies on Sonnet's semantic judgment, no PPO, no BM25. But here's a genuinely counterintuitive structural fact: **file-based memory > vector memory**. Not a quantitative "a bit better" — a qualitative difference. The industry, from Claude Code to Codex CLI to Cursor, has converged on Markdown files as the primary memory medium, with vectors as a secondary index. Why? Because in programming contexts, **determinism > probabilistic retrieval** — you can't let PPO weights decide "whether to remember that an API key leaked". The bigger problem: **the sophistication of the memory system and the stability of the Outer Loop are multiplicative, not additive**. No matter how precise your memory retrieval algorithm is, if the compaction middleware quietly drops replies, the user doesn't perceive "memory isn't good enough" — they perceive "the bot is dead". OpenClaw invested 80% of its innovation budget on the memory side (PPO, Dreaming, hybrid retrieval), but underinvested in Outer Loop defense (compaction safeguard, failure signals, degradation paths). Claude Code is the reverse — conservative on memory, but its harness has been validated by millions of agent sessions. This is the trap to watch for in tool selection: **benchmarks measure the ceiling of model+memory capability, but production systems live at the floor of the Outer Loop**. ## Compaction Design: Structural Differences Between the Prompts We've seen from OpenClaw's source code why its compaction prompt silently fails on overflow — `` tags come first, instructions come after, so the model never sees the instructions when context overflows. What does a better design look like? In March 2026, Anthropic's npm `.map` file accidentally leaked Claude Code's complete system prompt. [Piebald-AI/claude-code-system-prompts](https://github.com/Piebald-AI/claude-code-system-prompts) extracted and organized all prompts from the compiled JS. Comparing the two compaction prompts, the gap isn't in the model — both use similar LLMs for summarization — but in **prompt-level structural defenses**: **Summary format: 9 sections vs 5.** Claude Code requires 9 structured sections: 1. Primary Request and Intent — the user's core ask 2. Key Technical Concepts — technologies and frameworks 3. Files and Code Sections — file paths + full code snippets + why each matters 4. Errors and fixes — all errors + how fixed + user feedback on each 5. Problem Solving — solved and ongoing troubleshooting 6. **All user messages** — every user message (non-tool-result) 7. Pending Tasks — outstanding work 8. Current Work — precisely what's being worked on right now (with filenames + code) 9. Optional Next Step — the next action + **verbatim quotes from the conversation** OpenClaw only has 5 (Decisions / TODOs / Constraints / Asks / Identifiers). The missing ones are exactly what's most fatal in a compaction incident: error fix history, complete preservation of user messages, precise description of current work. **Security constraints preserved verbatim.** Claude Code's prompt explicitly states: *"Note any security-relevant instructions or constraints the user stated... These MUST be preserved verbatim in the summary so they continue to apply after compaction."* This means directives like "don't read .env files", "don't commit keys", "must use HTTPS not HTTP" remain in effect after compaction. OpenClaw only does identifier preservation (identifier policy), not security instruction preservation. **User feedback doesn't get compressed away.** Section 6 requires listing ALL user messages, and Section 4 requires recording user feedback on each error. Together, the effect is: the user said "don't do it that way, do it this way" — this survives compaction. Compare to this article's incident: a user message triggers compaction → the generated reply is silently discarded → the user only sees bot not responding. If the compaction summary forced retention of "what the user said", at minimum you could immediately see the trigger source during debugging. **Next step must quote verbatim.** Section 9 requires: *"Include direct quotes from the most recent conversation showing exactly what task you were working on and where you left off. This should be verbatim to ensure there's no drift in task interpretation."* This design prevents semantic drift where "the summary model rephrases the task in its own words" — a rephrased task can be understood completely differently by the next generation agent. **Analysis before output.** Claude Code's prompt requires first analyzing each message inside `` tags, then outputting ``. This isn't decorative — it embeds reasoning into quality control: if the analysis doesn't mention a particular error fix, the summary won't have it. OpenClaw's prompt has no such built-in completeness check. **Circuit breaker.** Stops after 3 consecutive auto-compact failures. OpenClaw has no such mechanism — theoretically it could loop indefinitely: compact → fail → compact → fail. **Partial compaction.** Claude Code supports compacting only the first half while keeping the latter half verbatim. A separate prompt tells the model: "this summary will be placed at the start of a continuing session; newer messages will follow" — the model knows its output is a prefix, and won't pretend to have full context. **Post-compaction file reference tracking.** After compaction, Claude Code injects a system reminder: "Note: X file was read before the last conversation was summarized, but the contents are too large to include. Use the Read tool if you need to access it." This tells the model what it knew before and might need to re-read. Here's the comparison in a table: | Design Dimension | Claude Code | OpenClaw | |---|---|---| | Summary format | 9 sections, incl. error fixes + user messages | 5 sections | | Security instruction retention | Verbatim preservation | Identifier preservation only | | User feedback protection | ALL user messages + verbatim quotes | None | | Pre-output analysis | `` → `` two-step | Direct output | | Circuit breaker | Stop after 3 consecutive failures | None | | Prompt architecture | Instructions first, conversation after | Conversation first, instructions after | | Failure signaling | Explicit error message | Silent fallback | | Partial compaction | Separate prompt, "prefix mindset" | Not supported | | File reference tracking | Post-compaction reminder to re-read | Post-compaction reads AGENTS.md | There's a structural observation worth pausing on: **Claude Code's 9-section format isn't "smarter" — it's "harder to lose things"**. Every design decision — verbatim security constraints, listing all user messages, quoting next steps — defends against the same risk: compaction is a natural information loss node, and the prompt's sole job is to minimize that loss. OpenClaw invested noticeably less prompt engineering into compaction — not lacking model capability, but lacking defensive design around "what absolutely must not be lost." This brings us back to the article's core thesis: **Outer Loop quality doesn't depend on how strong your model is — it depends on whether your prompt leaks under extreme conditions**. The `SAFETY_MARGIN = 1.2`, `SUMMARIZATION_OVERHEAD_TOKENS = 4096`, and ``-before-instructions ordering in this incident — none of these are model problems. They're prompt problems that didn't account for overflow scenarios. Claude Code's prompt, even if it overflows, would at least leave error fix records, user messages, and verbatim quotes — the context needed for debugging wouldn't entirely evaporate. ## Conclusion: Three Judgments Deeper Than the Pitfalls **1. Outer Loop is the new moat, not the model.** After 2025, model capabilities are rapidly converging — Gemini 2.5 Pro has 1M context, DeepSeek-V4 has 1M, GPT-5.5 has 200K. The gaps between models are narrowing, but the gaps in harness quality (compaction safeguard, verification loops, degradation paths, failure signals) are widening. The compaction incident in this article is evidence — it's not a "model not good enough" problem; it's the harness missing one line of config. Claude Code's ARR reaching ~$2B by January 2026 isn't because Claude is smarter than other models; it's because its Outer Loop has been validated by millions of agent sessions. **2. Determinism beats probability — in memory, in routing, in failure handling.** OpenClaw's PPO adaptive weights and hybrid retrieval are correct on paper, but when compaction silently fails, users don't care about your retrieval recall improvement. At system boundaries (compaction, fallback, error handling), **rule-based deterministic rules are safer than RL-based probabilistic rules**. Claude Code chose "crude" classification not because it can't do PPO, but because manual rules are more predictable when it comes to "not losing things." **3. The 2025-2026 industry trend is BYOK and harness open-sourcing.** A massive migration is happening from Cursor to Claude Code CLI, Aider, OpenCode — not because these tools have more features, but because BYOK (Bring Your Own Key) lets you **control the Outer Loop**. SaaS tools' compaction strategy, caching strategy, retry strategy are all black boxes — when something breaks, you can only wait for the vendor to fix it. With OpenClaw, every line of `openclaw.json` is visible, editable, and version-controllable. This is OpenClaw's real long-term value — not how advanced the memory system is, but that **the harness is fully transparent**. --- title: "Why We Moved from Celery to Temporal for Production Agent Pipelines" description: "Agent pipelines are not ordinary async tasks — they have state, they get stuck, they need replay debugging, and one failure must not sink the entire batch. We hit every one of these walls with Celery before understanding that Temporal solves a fundamentally different problem." url: "https://zhuoqidev.com/en/posts/why-temporal-not-celery/" language: "en" author: "Liu ZhuoQi" date: "2026-05-16" updated: "2026-07-30" tags: ["Agent Engineering","Temporal","Celery","Workflow Engine","Production","Backend","Python"] translation: "https://zhuoqidev.com/posts/why-temporal-not-celery/index.md" --- # Why We Moved from Celery to Temporal for Production Agent Pipelines > **TL;DR** In April 2026, we migrated seo-project's task queue from Celery to Temporal. We dropped exactly one dependency (celery), wrote 11 new files (src/infrastructure/temporal/), and renamed our containers from api/worker/beat to api/temporal_worker_blue/green with blue-green deployment. The most common question afterward: **why not just keep using Celery? If it's already running, what's the point?** This article is the answer. It doesn't come from documentation comparisons. It comes from production bugs we hit running Agent pipelines at scale. The diagram first separates the **problem layers**. Celery hands work to workers; Temporal also records event history so a multi-stage workflow can recover from the failure point. The migration was not about replacing a queue with a faster queue. It was about gaining a replayable state machine. ![Execution semantics of a Celery task queue versus a Temporal durable workflow](https://zhuoqidev.com/_astro/celery-vs-temporal-bilingual-v1-4k.PQ5-KLou_1Ciww1.webp "Celery and Temporal: the former excels at distributing independent async tasks; the latter adds event history, checkpoints, and replay for long-running multi-stage workflows. When stage C fails, the decisive question is full retry versus resume from C.") --- ## When Celery Works — and When It Doesn't Let's be clear: **Celery is a good tool**. For standard async workloads — sending an email, resizing an image, pushing a notification — it handles the job perfectly, and has for over a decade. Our workload, however, looks like this: ``` One Run contains N longtail articles Each longtail goes through 4 Agent stages: A → B → C → D Each stage calls one or more AI APIs Per-article latency: 60–180 seconds Every intermediate result must be persisted Any failure must answer: "where did it stop, why, and can I retry only this one step?" ``` This is a **stateful, long-running, multi-stage business process**. The line between a task queue and a workflow engine runs right through here. --- ## Crack #1: Task State Is Binary — Success or Failure Celery's task model: enqueue → worker picks up → execute → success or failure. The entire middle is a black box. When Agent D times out while generating SEO metadata, Celery tells you: "task failed, state=FAILURE." What you actually need to know: - Did Agent A (keyword analysis) complete? What was its output? - Did Agent B (outline generation) finish? - Did Agent C (article generation) persist its HTML to the database? - Which model was Agent D calling when it timed out? Should retry start from D, or from C? Celery's answer to all of these: "track it yourself." You can write state to Redis or a database, but you're essentially **hand-rolling a state machine on top of a task queue** — which is exactly what a Temporal workflow provides natively. In Temporal, your workflow code *is* the business logic. Every step's position, return value, and exception is automatically persisted. If the worker restarts, the workflow resumes from the last completed step with zero checkpoint code. ```python # Celery: you manage state yourself @app.task(bind=True, max_retries=3) def run_pipeline(self, longtail_id): try: result_a = agent_a(longtail_id) # succeeds save_checkpoint("step_a_done", result_a) result_b = agent_b(result_a) # fails here # On retry — where do you resume? You figure it out. except Exception: self.retry() # Temporal: the workflow *is* the state machine @workflow.defn class PipelineWorkflow: @workflow.run async def run(self, longtail_id: str) -> dict: result_a = await workflow.execute_activity(agent_a, longtail_id) result_b = await workflow.execute_activity(agent_b, result_a) result_c = await workflow.execute_activity(agent_c, result_b) result_d = await workflow.execute_activity(agent_d, result_c) return result_d ``` No checkpoint management. If `agent_c` times out, retry picks up from `agent_c` — the return values of `agent_a` and `agent_b` are already persisted in the workflow history. --- ## Crack #2: Batch Rollback — "One Timeout Sinks All Eight" This was a real production incident. Our batch pipeline ran N longtails inside a single task loop, each going through A→B→C→D. The database commit sat *outside* the loop: ```python # Anti-pattern: single transaction for the entire batch async with unit_of_work(autocommit=False) as uow: for longtail in batch: await run_pipeline(longtail) await uow.commit() # commit all at once ``` Then it happened: longtail #3's Agent C timed out → the SQLAlchemy session went `invalid` → every subsequent `get`/`save`/`commit` for #4–#8 threw `PendingRollbackError` → all 8 articles vanished. The fix was shrinking the transaction boundary to a single longtail: each iteration commits independently, and failed iterations *also* commit their failure status. With Temporal, this pattern becomes the natural default — **each longtail is an independent workflow execution with native idempotency and replayability**. In the Celery model, task failure = entire task rolled back = you control retry granularity by hand. In Temporal, one workflow's failure doesn't touch other workflows. Operations sees "1 failed / 7 succeeded" instead of "8 vanished." --- ## Crack #3: Long-Running Tasks vs. Worker Restarts Our Agent pipeline runs 60–180 seconds per article. Celery workers restarting (deployments, OOM kills, node recycling) will kill in-flight tasks and re-queue them from scratch. For sub-5-second tasks, this barely matters. For a task that's been running 120 seconds, has already burned a hundred thousand tokens, and has one step left — **restarting from zero means all that token cost goes up in smoke.** Temporal's durable execution model works differently: even if the worker dies, workflow state lives in the Temporal Server. A new worker picks up the workflow history and resumes from the last completed activity. This directly impacts AI pipeline cost control. --- ## Crack #4: Debugging Without Visibility When an Agent pipeline fails in production, here's the Celery debugging path: 1. Find the task ID 2. Open Flower (if you installed it) to check task state 3. grep through application logs for the trace_id 4. Manually reconstruct: "what did A produce → what did B produce → where did C get stuck" 5. If there were retries, the picture gets muddier Here's the Temporal debugging path: 1. Open the Temporal Web UI 2. Click into the workflow execution 3. **Every step's input, output, duration, and exception is visualized on a single timeline** 4. You see exactly: "Activity `generate_article` in Agent C timed out at 47 seconds, retried twice — expand to view the prompt and returned HTML for each attempt" This isn't "Temporal has a nice UI." It's an **order-of-magnitude difference in debugging efficiency**. For multi-Agent pipelines — long call chains, abundant state, critical intermediate artifacts — visual workflow history is a requirement, not a luxury. --- ## Crack #5: Reliable Scheduling Celery Beat is a standalone process started with `celery beat`. Known failure modes: - Beat goes down → scheduled tasks stop, no failover - Beat restarts → may double-trigger tasks at the same timestamp - Scheduling window missed → no automatic backfill, manual intervention required - Schedule changes → restart required Our Agent project has a real business requirement: generate a batch of articles per site every day. If the scheduler goes down or misses a window, we need to **automatically detect missing dates and backfill**. Celery's solution is "write a script to compare date gaps, manually dispatch." Temporal Schedule provides this natively: - **Overlap Policy**: what happens when the next trigger fires before the previous workflow finishes - **Catch-up Window**: auto-trigger missed executions within a configurable lookback - **Pause/Resume**: pause a schedule without editing cron expressions For content production — where missing a day means you have to catch up — Schedule's backfill semantics are a hard requirement. --- ## Decision Matrix | Dimension | Celery | Temporal | Winner for Agent Pipelines | |-----------|--------|----------|---------------------------| | **Task model** | Stateless task | Stateful workflow | Temporal (Agent pipelines are inherently stateful) | | **State persistence** | DIY (Redis/DB) | Automatic workflow history | Temporal (zero extra code) | | **Retry semantics** | Task-level, restart from scratch | Activity-level, resume from breakpoint | Temporal (long AI calls have real token cost) | | **Failure visibility** | Log grep + Flower | Web UI timeline + per-step I/O | Temporal (order-of-magnitude debugging efficiency) | | **Scheduling** | Beat process, no failover | Schedule, built-in catch-up | Temporal (content backfill is a hard requirement) | | **Batch failure radius** | Depends on your commit strategy | Each workflow naturally isolated | Temporal (independent idempotent units) | | **Worker restart impact** | In-flight tasks lost, restart from zero | Workflow resumes from breakpoint | Temporal (120s AI calls aren't wasted) | | **Operational complexity** | Worker + Beat + Flower + Broker (Redis/RabbitMQ) | Worker + Server (Web UI included) | Comparable (Temporal Server adds deployment cost but removes Flower + Beat overhead) | | **Learning curve** | Low (Python-native ecosystem) | Medium (determinism constraints require understanding) | Celery wins for onboarding speed | | **Best for** | Short, stateless, low-cost-of-failure tasks | Long-running, stateful, auditable, intermediate-artifact-heavy processes | Depends on workload | --- ## When You *Don't* Need Temporal Celery remains the better choice if: - Task latency is under 10 seconds; restarting from scratch is negligible - You don't need visibility into intermediate step I/O (e.g., sending email) - You don't need step-level retry (whole task succeeds or fails) - Your team has no bandwidth for learning Temporal's determinism constraints - Your existing Celery system runs reliably; migration cost > migration benefit Our decision wasn't "Temporal is better than Celery." It was: **"the characteristics of Agent pipelines happen to land squarely on Celery's weak spots and Temporal's core strengths."** --- ## Architecture Patterns Post-Migration The migration crystallized several architectural decisions: **1. One longtail = one workflow = one transaction boundary** Batch processing becomes: "dispatch N workflows externally, each workflow manages its own transaction internally." This is the natural marriage of per-iteration transaction boundaries (from `concepts/backend`) and Temporal's execution model. **2. Dual UoW semantics** FastAPI routes use `autocommit=True` (short request-scoped transactions). Temporal workers use `autocommit=False` (workflow-scoped, explicit commit/rollback control). Same UoW abstraction, different lifecycle semantics. **3. Blue-green deployment** Containers went from a single `worker` instance to `temporal_worker_blue` + `temporal_worker_green`. Deploy by routing new workflows to the fresh color; drain the old color once in-flight workflows complete. Temporal Task Queue routing makes this zero-downtime. **4. Replay is free** Every workflow execution's history lives in the Temporal Server. When something breaks, you can replay the entire workflow locally — same inputs, same code path, database calls mocked out — to pinpoint exactly which activity, which invocation, which return value caused the error. In the Celery era, this was hours of manual labor. --- ## Summary Framing Temporal as "a more powerful Celery" misunderstands both. They solve problems at **different layers**: - Celery answers: *"run this code on another machine"* - Temporal answers: *"guarantee this multi-step business process reaches completion, and make every step auditable"* Agent pipelines happen to be multi-step, long-running, intermediate-artifact-rich business processes with high failure costs. Using a task queue to emulate a workflow engine isn't using the wrong tool — it's using the right tool's predecessor, then hand-building the features the successor already ships. --- *Based on production experience from [seo-project](https://github.com/zhuoqidev/ai-seo-pipeline). Migration records in LLM Wiki → `log.md` (2026-04-10) and `concepts/backend.md`.* --- title: "Where Do ChatGPT Business Promo Codes Come From? An OSINT Correction" description: "A source-by-source recheck of OpenAI, partner, Stripe, and linux.do evidence: public links, screened email codes, and partner-funded rebates still coexist." url: "https://zhuoqidev.com/en/posts/chatgpt-business-promo-investigation/" language: "en" author: "Liu ZhuoQi" date: "2026-05-14" updated: "2026-07-30" tags: ["ChatGPT","OpenAI","Stripe","Promo Codes","OSINT","Fact-checking"] translation: "https://zhuoqidev.com/posts/chatgpt-business-promo-investigation/index.md" --- # Where Do ChatGPT Business Promo Codes Come From? An OSINT Correction > **TL;DR** In May 2026, a batch of company-branded ChatGPT Business promo codes spread through linux.do and other communities. The original article tried to identify their source but presented several plausible inferences as facts. > [!WARNING] > **Correction — July 29, 2026** > > The first version of this article combined three observations—a company name inside a promo code, a Stripe-hosted checkout, and community reposts—into a story that was too certain: Stripe's partner network issued the codes and employees of those companies leaked them. The available evidence does not establish that chain. > > The corrected finding is narrower: **OpenAI partner or channel campaigns are the best-supported upstream source. Stripe is payment and promotion-code infrastructure, not a proven campaign owner. Public links, eligibility-screened codes delivered by email, and partner-funded rebates coexist. There is also no evidence that OpenAI has globally retired company-level `/p/...` links.** This revision keeps only claims supported by primary pages or reproducible timestamps, and separates: - official facts; - statements published by partners; - community observations; - unverified inference. The diagram below defines the article's **evidentiary boundary**. It separates official facts, infrastructure capabilities, channel claims, and community observations. Its most important job is to prevent a leap from “Stripe can execute a discount” to “Stripe designed and distributed the campaign.” ![A four-level evidence ladder for tracing ChatGPT Business promotions](https://zhuoqidev.com/_astro/promotion-evidence-ladder-bilingual-v2-4k.9xRWR5hF_Z1HxlhG.webp "Evidence ladder: conclusions become stronger as they approach original pages, reproducible timestamps, archives, and independent confirmation. Payment infrastructure shows how a promotion can execute; it does not by itself identify the campaign owner or funder.") ## The short answer 1. **A real partner channel exists.** OpenAI publicly operates the [OpenAI Partner Network](https://openai.com/business/partners/) for co-selling, building, and delivering solutions. CodeStone announced that it joined the OpenAI SMB Channel Partner Programme, while First Focus announced an OpenAI Services Partner Agreement. 2. **There is no evidence that Stripe was the source.** Stripe supports subscription promotion codes, customer eligibility, first-order restrictions, expiration dates, and redemption limits. That explains how a discount can be enforced at checkout, not who designed, funded, or distributed the campaign. 3. **Public does not automatically mean leaked.** OpenAI itself hosts a public [Dropbox × ChatGPT Business offer](https://openai.com/business/partners/dropbox/). Think Technologies has also publicly promoted `chatgpt.com/p/THINK26` in company LinkedIn posts. 4. **Distribution has diversified, not uniformly switched.** AI Build now uses a work-email form, eligibility review, and email delivery. Public `/p/...` links still exist at the same time. 5. **linux.do recorded several different changes.** The end of an older first-month trial in late April, the failure of console-generated “Stripe long links” on June 4, and the status of partner promo links are separate events. A reader's note that the current UK AI Build flow sends a code after review is therefore correct **for that specific channel**. The broader claim that OpenAI canceled all company-level shared links and converted every offer to a personal code is not supported. AI Build's page itself does not say that each code is bound to one person or that OpenAI globally retired public links. ## What the original article got wrong | Original claim | Finding after review | Correction | |---|---|---| | “The real source is Stripe's enterprise partner network” | No primary evidence | “OpenAI partner/channel campaigns” is better supported; Stripe is only proven to be a technical layer | | “Employees leaked internal codes” | Unverifiable, with public-marketing counterexamples | We can document community redistribution, not the identity or authorization of the first sharer | | “Codes die by quota, never by date” | Stripe supports expiry, redemption limits, customer restrictions, and first-order restrictions | An “unable to use” message cannot identify the cause | | “Every code represents OpenAI partner subsidy” | Directly contradicted by First Focus's current terms | Some benefits are funded by the partner and must be assessed campaign by campaign | | “A matching region and code are enough” | Oversimplified | Account status, billing country, payment method, tax, and campaign eligibility may all matter | | “Act immediately” | Poor advice | Verify audience, renewal price, seat count, and final checkout total first | ## How far can the source be traced? ### 1. Confirmed: OpenAI has a formal partner channel The [OpenAI Partner Network](https://openai.com/business/partners/) describes co-selling, technical enablement, delivery, and a partner portal. That establishes a formal B2B channel. Several branded code names also match companies that publish partner claims: - [CodeStone's announcement](https://www.codestone.com/news/codestone-is-now-an-official-openai-partner/) says it joined the OpenAI SMB Channel Partner Programme and describes a formal commercial and technical relationship; - [First Focus's announcement](https://www.firstfocus.com.au/insights/first-focus-becomes-an-openai-services-partner/) says it entered an OpenAI Services Partner Agreement for Australia and New Zealand; - [AI Build's current offer page](https://www.ai-build.ltd/chatgpt-business/get-offer) describes the company as a UK OpenAI SMB Channel Partner and says its team reviews UK and Ireland eligibility. This supports an association between branded codes and partner campaigns. A string inside a code still cannot prove that a particular public post was authorized, much less that an employee leaked it. ### 2. Confirmed: Stripe can execute a discount, but that does not make Stripe the campaign owner [Stripe's official documentation](https://docs.stripe.com/billing/subscriptions/coupons) says merchants can create customer-facing promotion codes and restrict them by: - customer; - first-time order; - expiration date; - maximum redemptions; - minimum order value. [Stripe's Payment Links documentation](https://docs.stripe.com/payment-links/promotions) also describes prefilled promotion codes in shared links. This explains why users encountered Stripe-hosted checkout and why the same code might work for one account but not another. It does not show that Stripe chose OpenAI's partners, funded the offer, or controlled distribution. The original article confused infrastructure with commercial origin. ### 3. Confirmed: a public link need not be a leak OpenAI's own [Dropbox partner page](https://openai.com/business/partners/dropbox/) publicly advertises a limited-time ChatGPT Business offer: eligible Dropbox and ChatGPT users can use a promotional link to receive two Business seats and $50 in credits at no cost for 30 days. A [Think Technologies company post](https://www.linkedin.com/posts/think-technologies_chatgpt-activity-7464703850562785281-TvQV) also publicly promotes a buy-one-get-one offer and provides `chatgpt.com/p/THINK26`. As of July 29, 2026, opening: ```text https://chatgpt.com/p/THINK26 ``` still redirects to: ```text https://chatgpt.com/?promoCode=THINK26 ``` That confirms `/p/{slug}` remains a live redirect format. It **does not guarantee that a logged-in account qualifies or that inventory remains**, but it is enough to disprove the claim that OpenAI globally removed the link format. ## What did linux.do actually show was canceled? Community records are useful evidence of what users saw at a particular time, but they are not OpenAI policy documents. Placing the reports side by side reveals several distinct events: | Date | Reproducible record | What it supports | |---|---|---| | Apr 29 | Users report that a Team/Business first-month trial no longer appeared | One older trial ended; later partner campaigns are a separate question | | May 8–10 | Company-branded 48-month codes spread widely; a May 9 thread preserves the `THINKTECHNOLOGIESUS` link | The community heavily amplified partner-branded codes | | Jun 4 | Users report that a browser-console method for generating “Stripe long links” stopped working | A nonstandard checkout-link workflow changed or was restricted | | Jun 11–14 | New-offer discussions continued, with uncertainty about long-link generation | Promo eligibility and long-link tooling did not share one state | | Jun 30–Jul 1 | A public `penguinaius` link was posted and several users reported redemption outcomes | Public `?promoCode=` links continued circulating after June 4 | | July | Think publicly promoted `/p/THINK26`, while AI Build used reviewed email delivery | Public links and targeted delivery coexisted | The underlying community records are: - [April 29: first-month trial reported gone](https://linux.do/t/topic/2079203) - [May 9: 48-month offer discussion and preserved earlier link](https://linux.do/t/topic/2141860?page=2) - [June 4: console-generated Stripe long links reported broken](https://linux.do/t/topic/2302827) - [June 11: newer offers and uncertainty over long links](https://linux.do/t/topic/2380243) - [June 30: public `penguinaius` link and user reports](https://linux.do/t/topic/2502020) So “canceled” always needs an object. A specific batch of codes may have failed, one partner may have moved to reviewed email delivery, or a long-link technique may have stopped working. Those observations cannot be merged into a single global OpenAI policy. ## There is more than one distribution model ```mermaid flowchart LR A["OpenAI / partner campaign terms"] --> B["Public offer page or /p/ link"] A --> C["Eligibility form and emailed code"] A --> D["Partner-funded rebate"] B --> E["ChatGPT account and eligibility checks"] C --> E E --> F["Payment and subscription system"] D --> G["Partner service credits / invoice adjustment"] H["Community reposting"] --> B ``` At least four patterns are visible: 1. **An official OpenAI public campaign.** The Dropbox page provides a public redemption entry point and lists eligibility such as new customers or former customers who canceled at least 90 days earlier. 2. **A public partner link.** Think Technologies publicly distributes `/p/THINK26`. 3. **A screened partner code.** AI Build asks for a work email and team size, reviews eligibility, and says it usually emails a code within one business day. 4. **A partner-funded rebate.** [First Focus's current offer](https://www.firstfocus.com.au/services/core/chatgpt/) explicitly says customers buy licenses directly from OpenAI while First Focus funds a rebate of up to 50% for the first six months through service credits or invoice adjustments. It is not an OpenAI discount. The fourth pattern is particularly important: a “ChatGPT Business discount” is not necessarily a promotion code at checkout, much less a subsidy from OpenAI or Stripe. ## Price, duration, and failure conditions The old article's £11, AU$25, US$20, and 48-month figures were observations tied to particular dates, regions, and accounts. They should not be presented as currently purchasable prices. OpenAI's current [Business billing documentation](https://help.openai.com/en/articles/8792536-managing-billing-and-seats-in-chatgpt-business) gives the US self-serve baseline as: - $25 per user per month on monthly billing; - $20 per user per month on annual billing, billed annually; - a minimum purchase of two standard ChatGPT seats; - regional and local-currency variation. “Unable to use this offer” can have several causes: - the campaign expired or was deactivated; - the maximum redemption count was reached; - it is restricted to a customer or a first purchase; - the account, region, or billing information is ineligible; - it applies only to a new Business workspace; - the checkout flow or product configuration changed. Without campaign configuration or authoritative terms, an error message cannot prove that quota exhaustion was the cause. Before purchase, the issuer's latest terms and ChatGPT's final checkout page should control; verify renewal price, tax, seat count, and cancellation conditions. ## The corrected propagation claim linux.do was clearly an important Chinese-language aggregation hub in May 2026. It preserved codes, screenshots, timestamps, and redemption reports, and it accelerated distribution. The evidence supports this: > Partner-related campaigns reached public communities through intentional marketing, targeted distribution, or an unknown route, then forums and blogs amplified them. It does not support this: > Stripe issued codes to companies, employees leaked them, and linux.do reposted them. The first statement stays within the evidence. The second remains an unverified story. ## A checklist for future investigations 1. Find the OpenAI or campaign issuer's original page before treating a forum screenshot as terms. 2. Separate `/p/...` short links, `?promoCode=...`, Stripe Checkout Sessions, and partner rebates. 3. Record page update time, country, account eligibility, seats, promotional period, and renewal price. 4. Keep “one code failed,” “one checkout trick failed,” and “the entire channel was canceled” separate. 5. Do not bypass eligibility or payment controls with region spoofing, console scripts, or dubious payment tools. 6. Mark first appearance, leaker identity, and funding source as unknown unless primary evidence establishes them. ## Final assessment The original article got one broad pattern right: the branded codes were related to B2B partner channels, and linux.do was an efficient distribution node in the Chinese community. The accurate version is: **ChatGPT Business benefits come through multiple OpenAI or partner campaigns. Stripe supplies some of the payment and promotion-code machinery but is not a proven campaign source. Some partners moved to reviewed email delivery, while public offer pages and `/p/...` links still existed in July 2026. Audience, funding, expiration, and renewal rules must be verified for each campaign.** *Research checked through July 29, 2026. OpenAI and partners can change offers at any time; the issuer's latest terms and the final checkout page govern.* --- title: "RAG vs LLM Wiki vs Plain Text — A Decision Framework for Agent Long-Term Memory" description: "Cost, latency, accuracy, and maintainability trade-offs across three Agent memory approaches: RAG, structured knowledge bases, and plain-text context memory. Includes decision tree." url: "https://zhuoqidev.com/en/posts/memory-choice-framework/" language: "en" author: "Liu ZhuoQi" date: "2026-05-11" updated: "2026-07-30" tags: ["AI Agent","Memory","RAG","Context Engineering"] translation: "https://zhuoqidev.com/posts/memory-choice-framework/index.md" --- # RAG vs LLM Wiki vs Plain Text — A Decision Framework for Agent Long-Term Memory > **TL;DR** Every Agent builder hits this question eventually: where do I store user data so the agent remembers it next session? Three approaches dominate the landscape: RAG (vector retrieval), LLM Wiki (structured knowledge injection), and plain-text context memory (the CLAUDE.md / Cursor Rules pattern). Each has vocal advocates. But picking wrong is expensive — do RAG too light and it's a noise generator; do plain text too heavy and it's a token incinerator. Here's a decision framework you can use today. Start with the system boundaries. The important distinction is not the database brand; it is **when retrieval happens, how much content enters context, and who owns writing and maintenance**. ![A comparison of RAG, LLM Wiki, and plain-text context memory](https://zhuoqidev.com/_astro/three-memory-paths-bilingual-v1-4k.BXbpw5Dn_1ycmAD.webp "Three memory paths: RAG retrieves on demand from large heterogeneous collections; an LLM Wiki emphasizes structure and auditability; plain text fits a small set of high-value rules that need deterministic injection. No path dominates every dimension.") --- ## What Each Approach Actually Is | Approach | Core Mechanism | Examples | |----------|---------------|----------| | **RAG** | Vector retrieval → top-k chunks → inject into prompt | Mem0, Zep, LangChain RAG, Cursor Codebase Index | | **LLM Wiki** | Structured docs → full or on-demand injection into system prompt | Claude Projects, GPTs Knowledge, Notion AI | | **Plain Text** | Markdown/text files → directly concatenated into system prompt | CLAUDE.md, Cursor Rules, AGENTS.md, Devin Knowledge | The key difference isn't *where* data is stored — it's *how it's retrieved* and *when it's injected*. --- ## Decision Matrix | Dimension | RAG | LLM Wiki | Plain Text | |-----------|-----|----------|------------| | **Data volume** | Large (>100 docs) | Medium (10–100 docs) | Small (<10 files, <200 lines) | | **Update frequency** | High, real-time or near-real-time | Medium, weekly/daily | Low, project-level conventions | | **Retrieval need** | Semantic matching ("find the most relevant paragraph") | Structural navigation ("see chapter 3, section 4") | None, full-load | | **Latency** | +50–500ms (embed + retrieve + rerank) | 0 (preloaded) or +100ms (on-demand fetch) | 0 (fully in prompt) | | **Token cost** | Low (only relevant chunks injected) | Medium (per-chapter injection) | High (entire file every call) | | **Maintainability** | Low (chunk strategy, embedding model, retrieval params) | Medium (document structure needs upkeep) | High (it's Markdown — edit and commit) | | **Explainability** | Low ("why was this chunk retrieved?") | High ("because you asked about chapter 3") | Highest (everything is visible) | | **Hallucination risk** | High (retrieval noise → bad context → hallucination) | Low | Low | | **Best for** | Support KBs, codebase search, large-scale doc QA | Project docs, product manuals, compliance KBs | Coding conventions, project rules, personal preferences | --- ## When to Use RAG RAG is not a silver bullet. **Only reach for RAG when your data genuinely exceeds prompt capacity.** If you have 20 documents, shoving them all into the prompt beats RAG every time — the cost of retrieval noise far exceeds the token savings. **RAG makes sense when:** - You have >100 documents and users only care about 1–3 per query - You need semantic matching, not keyword matching - Your data updates in real time (e.g., connected to a live database) - You can tolerate occasional irrelevant retrievals **Common RAG failure modes:** - Building a vector DB for 10 documents — retrieval noise > signal gain - Arbitrary chunk sizes — too small loses context, too large kills precision - Skipping reranking — irrelevant chunks in top-k poison the model - Mismatched embedding and generation models — semantic space misalignment > Rule of thumb: **first, try to fit everything relevant into the prompt. Only go RAG when it genuinely won't fit.** This order matters — RAG is a last resort, not a default. --- ## When to Use LLM Wiki An LLM Wiki is structured documentation that the model can reference on-demand or in full. Unlike RAG, it doesn't rely on vector similarity. Unlike plain text, it isn't dumped wholesale — **it's a "table of contents" with actual content behind it.** **LLM Wiki fits when:** - Your knowledge has clear hierarchical structure (API docs, product manuals, compliance rules) - Users need to "flip to a section" rather than "search for a snippet" - You need human review and version control (critical for compliance) Claude Projects' Project Knowledge and GPTs' Knowledge feature are canonical LLM Wiki implementations. **LLM Wiki vs RAG — the essential difference:** RAG says *"I think these chunks match your question."* LLM Wiki says *"Here's the table of contents. Which chapter do you need?"* The former relies on semantic similarity; the latter on structural navigation. The former can guess wrong; the latter can't — but it requires the user or Agent to know which chapter to open. --- ## When to Use Plain Text Context Memory This is the CLAUDE.md / Cursor Rules / AGENTS.md pattern. One Markdown file, injected in full into the system prompt every call. **It sounds primitive. In the right context, it's optimal.** **Why the "dumb" approach often wins:** 1. **Auditable** — every change is a `git diff` 2. **Version-controlled** — memory has git history; you can roll back to Tuesday 3. **Zero latency** — no embedding, no retrieval, straight concatenation 4. **Zero noise** — every word you wrote is in the prompt; no "wrong chunk retrieved" 5. **Harder to prompt-inject** — content is human-written, not AI-auto-generated Cursor 1.2 adding mandatory user approval for Memories, and Devin defaulting to suggestion-only — these are post-prompt-injection design consensus. Plain text memory doesn't require "trusting what the AI remembered" because every line was written by a human. **Plain text shines for:** - Project-level conventions ("We use Java 17, Spring Boot 3.x") - Coding standards ("No Lombok, use records") - Personal preferences ("Answer in Chinese, code comments in English") - Agent behavior constraints ("Confirm before invoking tools") **Plain text fails when:** - Data exceeds ~200 lines — eats too much context window - Knowledge needs frequent updates — every change requires manual file edits - Knowledge is shared across projects — copy-paste leads to divergence --- ## The Decision Tree ```mermaid flowchart LR %% layout: tree Q(["How much knowledge data do you have?"]) Q --> A["<10 Markdown files
<200 lines total"] --> A1["Plain text context
CLAUDE.md / Cursor Rules
Why: zero latency, git-friendly, zero noise"] Q --> B["10–100 docs
clear structure"] --> B1["LLM Wiki
Claude Projects / GPTs Knowledge
Why: structured navigation, on-demand loading,
human-reviewable"] Q --> C[">100 docs, or semantic
search is essential"] --> C1["RAG
Mem0 / Zep / custom vector DB
Prerequisite: you've verified “shove it all in the prompt”
doesn't actually fit
Heads-up: RAG's maintenance burden is 10x the other two"] classDef q fill:#fff3bf,stroke:#f08c00 classDef pick fill:#d3f9d8,stroke:#2f9e44 class Q q class A1,B1,C1 pick ``` --- ## The Mistake Everyone Makes Many teams deploy all three simultaneously: ``` CLAUDE.md (plain text) + Vector DB RAG (dozens of docs) + Wiki knowledge base (product docs) ``` Three systems injecting into the same prompt. The result: - Token costs explode - Contradictory information (plain text says Go, stale Wiki says Python) - Debugging becomes "which chunk caused this answer?" **The fix: pick one as the primary memory layer. Use the others only when the primary one demonstrably falls short.** For individual developers and teams under 10 people, plain text + an LLM Wiki is almost always enough. RAG is for when you scale past that. --- ## Relationship to the LLM Memory Research This article is the engineering companion to [Why LLMs Have No Memory](https://zhuoqidev.com/en/posts/llm-memory-research/). That report covers the four-layer stack (Bare LLM → In-Architecture Memory → Long Context → Agent Memory Layer). This post focuses entirely on **L4 Agent Memory Layer selection**. One more time for Karpathy's analogy, because it's too useful: > **Weights = ROM** (burned in at training, static) > **Context Window = RAM** (directly addressable during inference) > **KV Cache = Working Memory** (formed at test-time) > **External Storage = Disk** (persistent but requires retrieval) Your choice determines what your Agent's "hard drive" looks like — a fast SSD (plain text), a mountable filesystem (LLM Wiki), or a database with a search engine (RAG). --- ## Summary | If you have... | Choose | |---------------|--------| | Small data, low change frequency, need auditability | **Plain Text Context** | | Structured knowledge, need chapter-level referencing | **LLM Wiki** | | Large data, need semantic search, can tolerate retrieval noise | **RAG** | No silver bullets. But one iron rule: **avoid RAG until you actually need it — and when you do, you'll know.** --- title: "Embedding CSS Animation Demos in Hugo Articles" description: "Use custom shortcodes to run live CSS animations directly in Hugo blog posts — no CodePen account needed." url: "https://zhuoqidev.com/en/posts/css-animation-demo/" language: "en" author: "Liu ZhuoQi" date: "2026-05-04" updated: "2026-07-30" tags: ["Hugo","CSS","Animation","Shortcode"] translation: "https://zhuoqidev.com/posts/css-animation-demo/index.md" --- # Embedding CSS Animation Demos in Hugo Articles > **TL;DR** Hugo shortcodes make it easy to embed live code demos. Here are three ways: The diagram first separates their boundaries: a native shortcode, a CodePen iframe, and a CodeSandbox app serve different complexity levels and execute code in different places. ![Three ways to embed an interactive demo in a Hugo article](https://zhuoqidev.com/_astro/hugo-embed-paths-bilingual-v1-4k.CwTl65a1_1W7utd.webp "Three embed paths: prefer a native shortcode for simple, versioned effects; CodePen for shareable snippets; and CodeSandbox for a complete interactive application.") ## 1. Inline CSS Demo (No External Service) A spinning loader animation, right in the article: ```html demo height=200 caption="Pure CSS Spinner"
``` A gradient text animation: ```html demo height=160 caption="CSS Gradient Text"
ZhuoQi Dev
``` ## 2. Embed CodePen If you already have CodePen creations, embed them with a single shortcode: ``` {{< codepen id="yourPenID" height="400" tab="result" >}} ``` ## 3. Embed CodeSandbox For React / Vue components, use CodeSandbox: ``` {{< codesandbox id="yourSandboxID" height="450" view="preview" >}} ``` --- These three shortcodes cover most code demo scenarios — no extra tools needed. --- title: "Building a Personal Site with Hugo and Dual-Stack CDN" description: "How I set up Hugo + Blowfish with Alibaba Cloud OSS/CDN for China and Cloudflare Pages for international visitors — ICP filing, geo-DNS routing, and GitHub Actions dual-stack deployment." url: "https://zhuoqidev.com/en/posts/hugo-dual-cdn-blog/" language: "en" author: "Liu ZhuoQi" date: "2026-05-04" updated: "2026-08-03" tags: ["Hugo","Alibaba Cloud","Cloudflare","CDN","ICP Filing"] translation: "https://zhuoqidev.com/posts/hugo-dual-cdn-blog/index.md" --- # Building a Personal Site with Hugo and Dual-Stack CDN > **TL;DR** One Hugo artifact is published to both Alibaba Cloud OSS/CDN and Cloudflare Pages, while geo-aware DNS sends domestic and international visitors to different edge networks. ## Why Hugo When picking a framework for a personal blog, my top criterion was **low maintenance cost** — I didn't want to abandon writing three months later because of npm dependency hell. Hugo is a single binary, requires no Node.js, builds thousands of posts in 1-2 seconds, and the Blowfish theme comes with dark mode, full-text search, multilingual support, RSS, Open Graph, and reading time estimates out of the box. Day-to-day writing only requires touching Markdown files. ## Architecture The diagram contains two distinct flows. Read downward from Git Push for the **publishing path**; read from GeoDNS toward the two CDNs for the **request path**. Both destinations must come from the same build to prevent content-version drift. ![A dual-stack Hugo deployment across Alibaba Cloud and Cloudflare](https://zhuoqidev.com/_astro/dual-cdn-publishing-bilingual-v1-4k.CTZ4_5IA_Z1K6zKQ.webp "Dual-CDN publishing: GitHub Actions builds once, then uploads the same artifact to Alibaba Cloud OSS/CDN and Cloudflare Pages. GeoDNS selects the visitor path while the content commit stays identical.") Annual cost: approximately ¥212 (~$30 USD): - Domain `zhuoqidev.com`: ¥85/yr (bought 3 years) - Function Compute resource pack (ICP filing): ¥101/yr (¥126 annual pack with 20% ongoing discount) - Alibaba CDN 100GB traffic pack: ¥20/yr - OSS storage: ~¥6/yr - Cloudflare Pages: ¥0 ## ICP Filing Without a Server Websites served to mainland China visitors need an ICP filing, which requires a "filing carrier" (a server IP). Instead of buying a full server, Alibaba Cloud's **Function Compute resource pack** (¥101/yr) works as a filing carrier and provides a filing service code. Timeline: Alibaba Cloud initial review 1-5 working days + MIIT review 10-20 business days, about 3-6 weeks total. Plenty of time to finish the site while waiting. ## Geo-DNS Routing Alibaba Cloud DNS free tier supports "domestic / international" split routing: - Domestic → Alibaba CDN CNAME - International (default) → Cloudflare Pages CNAME Domestic visitors get the ICP-compliant Alibaba CDN; international visitors get Cloudflare's free global CDN — one domain, two acceleration paths. ## Deployment Push to GitHub → Actions runs `hugo build` → uploads in parallel to OSS and Cloudflare Pages. The whole process takes 2-3 minutes. Publishing a post is nearly instant. ## A Later Addition: an Image Viewer for 4K Architecture Diagrams Architecture diagrams in technical posts often contain small labels. Blowfish's built-in `medium-zoom` only enlarges an image to fit the screen and closes it on the next click. Even when the source is 4K, readers cannot keep zooming by percentage or pan across the details, so I replaced it with a site-wide image viewer. The implementation adds no new frontend dependency. The built-in zoom is disabled in `hugo.yaml`, while `assets/js/image-viewer.js` and its CSS are added to the Hugo Pipes asset pipeline. Hugo minifies and fingerprints them during the build, then the same static artifact is published to both CDNs. The script only attaches to `.article-content img:not(.nozoom)`, so article illustrations open in the viewer without hijacking avatars, icons, or card thumbnails. On open, the viewer calculates a fit-to-screen scale from the viewport and the image's natural dimensions. Wheel and button zooming preserve the point under the cursor, so the detail being inspected stays in place. A zoomed image can be dragged within bounded edges. Pointer Events track two contacts for pinch zoom on touch devices, while the keyboard supports `+`, `-`, arrow keys, `0` for fit, `1` for actual size, and `Esc`. A click on either the image or the surrounding empty space also closes the viewer, while movement beyond the drag threshold prevents accidental dismissal. The toolbar reports the live percentage and switches its labels with the article language. You can try it by clicking any 4K diagram in the [Codex Agent design deep dive](https://zhuoqidev.com/en/posts/codex-agent-design/). It is also a useful property of a static site: richer interaction remains cacheable JavaScript and CSS, so the dual-CDN publishing topology does not need to change. --- Upcoming posts will dive into Agent architecture design, memory layer selection, and multi-agent orchestration.