◆projects · ctxd

vv0.1.0 · Apache-2.0 · GitHub

Abstract teal and amber illustration for ctxd

ctxd is a proxy for LLM APIs. Point a client at it and it forwards each request unchanged, until the conversation grows past its window. Then it folds the oldest part into a summary and keeps the original text in a local store.

The problem

Long agent sessions fill the context window. Each turn resends the whole history, so the cost grows, the prompt cache misses, and eventually the request fails. Trimming the history loses what the agent will need later: the file it read, the error it hit, the decision it made. ctxd keeps those details. The summary carries a fold id, and the original messages stay in the store, so the agent can ask for them back.

How it works

ctxd illustration

ctxd listens on 127.0.0.1 only. It routes two paths: /v1/messages, the Anthropic shape that Claude Code uses, and /v1/chat/completions, the OpenAI shape. Each path has its own upstream and its own window, set in ctxd.toml or by environment variable.

On each request ctxd estimates the size as bytes divided by four. If the estimate passes 80% of the window, it takes the oldest block of messages outside the last six, cuts it at a tool-call boundary, and sends it to a summariser. The block is replaced by one message, [fold#<id>] <summary>. Fold ids are stable. ctxd puts the cache_control marker back, so the prompt cache can still hit.

The summariser is any OpenAI-compatible endpoint, with a model name from the config. Its prompt asks for exact file paths, identifiers, numbers, decisions, errors and open tasks, with no preamble. If folding fails, ctxd forwards the original request unchanged and logs FOLD-SKIP with the reason. Without a [summarizer] url, nothing is folded and ctxd is a plain passthrough.

The store keeps each session in a directory under ~/.cache/ctxd/<id> with three flat files. log.bin holds the raw message JSON and the summaries, back to back. idx.bin gives each message's offset and length. folds.bin records which messages each fold covers. Message ids are positions and never change. When the store opens after a crash, it cuts a torn tail.

Capabilities

  • Passthrough: with no summariser, requests and responses go through unchanged.
  • Folding on both routes: the example config sets a window of 200,000 tokens for messages and 128,000 for chat.
  • Header and error relay: upstream headers and error frames are passed back to the client, covered by tests/gate-s2.sh.
  • ctx_expand: returns the original messages behind a fold placeholder, byte for byte.
  • ctx_search: searches the original messages of the session, folded ones included, for an exact substring.
  • compress: asks ctxd to fold older history now, instead of waiting for the window threshold.
  • Tool handling: the three ctx_ tools are offered only to sessions that hold folds. ctxd runs those calls itself and answers the client with the final message.
  • Docker: docker/build.sh builds ctxd:latest. docker/run.sh binds 127.0.0.1:8787 and keeps the store in ~/.cache/ctxd.
  • systemd user service: service/install.sh builds ctxd into ~/.local/opt/ctxd and enables ctxd.service. The config goes in ~/.config/ctxd/ctxd.toml, and the summariser key in ~/.config/ctxd/env, mode 600.
  • Environment overrides: CTXD_PORT, CTXD_UPSTREAM_MESSAGES, CTXD_UPSTREAM_CHAT, CTXD_WINDOW_MESSAGES, CTXD_WINDOW_CHAT and CTXD_SUMMARIZER_KEY.

Numbers

ctxd illustration

tests/gate-s1.sh covers the store. It ran on 2026-11-16 and passed in 4.5 seconds. It writes 20,000 messages totalling 4.76 MB, about 1.19 million tokens by the bytes-over-four estimate, then reopens them.

measure result
append 20,000 messages 91.7 ms, about 218,000 messages per second
reopen and load 20,001 messages resident memory up 5.1 MB
overhead beyond the raw bytes 23.6 bytes per message
search, one hit at the last message 0.095 ms
search, 17,188 hits 0.87 ms

The same gate round-trips 2,000 messages, including messages of 1.5 MB and 3 MB, byte for byte after a reopen. The repository has no recorded run of the fold policy on billed tokens. docs/PLAN.md names that A/B test, on Claude Code and on a local model, as the acceptance test for folding, and it is still to be run.

Install and first run

With Docker:

docker/build.sh && docker/run.sh
ANTHROPIC_BASE_URL=http://127.0.0.1:8787 claude

From source, ctxd needs Mojo and a mojo-bento checkout next to the repository, with its scripts/deps.sh run once. Set MOJO_BENTO if the checkout lives elsewhere. The gates run locally, with no paid API: tests/gate-s0.sh to tests/gate-s3.sh, one at a time. tests/gate-s4.sh checks the Docker image and needs Docker.

Limits

  • It needs a mojo-bento checkout beside it. There is no package yet.
  • It listens on localhost only. It is not built to be a network service.
  • Release 0.1.0 was shipped without the Docker gate, tests/gate-s4.sh.
  • The summaries come from a model, and the repository does not measure how faithful they are.
  • ctx_search looks for exact substrings. It does not do semantic search.
  • A store's log.bin is limited to 4 GiB, because offsets are 32-bit. Past that, the store raises an error.
  • Release 0.1.0 is the first public release, dated 2026-11-11. History before it is not public.

Source and license

Source: https://github.com/amarbarosys/ctxd (public, Apache-2.0). Release notes for 0.1.0 are in docs/releases/v0.1.0.md.