What caused the GLM-5.3 data exposure on Cloudflare Workers AI?

Cloudflare's Workers AI leaked data between customers through the GLM-5.3 model (@cf/zai-org/glm-5.3) from August 27 to September 9, 2026. A configuration mismatch combined with a bug in the SGLang inference engine meant 155 responses to prompts longer than 681,280 tokens contained data from other customers' requests. Cloudflare says it found no evidence of deliberate exploitation and that no other Workers AI model was affected ^[1].

What happened on Workers AI?

A small number of customers calling GLM-5.3 on Workers AI got responses that contained data from other customers' requests to the same model, according to Cloudflare's incident report published on October 26, 2026 ^[1]. Cloudflare counts 155 affected responses. All of them came from prompts longer than 681,280 tokens.

Gadi Evron of Knostic reported the issue through HackerOne on September 8 and included a reproducible script ^[1]. Cloudflare met the SGLang maintainers at RadixArk on September 10, the day after the incident, and timed the publication with the researcher and RadixArk.

All times and dates in the table come from Cloudflare's timeline ^[1].

How did the KV cache leak work?

The leak came from two overlapping issues: a prefill memory pool smaller than the advertised context window, and an SGLang code path that let a rejected request continue ^[1]. Fixing the pool size alone prevents the bug from being exercised ^[1].

Workers AI serves GLM-5.3 with disaggregated prefill and decode. A pool of prefill GPUs processes the prompt and builds its KV cache, the per-token attention state, then hands that cache over RDMA to a pool of decode GPUs that generate the response ^[1].

Fault one: the pool was smaller than the context window

Cloudflare deployed the model with a context window of 1,048,576 tokens, but the prefill KV pool held only 681,280 tokens. "We were aware of this mismatch before launch," the report says ^[1]. Pre-launch testing checked HTTP status codes and streaming behavior, so it confirmed that long requests returned successfully. It did not check whether the output was correct.

Fault two: a rejected request kept going

When a prompt did not fit the pool, SGLang marked the request as rejected and replaced the prompt with a one-token placeholder. In disaggregated mode, the admission queues checked whether a request was finished, not whether it was rejected, so the request went on through the pipeline ^[1]:

1. Decode reserved 8 KV pages for the request.
2. Prefill admitted the rejected request and ran a forward pass on the placeholder.
3. Prefill transferred one page, P0, and signaled completion.
4. Decode accepted the handoff and attended over all 8 pages. Pages P1 to P7 still held earlier requests' data.
5. The client received HTTP 200 instead of HTTP 400.

The stale pages were readable because "Pages are not currently zeroed between requests; the next request that gets a page simply overwrites it" ^[1]. That is normal for inference engines, but it makes correct page-ownership tracking a security property rather than an efficiency detail.

The engine bug was already public. SGLang issue #30233, opened on July 6, 2026, describes the same pattern: an over-length request aborted on the prefill side transfers one token of KV cache, and decode then produces garbage output from the incomplete cache ^[2]. The reporter hit it with SGLang 0.5.11, GLM-5.2-FP8 and the Mooncake transfer backend, on a 284,937-token request against a 284,922-token limit. The issue was closed for inactivity on September 5, 2026, three days before the HackerOne report.

How did Cloudflare and SGLang fix it?

Cloudflare fixed the configuration first. The prefill KV pool now holds about 2.04 million tokens, roughly twice the context length, so "A prompt that fits the context window now always fits the pool" ^[1].

The engine fix retires rejected requests at the admission queue, before any bootstrap, allocation or KV transfer. Upstream, SGLang PR #38935, "[PD] Do not admit intake-rejected requests to a PD handoff", adds that check at both admission points, PrefillBootstrapQueue.add and DecodePreallocQueue.add; it was merged on September 16, 2026 ^[3]. Cloudflare lists the related upstream changes by release ^[1]:

The checksum option from PR #39500 is off by default. When it is on, prefill computes Adler-32 checksums of the KV data and sends them with the final chunk, decode verifies them before batching, and a mismatch aborts the request; both engines must enable the flag and use matching KV layouts ^[4]. The v0.5.22 release notes include the change "expose deferred decode KV release metrics" ^[5].

Cloudflare also deferred KV release, bounded Mooncake transfer batches, changed its load balancer so client disconnects no longer abort a request mid-transfer, and added deployment checks that compare pool size with context length ^[1].

What does this mean for teams running AI agents?

The trigger was a long prompt, and agents generate long prompts. Accumulated tool output, retrieved documents and conversation history are what push a request past several hundred thousand tokens, and that is exactly the size range that hit the bug ^[1]. An agent that received a corrupted answer would have seen HTTP 200 and continued working with it.

The report does not say whether affected customers were notified or whether they need to do anything ^[1]. If your team sent prompts longer than 681,280 tokens to @cf/zai-org/glm-5.3 between August 27 and September 9, 2026, check whether any of those responses fed records, tickets or messages, and ask Cloudflare directly about your account.

On the client side, three habits reduce exposure to this class of failure. Count tokens before sending and keep long-running agents well below a model's documented limit. Validate that answers make sense for the request, for example that a structured reply mentions the entities you asked about. Do not put secrets or personal data into long agent contexts that do not need them.

How do you check a self-hosted SGLang deployment?

If you run SGLang with prefill/decode disaggregation, Cloudflare recommends upgrading to v0.5.22 or later and lists five checks for operators ^[1]:

1. After any change, compare the KV pool size in the startup logs with the context length, on every prefill and decode node.
2. Upgrade SGLang to v0.5.22 or later.
3. Enable deferred KV release on both roles and bound Mooncake transfer batches.
4. Turn on corruption detection where your model supports it, and confirm that requests are being verified.
5. Configure load balancers so they do not cause unnecessary aborts.

For step 4, the checksum flag goes on both the prefill and the decode engine ^[4], which you start with --disaggregation-mode prefill and --disaggregation-mode decode ^[6]:

pip install --upgrade "sglang>=0.5.22"

python -m sglang.launch_server --model-path <model> \
  --disaggregation-mode prefill --disaggregation-enable-kv-checksum

python -m sglang.launch_server --model-path <model> \
  --disaggregation-mode decode --disaggregation-enable-kv-checksum

What are the limits and open questions?

The report leaves several things open. It does not say how many accounts received another customer's data, which customers' data appeared, or whether leaked content included readable fragments of other prompts or only generated text that was conditioned on them ^[1].

The remaining fixes are planned, not shipped. Cloudflare plans page-level ownership tags, where every physical KV page carries a tag that changes when the page is reallocated; its prototype costs under 1% in throughput and tail latency. It also plans formal verification of its RDMA-based KV cache transfer protocol ^[1]. Until engines track page ownership that way, isolation between tenants depends on every code path that frees, reuses or transfers a page being correct.