Five AI workers walked into a rate limit at the same time

There is a particular kind of bug that only appears on a real phone, on a real network, on a cold start. This is one of them.

What happens when the app opens

Opening the app can wake up five separate background jobs:

• the resource sync that indexes syllabus documents,
• the embedding pipeline that turns text into vectors,
• a backfill pass for documents indexed by an older version,
• topic extraction,
• and the retry engine that finishes generating flashcards and quizzes that were interrupted.

Each of those had its own little boolean guard — an isRunning flag, so a job could not start twice over. That felt responsible. It was also useless for the problem we actually had.

A per-job flag stops a job racing itself. It does nothing about five different jobs racing each other. And when a student opens the app for the first time after installing it, all five have work to do and no reason to wait.

The symptom

What the student saw was slow AI features. What we saw in the logs was a wall of HTTP 429 — too many requests — from a provider key on a free tier that had a perfectly reasonable rate limit, being asked for five things at once.

The nastier problem was quieter than that. Two of those jobs could be processing the same document at the same time. Neither knew about the other, and both would write their results when they finished. Which one won was down to timing.

The fix is a queue with one rule

Every background AI call now goes through a single gate:

AiWorkQueue.instance.run(label, task)

Exactly one background task runs at a time. The interesting part is the granularity.

The queue holds a unit — one topic, one document, one resource — not a whole job. That distinction matters more than it sounds. A full resource sync can run for minutes. If it held the gate for its entire duration, a student who ticked a topic to generate flashcards would sit behind the whole sync before their deck even started building. So the sync releases the gate between resources, and the flashcard job gets in between them.

What is not allowed through the gate

Interactive chat deliberately bypasses it.

It is tempting to route everything through one queue and call the problem solved. But the queue's job is to stop background work from stampeding the provider. A student waiting for an answer they just asked for is not background work. If they were queued behind a sync that has three hundred documents left, the app would look broken — and they would be right.

So the rule is: if a human is waiting on it, it does not queue.

The details that bit us

Re-entrancy. A task that is already inside the queue sometimes needs to call something that also wants the queue. Rather than deadlock, the gate detects that it is already held by the current zone and just runs the nested task inline.

Timers. Our retry logic has timeouts on it. A timeout that starts ticking while the task is still queued, rather than running, will fire before the work has begun — and you get a timeout error for something that never started. Timeouts now start after the gate is acquired, not before.

Why not just raise the rate limit?

Because the rate limit was never really the issue. Even on a generous tier, five jobs writing the same document concurrently is a correctness problem, and paying for a bigger quota would have hidden it rather than fixed it. The queue is about ordering and exclusivity; the 429s were just how the bug made itself visible.