Direct answer
Queue backpressure is the policy that decides which inference requests may enter a bounded work queue, which should wait, which can be coalesced, and which must be rejected or cancelled. It is not the same problem as choosing a worker thread. A worker-thread design moves expensive native calls off the UI thread; admission control prevents the application from accepting more work than its single runtime owner can safely process. Without a bound, repeated taps, restored screens, and background triggers can retain prompts, image buffers, result state, or coroutine closures while waiting for inference. Begin with product semantics. A live camera preview may prefer the newest frame and discard stale frames; a user-submitted translation may need FIFO ordering; a duplicate tap for the same request may be ignored while the first is active; a cancellable summary may be replaced by a newer user edit. These are different policies and must not be implemented by one generic unlimited queue. Define request classes, maximum in-flight and queued work, expiry behavior, cancellation owner, and the user-visible outcome for full capacity. State whether fairness is FIFO, priority-weighted, or per-feature quota. A high-priority task should not permanently starve older accepted work unless that policy is intentional and observable. Coalescing is safe only when requests are semantically equivalent and share the same artifact, runtime, prompt, input, and result contract. Two image frames are not duplicates merely because they came from the same screen. A queue can merge repeated refresh intent while retaining the latest input, but it must preserve which request IDs were superseded and return a result only to the surviving owner. When identity cannot be proven, reject or queue separately rather than silently merging user content. For privacy and memory, avoid storing raw inputs in backlog diagnostics; keep only request IDs, class, enqueue time, queue age, state, and cancellation/rejection reason. Make the queue lifecycle explicit: submitted, accepted, queued, started, streaming, completed, cancelled, expired, rejected, or failed. Publish backlog depth, oldest age, in-flight count, and per-class wait state as operational observations; do not call them performance measurements unless a defined test captured them. When cancellation occurs, remove queued work immediately if it has not started. For active native inference, propagate cancellation through the supported runtime boundary and do not release shared resources until cleanup has completed. A UI collector disappearing is not by itself proof that the queued request or native call stopped. Test bursts and races: repeated taps, navigation away/back, new input replacing old input, cancellation while queued, cancellation while active, a full queue, worker failure, and runtime close. Assert the queue never exceeds its configured capacity, no request receives another request’s output, old jobs cannot update a new screen, and every accepted request reaches one terminal state. Measure fairness and memory only under a fixed fixture and exact device/runtime; the repository does not provide a complete benchmark dataset. The deliverable is predictable overload behavior and auditable state, not a universal queue size or latency claim.
Repeated button taps build a backlog behind one native inference
A user taps an on-device summary action several times while the first request is still running. A screen recreation adds another request, while an older background task is also waiting. The application currently starts every coroutine and provides no visible state for accepted versus pending work.
Decision
Introduce an application-owned bounded admission coordinator with per-request identity, a documented ordering/coalescing policy, explicit cancellation, and visible rejection when full.
Expected outcome
The user sees which request is active or queued, obsolete work is not retained indefinitely, and completed output cannot be delivered to a different request.
Evidence to collect
The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:
- Queue ownership policy
- Overload fixtures
- Rejected and recovered outcomes
A runtime comparison is publishable only when artifact identity, build configuration, selected backend, task fixture, and failures are recorded.
Implementation workflow
Recorded evidence items must be connected and compared to support assessment of queue behavior. Queue ownership policy, overload fixtures, and rejected and recovered outcomes are gathered together as documented evidence so they can be recorded side by side and compared against one another. The connection lies in linking who owns the queue, what overload conditions were applied via fixtures, and which requests were rejected or subsequently recovered, enabling consistent comparison rather than drawing conclusions beyond these recorded items.
- Define request classes and semantic value. List interactive generation, replaceable preview, user edits, retries, and background maintenance separately. For each class, state whether FIFO, latest-wins, deduplication, or rejection preserves product meaning; do not merge inputs just because they share a screen.
- Set queue and in-flight bounds. Choose explicit limits for active native calls and waiting requests based on the runtime ownership contract and measured memory behavior for the target test setup. Define behavior at capacity, expiry for stale work, and fairness across classes. Do not present a guessed universal count as a supported limit.
- Assign identity, ownership, and cancellation. Give every accepted request an ID, artifact/runtime identity, task owner, and terminal-state path. Queued cancellation removes the request before it starts; active cancellation uses the documented runtime boundary and waits for cleanup before releasing shared resources.
- Implement safe coalescing and admission outcomes. Coalesce only requests whose input, artifact, prompt, and result contract are demonstrably equivalent. Record superseded IDs and deliver the surviving result only to its owner. Otherwise queue separately or reject with a user-visible reason.
- Test overload, fairness, and observability. Inject tap bursts, screen recreation, replacement input, queued/active cancellation, full capacity, worker failure, and runtime close. Assert bounded depth, one terminal state per accepted ID, fair service under the chosen policy, and no cross-request output. Track queue age and state without logging raw inputs.
Acceptance criteria
- Queue capacity, per-class ordering/fairness, expiry, coalescing rule, and full-queue behavior are explicit and testable.
- Every request ID reaches exactly one terminal state and output is delivered only to its owning task/session.
- Queued work is removed when cancelled before start; active cancellation waits for supported runtime cleanup before resource reuse.
- Backlog observability uses redacted metadata and does not claim a universal queue size, latency, or memory limit.
Decision table
| Dimension | Choice or evidence | How to interpret it |
|---|---|---|
| Unbounded coroutine launch | Accept every tap immediately | Moves calls off the UI thread but permits unbounded retained work and ambiguous ownership. |
| Bounded FIFO queue | Preserve accepted order until capacity, then reject or wait | Predictable for ordered tasks; can make newer interactive work wait behind obsolete requests. |
| Latest-wins/coalescing queue | Replace equivalent stale intent with newer input | Useful for replaceable previews or edits only when semantic equivalence and superseded IDs are explicit. |
Failure patterns to prevent
- Accepting unlimited work
- Hiding rejected requests
Accepting unlimited work and hiding rejected requests are pitfalls to rephrase, with release wording limited to recorded evidence. Accepting unlimited work means taking on queued items without a stated bound, and hiding rejected requests means keeping rejected outcomes from view. The phrasing here is drawn from recorded evidence and does not extend to mechanisms, causes, or implementation behavior beyond what is stated.
Minimum reproducibility record
| Layer | Record |
|---|---|
| Device | Manufacturer, model, chipset, RAM class, operating-system build |
| Software | Application version and git commit |
| Model | Family, variant, revision, format, file length, hash, quantization |
| Runtime | Name, revision, requested backend, observed backend evidence |
| Workload | Fixture revision, input hash, prompt hash, output policy |
| Outcome | Completed, failed, cancelled, fallback, and privacy-safe diagnostics |
Release checklist
- ☐ The primary query is answered without an unsupported number.
- ☐ Every artifact and runtime is pinned.
- ☐ The representative task and failure policy are explicit.
- ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
- ☐ User-content and network boundaries are documented.
- ☐ Result wording applies only to the recorded configuration.
- ☐ The page links to raw method or evidence when results are added.
Sources and related evidence
- Android Memory Profiler
- Android threading and task management guidance
- Kotlin channels and buffered communication
Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.
Frequently asked questions
Is a bounded inference queue the same as a worker thread?
No. A worker thread controls where work runs; backpressure controls how much work is accepted and retained. An app can use worker threads and still create an unbounded queue.
Can I coalesce all requests from the same screen?
No. Inputs, prompts, artifact versions, and result contracts may differ. Coalesce only requests proven equivalent, preserve superseded IDs, and otherwise queue or reject separately.
Does cancelling the screen stop queued or native inference?
Not automatically. Define task ownership separately from UI observation. Remove queued work when cancelled and use the runtime’s supported cancellation/cleanup boundary for active native work.