Test Thread Count for On-Device AI

Direct answer

CPU threadCount is a configuration input for a CPU backend, not the device core count, a guaranteed worker count, or a performance result. Evaluate it with artifact, resolved runtime, task, app build, and device held constant; change only supported CPU settings and preserve failures as well as successful runs. Compare the native default with predeclared explicit values. Do not select only the fastest run after reviewing results. For the API, pin the source first. In LiteRT-LM Config.kt at commit 84d96699d4b8712792c54dcface7a436dd8a4d92, Backend.CPU.threadCount is a CPU-backend setting and numOfThreads is deprecated. With the deprecated field unset, null or zero selects the native default. Engine.kt still falls back to numOfThreads when threadCount is null, so do not mix both fields or label an inherited positive legacy value as default. That immutable snapshot does not prove Cove resolved the same API: capture the actual Gradle artifact and inspect its available constructor before naming a field in a current-app report. Thread count is executor-specific. EngineConfig distinguishes backend, visionBackend, and audioBackend. The pinned Engine.kt extracts separate CPU counts for the main backend and audio backend, but does not show a separate vision count at that native call. Do not infer that text, image, and audio share one independently tunable value. Cove's current source defines CPU_THREADS as 4, prefers GPU for the main backend then falls back to CPU(4), configures audio as CPU(4), and routes vision through the main backend. These are configuration facts, not observed worker counts, evidence of four-thread performance, or a recommendation for any device. Before measuring, define one modality and accepted task: fixed non-sensitive input, prompt or preprocessing, language, output limit, and semantic or structural checks. Keep text completion, image analysis, and audio transcription in separate matrices. A faster output that fails acceptance is not equivalent. If internal worker count or actual backend is not observable, report the requested setting and that limitation separately. Use Android Studio system traces to inspect app and system threads, CPU-core scheduling, and UI rendering around named events. A trace can reveal overlap or contention; a thread row alone cannot identify every LiteRT-LM native worker or prove the requested count was honored. Mark initialization, task start, first accepted output, terminal result, and cleanup. Note profiler overhead and keep instrumented timings distinct from production measurements. Control run order and thermal drift. Hold device, OS, app build, artifact, fixture, charging/background state, and warm-up rule constant; separate cold initialization from repeated inference. Where practical, interleave or randomize settings. Retain every sample, fallback, failure, and inconclusive run. Record runtime/API revision, backend request and observed path, threadCount or native-default marker, timing boundaries, acceptance result, and available memory/thermal observations. Decide only for the tested combination. Keeping the native default is reasonable when explicit values show no repeatable accepted-task benefit, profiling suggests contention, or the runtime cannot confirm the selected path. A result for one CPU executor does not apply to GPU, vision, or audio unless each relevant path was tested. The repository has no complete real-device benchmark dataset: this is a protocol, not a result. Without BENCHMARK-01 raw records and THERMAL-01 repeated-workload evidence, publish no tokens-per-second, latency, battery, memory, or best-value claim. Mark hidden internals unknown; runtime or model updates require a fresh comparison.

A CPU fallback needs a thread-setting comparison

A team maintains a local model path that requests GPU and falls back to CPU when initialization fails. The current source configures a CPU thread constant, but the app has no raw device runs showing whether another supported count changes task completion, UI responsiveness, or thermal behavior.

Decision

Treat the source constant as a configuration baseline only, pin the resolved LiteRT-LM API, and compare explicit CPU values with the native default on one fixed artifact/device/task at a time.

Expected outcome

The team can choose a scoped CPU policy or keep the default, while reporting requested count, observed task path, accepted output, and unknown backend details separately.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Thread configuration
• Ordered task series
• Thermal and completion record

A runtime comparison is publishable only when artifact identity, build configuration, selected backend, task fixture, and failures are recorded.

Implementation workflow

1. Pin the resolved API and executor. Record app build, resolved LiteRT-LM artifact, model revision, device/OS, and the exact backend field. Confirm threadCount exists in the resolved dependency; the pinned upstream source is a reference, not proof of the app's version.
2. Define one workload and acceptance rule. Choose text, vision, or audio separately, freeze its input fixture, prompt/preprocessing, output reserve, and required result properties. Keep initialization and inference events separate.
3. Predeclare comparable settings. Include the native default marker and only explicit values supported by the exact API. Label the current Cove constant as configured baseline where relevant, never as optimal or measured.
4. Capture timing and scheduling evidence. Use named start/end events and an Android system trace for thread/core scheduling and UI interaction. Record what the trace can show without claiming it identifies every native worker or proves the requested count was honored.
5. Repeat under a controlled thermal method. Keep device state and fixture fixed, apply the same warm-up rule, vary only threadCount, interleave runs where practical, and retain every failure, fallback, and inconclusive observation.
6. Decide within the evidence scope. Compare accepted task outcomes as well as resource observations. Keep results separate by executor and backend; do not publish a general best setting, speed, battery, or compatibility claim without validated raw records.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• The record binds every run to the resolved LiteRT-LM API, exact artifact, device/OS, executor/backend, task fixture, and threadCount or native-default setting.
• Only one thread setting changes at a time; initialization, first output, terminal task result, fallback, failures, and thermal conditions remain separately recorded.
• Conclusions are limited to tested configurations, and no preferred value, speed, battery, memory, or device-support claim is published without raw validated evidence.

Decision table

Failure patterns to prevent

• Assuming more threads are faster
• Comparing different workloads

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• LiteRT-LM Config.kt at pinned commit 84d96699d4b8712792c54dcface7a436dd8a4d92
• LiteRT-LM Engine.kt at the same pinned commit
• Cove InferenceEngine.kt at published commit fbadbb48fd7abd43dd943b38450e8bc6c658fc04 (configuration evidence, not benchmark data)
• Android Studio Profiler — record a system trace
• Android Developers — threading and CPU resources

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions