Direct answer
Measure local model readiness as named events rather than one unlabeled stopwatch value. Cold app launch describes process and Activity startup conditions; cold model loading additionally depends on whether the runtime and artifact are initialized or cached. Warm can mean an Activity restart in the same process, a second request in an open runtime, or file pages served from cache. These conditions are not interchangeable. Android Macrobenchmark defines cold, warm, and hot app startup modes and can repeat user-visible journeys, but its startup metric does not automatically measure model readiness. Add app-level instrumentation that records artifact lookup and verification, runtime initialization, first request submission, first output, and completion of a representative task. Publish the event definitions beside any result. Separate at least five intervals: process launch to first frame; first frame to a usable model screen; artifact lookup and integrity verification; native engine or session initialization; and first inference to accepted output. A user may see a screen before the model is ready, or the model may be ready while the task still takes time. Do not call the first frame AI-ready, and do not call total task time model-load time if it includes permissions, file selection, downloads, animations, or speech playback. Fix the artifact identity, runtime build, app variant, backend request, prompt fixture, output rule, and device state for comparisons. If the artifact is downloaded between runs, that is a delivery test; if Android kills the process, that is a new process but may leave file pages cached. Macrobenchmark cold mode ends the process between preparation and measurement, while warm mode restarts activities without ending the process. Neither mode guarantees a cold storage cache or a new native-library state. Say exactly what was reset and retained. Retain every iteration and environment record. Use a release-like target with the profiling configuration required by the tool, and keep measurement code from changing the path being measured. Define ready through a user-relevant condition, such as an enabled action and a completed fixed fixture, not a non-null engine reference. Keep failed initialization, timeout, cancellation, fallback, and incomplete output instead of discarding them. A report should identify device model and OS build, app revision and variant, artifact revision and byte length, resolved runtime version, requested and observed backend evidence, fixture, startup mode, cache/reset method, power and thermal notes, and tool. Cove has no complete real-device benchmark dataset, so this seed can support a method but not a Cove phone ranking, load duration, or minimum-device requirement. Distinguish runtime initialization from first-frame delay, token generation, and output-rule failure. Preserve raw traces and per-run records so a later reviewer can reproduce the comparison rather than trust a headline. Define the question and user-ready condition first: app startup, artifact verification, engine creation, first inference, or task completion. Pin app, runtime, artifact, fixture, backend request, output requirement, device, and reset procedure using resolved versions. Instrument each boundary and rehearse once without counting that run. Keep download and setup outside unless those are the subject. Run cold and warm conditions using a named Macrobenchmark mode or documented app-level procedure. Retain every trace, error, cancellation, and fallback. Check task equivalence before comparison and report method, scope, statistic, and limitations together. This is a measurement protocol, not a performance result; conclusions apply only to the exact artifact, runtime, app build, device, task, and state observed.
A fast splash screen hides delayed model readiness
A local assistant draws its first frame quickly, then keeps the main action disabled while a native model opens. A return visit in the same process feels faster. The team needs to separate app startup, model initialization, verification, and warm reuse without a complete real-device benchmark dataset.
Decision
Instrument UI and runtime intervals separately, compare named cold and warm conditions with one pinned artifact and fixture, and preserve every run without making a general device claim.
Expected outcome
The report identifies which boundary changed and whether the user task became ready sooner, without presenting first-frame timing as proof that inference is available.
Evidence to collect
The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:
- Timing event definitions
- Cold and warm fixture protocol
- All-run logs with device and artifact identity
A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.
Implementation workflow
- Define event boundaries. Specify timestamps for process launch, first frame, artifact lookup, integrity check, engine initialization, first inference, and task acceptance. State whether model initialization is eager, screen-triggered, or user-requested.
- Pin comparison inputs. Record app revision and variant, resolved runtime version, artifact revision and length, backend request, fixture, device and OS build, and output rule. Label any material change rather than mixing unlike runs.
- Describe cold and warm state. Name the Macrobenchmark mode and document whether process, Activity, runtime, model session, file-page cache, shader cache, and compilation state were reset or retained. Cold does not mean every cache is empty.
- Capture every outcome. Keep timestamps, traces, initialization errors, incomplete output, cancellation, and fallback. Define readiness with a representative task passing its acceptance rule, not a visible screen or engine object.
- Report only measured scope. Publish method, exact device and artifact, event boundaries, every run, and the summary statistic. Do not convert one startup or load result into a general speed, RAM, or compatibility claim.
Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.
Acceptance criteria
- The report distinguishes first frame, model ready, first output, and task completion and defines each timestamp boundary.
- Cold and warm runs state which process, runtime, artifact-cache, and compilation conditions were reset or retained.
- Every iteration, failure, cancellation, and fallback is retained with artifact, runtime, fixture, device, and app-build identity.
- No speed or compatibility claim exceeds the exact tested configuration and task.
Decision table
| Dimension | Choice or evidence | How to interpret it |
|---|---|---|
| Cold app startup | Process and Activity startup to a named display event | Measures launch experience; it does not by itself show that a local model is ready. |
| Cold model initialization | Artifact and runtime events through a ready state | Must state cache and session state; inference is excluded unless explicitly measured. |
| Warm reuse | Repeated task with retained process or runtime | A separate condition, not equivalent to a new process or disk read. |
| End-to-end readiness | Fixed input through accepted output | Closest to user completion, but applies only to the tested path and fixture. |
Failure patterns to prevent
- Publishing timing boundaries as inference speed
- Selecting only the fastest run
Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.
Minimum reproducibility record
| Layer | Record |
|---|---|
| Device | Manufacturer, model, chipset, RAM class, operating-system build |
| Software | Application version and git commit |
| Model | Family, variant, revision, format, file length, hash, quantization |
| Runtime | Name, revision, requested backend, observed backend evidence |
| Workload | Fixture revision, input hash, prompt hash, output policy |
| Outcome | Completed, failed, cancelled, fallback, and privacy-safe diagnostics |
Release checklist
- ☐ The primary query is answered without an unsupported number.
- ☐ Every artifact and runtime is pinned.
- ☐ The representative task and failure policy are explicit.
- ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
- ☐ User-content and network boundaries are documented.
- ☐ Result wording applies only to the recorded configuration.
- ☐ The page links to raw method or evidence when results are added.
Sources and related evidence
- Cove benchmark method
- Android Developers — Macrobenchmark overview and startup modes
- Android Developers — Inspect app performance with Macrobenchmark
- Cove benchmark raw-data requirements
Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.
Frequently asked questions
Does Macrobenchmark cold startup measure a cold model load?
Not automatically. Startup modes control app-process and Activity conditions. Instrument artifact access, runtime initialization, and model readiness separately, and describe file-cache and native-runtime state.
Can I publish only the average across runs?
Retain every iteration and state the summary method. An average may hide failed loads or outliers; report the distribution and failures, and use only a documented exclusion rule.
If the second request is faster, did inference improve?
Not necessarily. It may be a warm path with a retained process, session, allocator, cache, compilation state, or different prompt. Compare like-for-like conditions and separate generation from initialization.
Does the splash screen disappearing mean the model is ready?
Only when the product ties that event to a verified ready condition. A first frame may precede artifact verification and engine initialization; task readiness needs a representative accepted task.