How to Test an On-Device AI Model: A Reproducible Method

To test an on-device AI model, freeze the device, operating system, model artifact, runtime, backend, input, output policy, and measurement procedure. Then record cold start, representative work, repeated work, memory, thermal context, and every failure. A model name plus one speed number is not a reproducible benchmark.

Start with the decision the benchmark must support

A benchmark should answer a release question, such as:

• Can this artifact complete the product’s minimum task on the lowest target device?
• Does a runtime change improve the task without changing output quality?
• Does a larger context remain stable during repeated use?
• Can the application fall back when the preferred backend is unavailable?
• Is a new artifact safe to release to the current device set?

Do not begin by collecting every metric a profiler exposes. Write the decision and its pass condition first. Otherwise the result becomes a dashboard without a product conclusion.

Record artifact identity

The test record should identify what actually ran:

{
  "model_family": "...",
  "model_variant": "...",
  "artifact_revision": "...",
  "artifact_format": "...",
  "artifact_bytes": 0,
  "artifact_sha256": "...",
  "runtime_revision": "...",
  "backend": "...",
  "build_type": "release"
}

If the hash is unavailable, store that absence explicitly. Do not substitute a model-card label for artifact identity. Conversion, quantization, packaging, and runtime configuration can change results even when two files share a family name.

Define task fixtures and expected behavior

Use inputs taken from the product’s real task shape, then remove or replace sensitive content. A translation app needs representative text, image, and audio cases. A transcription app needs versioned recordings and reference text. A RAG feature needs a fixed document collection, queries, and expected source passages.

Each fixture should define:

• input artifact and checksum;
• preprocessing configuration;
• prompt or instruction;
• output limit and stopping rule;
• acceptance criteria;
• failure severity;
• whether output must be deterministic or only satisfy a property.

Separate format checks from quality checks. Valid JSON is not a correct answer. A fluent answer is not necessarily grounded in the supplied source.

Capture the environment

Record device model, chipset, RAM class, operating-system build, battery state, power connection, application build, and whether other normal workloads were present. Record the selected execution path rather than inferring it from hardware marketing.

A benchmark that cannot confirm whether it used CPU, GPU, or NPU cannot support a backend comparison. If the runtime does not expose the path directly, record the available diagnostic evidence and label the path unresolved.

Keep device preparation consistent. A cold device and a recently stressed device are different test environments. Neither is “more real” in isolation; both may be relevant to the product.

Separate cold and warm phases

Measure phases independently:

1. application process start;
2. artifact discovery and validation;
3. runtime creation;
4. model initialization;
5. first representative request;
6. later requests in the same process;
7. release and reload.

Users experience the complete path, but developers need the breakdown. An optimization that improves generation while doubling initialization may be wrong for a short, occasional task.

Measure memory as a timeline

Peak memory includes more than the model download: weights, runtime buffers, KV cache, input tensors, operating-system headroom, and allocator behavior all contribute. Record memory at baseline, after model load, during the representative request, at maximum promised context, after repeated work, and after cleanup.

Android Studio Memory Profiler and system memory diagnostics can observe application behavior. Use the same measurement path for every comparison and document what metric was recorded.

Do not restart the process between every request if the product keeps the model loaded. That pattern hides accumulation and cleanup failures.

Test sustained performance and thermal behavior

One successful inference does not establish sustained performance. Repeat a fixed workload while recording elapsed time, memory, device temperature or available thermal signals, execution path, and failures. The goal is not to publish an impressive first run. It is to learn whether the user experience changes during continued work.

Define the repetition rule before the test. Keep input, output limit, and idle interval constant. Stop on a documented condition rather than when the chart looks convincing.

Report raw sequences, not only an average. Averages can hide slow starts, degradation, retries, or termination. Mark any run that changed backend or recovered from an error.

Include failure as data

Store failed initialization, allocation failure, invalid output, cancellation failure, timeout, backend fallback, process death, and corrupted artifact as first-class results. A device that completes only some runs should not be summarized by the successful subset.

Use normalized error categories while preserving a privacy-safe diagnostic attachment for investigation. Never publish an error-frequency article until a real record or official issue exists.

Comparison table template

Do not compare unlike fixtures or silently change output limits. If equivalent model artifacts are unavailable across runtimes, describe the result as a product-path comparison.

Cove’s current evidence status

The Cove repository does not yet contain a complete real-device benchmark dataset for its on-device AI stack. Cove can publish architecture, artifact, and methodology facts now, but performance and device-compatibility conclusions must wait for raw records containing device, chipset, OS, artifact, runtime, backend, input, timing, memory, thermal context, and failure outcome.

That absence is useful because it defines the next engineering artifact instead of encouraging invented numbers. Benchmark pages should be generated from stored records once the closed-testing device runs begin.

Release checklist

• State the release decision the benchmark supports.
• Pin the exact artifact and runtime.
• Version fixtures and preprocessing.
• Record actual execution-path evidence.
• Separate cold, warm, and sustained phases.
• Measure memory throughout the lifecycle.
• Preserve raw run sequences.
• Store failures and fallbacks, not only successes.
• Repeat on the minimum intended device tier.
• Keep private user content out of diagnostics.
• Publish the method and raw schema with any conclusion.

Continue with RAM requirements for memory details and LiteRT-LM vs llama.cpp for a runtime comparison framework.

Last reviewed: 2026-09-16.