Isolate Android Local AI Instrumentation Tests

Direct answer

AndroidX Test Orchestrator is an instrumentation-test harness option, not a general device-reset service. Its documented model invokes each test in a separate Instrumentation instance. With clearPackageData enabled, it clears the target application package data between tests by running pm clear. That helps expose accidental dependence on persistent app preferences, databases, and files, and gives a crashed test a better chance of not contaminating the next invocation. It does not promise to erase Android system caches, other packages, kernel state, hardware state, thermal history, or every source of device-to-device variation. Describe the boundary as application-package isolation, not a factory-clean device. The choice is about what the test is meant to prove. Use ordinary instrumentation execution when a test deliberately exercises state across steps or when repeatedly launching the test app would obscure the behavior. Use Orchestrator when independent test cases must not inherit app-owned state, or when one test can crash the process and should not prevent later cases from starting. If the app keeps model files in package-owned storage, clearing package data can also remove a large installed fixture. The runner does not make that fixture reappear automatically; setup and transfer cost belong in the test design and runtime budget. Pin the Android Gradle Plugin and AndroidX Test dependencies before configuring the option. The AndroidX runner guide shows the ANDROIDX_TEST_ORCHESTRATOR label in its setup example, while the AGP 8.7 TestOptions.execution reference documents ANDROID_TEST_ORCHESTRATOR and says HOST disables on-device orchestration. Do not combine a token copied from one page with coordinates from another and present it as a universally copyable configuration. Check the exact resolved AGP DSL and its matching AndroidX runner/orchestrator dependency combination, then verify the dependency graph and task actually launched by CI. Use explicit compatible coordinates from the project version catalog; do not paste historical sample versions or use a dynamic latest coordinate. Treat clearing as a boundary with consequences, not as a substitute for fixtures. A reliable local-inference test should create or provision its artifact deliberately, verify its identity and readable bytes, then run the behavior under test. If the artifact is too expensive to reinstall for every case, partition tests: keep small state-isolation cases under package clearing and run artifact-heavy integration cases with explicit setup and cleanup. A suite may also use a dedicated test-only app-owned directory or a setup phase, but it must not quietly reuse stale production state and then call the test isolated. Record whether the model was staged before the test, restored by harness setup, downloaded, or absent. Do not assume a device-wide state is clean because the application package was cleared. Android may retain system-managed compilation artifacts or unrelated caches, and the Orchestrator documentation does not make a universal guarantee about those layers. Hardware, OS scheduling, thermal state, external storage, provider documents, network state, and other installed packages remain separate environmental inputs. If a result depends on any of them, declare and control that dependency with an appropriate fixture or test environment. If it cannot be controlled, report it as a limitation rather than attributing the difference to app data isolation. Measure the cost using the suite that engineers will actually run. Compare ordinary execution with Orchestrator using the same device or emulator image, test selection, build, and fixture policy. Include startup overhead, package clearing, artifact provisioning, retries, and failures in the wall-clock total; do not time only the test method. A slower isolated suite may still be the right choice for correctness, while a fast stateful suite may remain useful for focused development. Keep correctness coverage and throughput as separate decisions instead of weakening the reset boundary to meet a target duration. The generic protocol here makes no claim about Cove configuration or measured Cove behavior. Before adopting it in any app, inspect that project’s actual AGP version, runner setup, dependency resolution, model storage path, and CI device. A green emulator run demonstrates only the tested harness, package, artifact, and environment. It does not show that all devices clear identical state or that a production model can be restored for free. Preserve the distinction between application data, model bytes, and environmental state in test names and reports.

Independent tests keep inheriting a model selection

An instrumentation suite tests a local assistant whose selected-model preference and downloaded artifact persist in app storage. A test passes by reusing state left by an earlier test, while another occasionally crashes and leaves later cases unreliable.

Decision

Use Orchestrator for tests that require independent app-package state, explicitly provision the artifact after clearing when it is needed, and retain a separate integration path for expensive model setup.

Expected outcome

The suite can distinguish state-isolation failures from artifact setup failures and report the added harness and provisioning time without calling the emulator fully clean.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Resolved test execution configuration
• Model fixture location and reset policy
• Per-test setup and cleanup evidence

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Pin the runner toolchain. Record AGP, AndroidX Test runner and orchestrator versions, resolved dependency graph, device image, and CI task. Configure the execution enum supported by that AGP version and verify that CI actually launches Orchestrator.
2. Name the state boundary. List app preferences, databases, package files, model artifacts, external documents, system-managed caches, and device conditions separately. State which are expected to be cleared and which remain outside the package-data guarantee.
3. Classify tests by fixture cost. Separate small tests of app-owned state from model-heavy integration cases. Decide whether each case needs package clearing, an explicit staged artifact, or a deliberate multi-step stateful scenario.
4. Provision and verify model inputs. After any package clear, install required fixtures through a documented test setup path. Verify artifact identity, readable content, and readiness before asserting inference behavior; do not depend on a prior case.
5. Compare suite behavior and cost. Run matched selections with and without orchestration where appropriate. Include invocation startup, clearing, fixture setup, retries, failures, and cleanup in suite duration, and retain correctness outcomes separately from speed.
6. Report only the tested boundary. Name package clearing, runner mode, artifact setup, device image, and known uncontrolled conditions in the report. Avoid claims that system caches, hardware state, or every device were reset.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• The CI report identifies the resolved AGP and AndroidX Test versions and confirms which instrumentation execution mode actually ran.
• Tests that need model bytes provision and verify them after package clearing instead of inheriting an earlier case’s artifact or ready flag.
• Isolation claims are limited to the target application package; system caches, other apps, hardware, and device conditions are documented separately.

Decision table

Failure patterns to prevent

• Treating process isolation as a hardware reset
• Assuming large model fixtures survive pm clear

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• Android Developers: AndroidJUnitRunner and Test Orchestrator
• Android Gradle Plugin 8.7 API: TestOptions.execution

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions