Direct answer
When checking differences in outputs from a local AI system, compare the versioned outputs by their required properties, their provenance, and their failure state rather than relying only on surface-level wording. Do not infer beyond this stated focus.
Evidence to collect
The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:
- Paired result fixtures
- Configuration identities
- Property-level review record
A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.
Implementation workflow
Maintain paired result fixtures together with their configuration identities, and record a property-level review record that captures each comparison made. These items should be kept together and reviewed against one another so that any observed difference can be traced back to the specific configuration and the specific properties being checked. Do not infer beyond this stated evidence.
- State the decision the test must support.
- Pin device, software, artifact, runtime, and backend.
- Version the workload and expected properties.
- Separate cold, representative, and sustained phases.
- Keep ordered raw records including failure.
- Publish method and evidence before conclusions.
Failure patterns to prevent
- Treating text difference as regression
- Comparing unlike configurations
A common pitfall is treating a text difference as a regression, and another is comparing unlike configurations against each other. Release wording should be limited strictly to the evidence that has actually been recorded in the review record, and should not extend beyond what those recorded entries support. Do not infer beyond the stated pitfalls.
Minimum reproducibility record
| Layer | Record |
|---|---|
| Device | Manufacturer, model, chipset, RAM class, operating-system build |
| Software | Application version and git commit |
| Model | Family, variant, revision, format, file length, hash, quantization |
| Runtime | Name, revision, requested backend, observed backend evidence |
| Workload | Fixture revision, input hash, prompt hash, output policy |
| Outcome | Completed, failed, cancelled, fallback, and privacy-safe diagnostics |
Release checklist
- ☐ The primary query is answered without an unsupported number.
- ☐ Every artifact and runtime is pinned.
- ☐ The representative task and failure policy are explicit.
- ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
- ☐ User-content and network boundaries are documented.
- ☐ Result wording applies only to the recorded configuration.
- ☐ The page links to raw method or evidence when results are added.
Sources and related evidence
Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.