On-Device OCR on Android: Separate Recognition from Understanding

On-device OCR on Android should be designed as two separate questions: “what characters are visible?” and “what does this image mean?” A dedicated recognition stage and a multimodal understanding stage can be combined, but their outputs, tests, and failure modes should remain distinguishable. If the product collapses them into one confident paragraph, users cannot tell whether the text was recognized incorrectly or interpreted incorrectly.

Define OCR output before choosing a model

OCR can mean several products:

• plain text copied from a page;
• text blocks with bounding boxes;
• reading order for a multi-column document;
• key-value extraction from a receipt or form;
• translation of recognized text;
• a semantic description of an image that happens to contain text.

These are not equivalent. A product that promises “copy all text” needs a loss-aware transcript. A product that promises “explain this sign” may prioritize meaning over exact punctuation. Write the output contract first and make the UI show which result it is presenting.

A testable pipeline

Use visible boundaries between stages:

capture -> orientation -> crop -> image normalization
        -> text recognition -> structure -> optional interpretation
        -> review -> copy/export

Each stage should preserve enough metadata to diagnose the next one. Keep the original image or a user-approved working copy until processing is complete. Record orientation and crop transforms so recognized regions can be traced back to source pixels.

For camera input, test more than a centered document. Include perspective, glare, shadow, low contrast, rotation, curved pages, mixed scripts, handwriting, dense tables, and partial text near frame edges. The goal is not to guarantee success everywhere. It is to define what the product does when recognition is incomplete.

Dedicated OCR and multimodal extraction

Google ML Kit’s official Text Recognition API represents recognized text through structural units such as blocks, lines, and elements. That output shape illustrates how a dedicated OCR contract differs from a free-form multimodal description.

A dedicated OCR engine usually exposes recognition-oriented structures such as lines, blocks, positions, or confidence. A multimodal language model may instead return text inside a broader response. The latter can be useful for semantic tasks, but output formatting alone does not prove literal fidelity.

Cove Photo currently uses the multimodal model path. Its analysis prompt asks the model to return visible text in a structured Text field. The parser stores that field as detectedText; the analysis UI can copy it, and history supports an OCR-only export. This is accurately described as the current product pipeline. It should not be described as a separate dedicated OCR engine unless the implementation changes.

That distinction matters for evaluation. The current pipeline should be tested for both extraction fidelity and response-format stability. If a dedicated recognizer is added later, its literal output should be stored separately from any model-generated interpretation.

Preserve raw, structured, and interpreted layers

A durable data model can retain three layers:

1. Raw recognition: text exactly as the recognition stage returned it.
2. Structured extraction: fields, blocks, order, and source regions.
3. Interpretation: description, summary, translation, or answers.

Users may want to copy raw text even when the interpretation is poor. Developers may want to rerun interpretation after a prompt update without repeating image capture. Keeping the layers separate supports both needs.

Never silently “correct” names, serial numbers, medicine labels, or amounts in the raw layer. A language model may produce more fluent output while making the text less faithful. Corrections belong in a visible derived layer.

Preprocessing is part of the product

Image preprocessing changes both recognition quality and resource use. Define and version:

• orientation handling;
• crop policy;
• maximum working resolution;
• color or grayscale conversion;
• contrast and sharpening policy;
• compression used before model input;
• whether multiple regions are processed separately.

A large source photo may be resized before inference. That can reduce memory pressure but also erase small characters. The correct setting depends on the actual documents and camera distances users provide. Build a representative image set and compare the complete product output after every preprocessing change.

Memory and lifecycle

Peak memory is not determined by the model file alone. Decoded images, resized tensors, runtime buffers, model weights, application bitmaps, operating-system headroom, and allocator behavior all contribute. Camera workflows can hold several representations of one image if capture, preview, preprocessing, and inference overlap.

Release resources stage by stage. Avoid retaining the full camera bitmap in navigation state after a durable file or reduced working copy exists. Test repeated capture and analysis in one process, because a single successful image will not expose accumulation.

Cancellation also needs a policy. If the user leaves during inference, decide whether the task finishes in the background, is cancelled, or remains resumable. The UI and persistence layer must agree on the result state.

Privacy and export

Local recognition removes the need to send image content to a remote inference server, but it does not prove that the app makes no other network requests. Test the complete capture-to-export workflow in airplane mode, then inspect traffic after connectivity returns.

Separate processing from user-directed sharing. Copying to the system clipboard and invoking a share sheet are explicit boundary crossings controlled by the user. Background upload is a different behavior and must never be implied by a local OCR claim.

Also decide whether images and extracted text participate in device backups. “Not uploaded by the app” and “never leaves the physical phone through any system service” are different claims.

Evaluation without invented accuracy

Create a versioned evaluation set from the documents the product targets. Store source images and verified reference text. Define normalization rules for whitespace, punctuation, casing, and line breaks before calculating any score.

Report results by failure class:

• missed text;
• substituted characters;
• wrong reading order;
• merged or split lines;
• hallucinated text not present in the image;
• correct recognition but incorrect interpretation;
• correct text but failed structured output.

Do not publish one universal accuracy number without the corpus, normalization, artifact, runtime, preprocessing, and language breakdown. A reproducible failure table is more useful than a polished percentage with no test definition.

Release checklist

• Define whether the product promises literal text, structure, or interpretation.
• Preserve raw recognition separately from generated explanation.
• Version preprocessing settings.
• Test small text, perspective, glare, rotation, mixed scripts, and tables.
• Measure repeated capture and inference memory.
• Make incomplete recognition visible.
• Verify airplane-mode processing.
• Keep copy and share as explicit user actions.
• Document backup behavior separately from inference traffic.
• Publish evaluation methodology before publishing an accuracy claim.

See offline camera translation for a user-facing scenario, camera translation in difficult conditions for capture constraints, and on-device AI RAM requirements for memory methodology.

Last reviewed: 2026-09-16.