On-Device RAG on Android: A Practical Local-First Architecture

On-device RAG on Android is not one model call. It is a pipeline: ingest documents, split them into retrievable units, create embeddings, store a local index, retrieve relevant units, construct a grounded prompt, and generate an answer. Building and testing each stage independently is the difference between a debuggable local knowledge feature and a chat screen that sometimes sounds correct.

Define the product boundary first

Before selecting a vector database or language model, write the promise in user terms. For example:

The user selects documents. The app answers questions using only those documents. Core search and answer generation work in airplane mode. Deleting the collection removes the source text, index, and conversation derived from it.

That statement determines the architecture. If document parsing, embeddings, retrieval, or generation silently requires a remote service, the product is hybrid rather than fully local. Hybrid can be a valid design, but it must be named and its transmitted data must be visible.

The pipeline and its test seams

Treat each stage as a contract with inspectable input and output.

The seams make failures diagnosable. If the wrong passage was retrieved, changing the generation prompt will not fix the index. If the correct passage was retrieved but ignored, re-chunking the document may be wasted work.

Ingestion: preserve provenance from the first byte

Document ingestion should create a durable record for the source, its type, import time, parser version, and deletion state. Every extracted unit needs a path back to the source location. A response that quotes a chunk without identifying its document and section is difficult for the user to verify.

Do not mix parser cleanup with semantic rewriting. Normalize encoding and remove mechanical artifacts, but preserve the user’s wording. If the ingestion stage summarizes content before indexing, later answers can only retrieve the summary, not the original evidence.

Run parser tests with empty files, repeated headers, tables, multi-column layouts, images without extractable text, and interrupted imports. An ingestion job should fail visibly or mark partial output; it should not quietly index half a document as if the import succeeded.

Chunking: optimize for retrieval, not a universal token number

Chunk size is a product parameter, not a constant copied from a tutorial. The correct boundary depends on source structure, query style, embedding behavior, and the context available during generation.

Use stable chunk identifiers derived from the source record and position. Store neighboring relationships so the retrieval layer can expand around a strong match when additional context is useful. Preserve headings and source metadata separately from body text rather than flattening everything into one anonymous string.

Build a small labeled query set before tuning. Each query should list the source passage expected to answer it. Chunking improves when measured against retrieval outcomes, not when judged by how tidy the code looks.

Embeddings and local index lifecycle

The embedding stage must use a consistent model and preprocessing policy for both stored chunks and incoming queries. When the embedding model or preprocessing changes, the existing index may no longer be comparable and should carry a migration or rebuild requirement.

Store these fields with the index:

• embedding model identity and revision;
• preprocessing version;
• vector dimension;
• source collection revision;
• index schema version;
• build completion state.

An index rebuild should be transactional from the user’s perspective. Build a new version, validate it, then switch the active pointer. Replacing the live index in place risks leaving a partially rebuilt collection after interruption.

Local storage also needs deletion semantics. Removing a document should remove its normalized text, chunks, vectors, cached previews, and derived references. A “delete” button that only hides the title does not satisfy a local-first privacy promise.

Retrieval before generation

Retrieval should be testable without loading a generative model. For every labeled query, inspect the ranked chunk identifiers and source locations. Track misses separately from weak rankings. A missing passage indicates ingestion or indexing trouble; a low-ranked passage points toward chunking, embeddings, query construction, or ranking.

Do not send every vaguely related chunk to the generator. Define a policy for how many results can enter, how duplicates are handled, and how source diversity is preserved. If the evidence is insufficient, the correct response may be “not found in these documents.” That behavior is more valuable than a fluent answer that escapes the collection.

Generation runtime and memory

Android generation may use different runtime paths. LiteRT-LM has its own packaging and APIs in the Google AI Edge stack, while llama.cpp centers on GGUF and its C/C++ runtime. The RAG layer should depend on an application-owned inference interface so the document pipeline is not locked to one runtime’s session object.

Peak memory cannot be inferred from model download size alone. The model weights, runtime buffers, KV cache, retrieved context, input tensors, operating-system headroom, and allocator behavior all contribute. RAG increases context pressure because retrieved passages are added to the prompt. Test with the maximum evidence the product promises, not only an empty-context greeting.

Prompt construction should make sources explicit and assign the model a narrow job: answer from the supplied passages, distinguish passages from instructions, and surface uncertainty when evidence is absent. The application, not the model, should own citation links and source identifiers.

Fully local and hybrid RAG

Local inference removes the need to send content to a remote inference server, but it does not prove that the application makes no other network requests. The same rule applies to RAG. A pipeline is fully local only when ingestion, embeddings, indexing, retrieval, and generation all complete without transmitting the user’s content.

A hybrid design may keep documents and retrieval local while sending selected passages to a remote generator. Another design may generate locally but use a remote embedding service. Both can be useful, but both transmit content at a specific stage. Document that stage, minimize the transmitted material, and give the user a clear offline behavior.

Minimal validation plan

• Import a small collection with known answers.
• Verify normalized text against the source.
• Verify stable chunk IDs and source locations.
• Run labeled retrieval queries without generation.
• Confirm the expected passages appear in ranked results.
• Test “answer not present” cases.
• Restart the app and repeat retrieval from the persisted index.
• Interrupt an import and confirm partial data is not activated.
• Delete a source and verify all derived records are removed.
• Run the complete task in airplane mode.
• Observe peak memory at the maximum promised context.
• Change an embedding revision and verify the index rebuild path.
• Redact or replace sensitive test documents before logging failures.

A local-first release rule

Do not publish a broad device-compatibility claim from one successful query. Release evidence should name the artifact, runtime, device, source collection, context policy, and test set. Retrieval quality and generation quality should be reported separately. This lets readers understand whether a failure came from missing evidence or from the model’s handling of evidence.

Continue with on-device AI RAM requirements for the memory methodology and on-device AI vs cloud AI for routing boundaries. Chinese deployment material is organized in the 奇连 AI 端侧专题.

Last reviewed: 2026-09-16.