Context Tokens vs Words in Multilingual Mobile AI

Direct answer

Words and model tokens are different counting systems. A word is a language or user-interface unit defined by a linguistic or application rule; a token is a unit produced by one specific tokenizer from one exact input representation. Some languages mark word boundaries with spaces more consistently than others. Scripts, punctuation, numbers, names, normalization, special control markers, and subword vocabulary can all change how text becomes tokens. A character count, a space-separated word count, or a count from a different model’s tokenizer is not an exact estimate of the current model’s context use. Start by naming the model artifact and runtime, then identify whether that integration exposes the tokenizer or an input-token count. The tokenizer must correspond to the exact artifact revision and the same preprocessing and chat template used at inference. A displayed source paragraph may expand into system instructions, role markers, separators, prior turns, tool definitions, and modality markers before it reaches the model. Count the serialized request only when the current API exposes that representation; otherwise report the exact count as unavailable. Do not substitute a tokenizer from a similarly named model or use a cloud model’s counting endpoint as if it measured an offline runtime. For a multilingual comparison, prepare parallel fixtures with the same user task and meaning, written by qualified reviewers in each language and script under test. Preserve the source text, normalization form, punctuation, whitespace policy, language direction, and template version. Include both ordinary short prompts and deliberately long examples; add mixed-script names, digits, code, and punctuation only when they match the actual product task. A sentence-by-sentence translation is not guaranteed to have equal words, characters, semantic content, or token count. Do not publish a ratio such as “one word equals N tokens” or rank a language’s efficiency using a tiny hand-picked sample. Keep input measurement separate from the context budget. A context limit belongs to an exact model variant and runtime path; it may include control tokens, previous conversation turns, tool schemas, or non-text content. For multimodal models, an image or audio payload can consume model-specific representation beyond the visible prompt text. Gemma 4 documentation describes context in tokens and notes a configurable visual-token budget for images; that page does not give a token-per-word rate for every language, nor does a family-level limit prove that a particular mobile package accepts that total. The current Cove Gemma E2B configuration has an empty SHA-256 field according to COVE-MODEL-02, so do not claim an exact hash-verified artifact when reporting a measurement. Measure the request that matters. Record source characters and a clearly defined word count for human-readable comparison, exact tokenizer token count if exposed, rendered prompt count if exposed, prior-turn and tool-definition contribution, configured output reserve, truncation or rejection behavior, and runtime acceptance result. Note whether the runtime counts special tokens or inserts template text internally. If the total token count is hidden, use a fixed boundary corpus with unique markers and report what the integration accepts or rejects; do not reverse-engineer a conversion ratio from a few examples. Token count is not a memory measurement. MEMORY-01 states that peak inference memory depends on weights, runtime buffers, KV cache, input tensors, operating-system headroom, and allocator behavior, not download size alone. A prompt that fits a context limit can still exceed a tested memory envelope; conversely, a character count cannot tell whether an input is accepted. Test representative maximum cases on the exact app build, artifact, runtime, device, and OS, but keep measured resource results separate from tokenization data. The repository has no complete raw device benchmark set, so this guide gives a measurement protocol rather than a performance or device-support claim. Define outcomes before running the fixture set: accepted unchanged, rejected with a clear explanation, truncated at a named observable boundary, or unavailable because the runtime hides the relevant count. Preserve a test record with fixture revision, artifact identity, tokenizer identity if known, runtime, template, input language, output allowance, and failure stage. Keep private user text out of telemetry; use synthetic or approved fixtures for repeatable comparisons. Report per-language distributions only after enough reviewed examples and a stated sampling method exist. A useful conclusion is scoped to those fixtures and the exact tokenizer/runtime integration. It does not establish universal token ratios, comparative language quality, maximum device capacity, or support for every language.

A multilingual summary request reaches the app’s length warning

A developer compares an English prompt and a Japanese translation. The English version has more whitespace-delimited words, while the Japanese version has more visible characters. The app shows a warning near its input limit, but the local runtime exposes no tokenizer count.

Decision

Do not infer which prompt uses more context from word or character totals. Freeze the exact artifact, template, and runtime, then test accepted behavior with reviewed parallel fixtures and mark hidden token counts unknown.

Expected outcome

The report separates human-readable length, exposed token evidence, runtime acceptance, and any resource measurement without publishing a universal language ratio.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Tokenizer and artifact identity
• Versioned multilingual text fixtures
• Token count, acceptance, and failure records

A device badge must come from an exact device, OS, artifact, runtime, backend, and workload record, not parameter-count intuition.

Implementation workflow

1. Pin the tokenizer and execution identity. Record the exact artifact and revision, tokenizer identity or bundled file if known, runtime package/version, template, prompt construction, app build, and device/OS for every comparison.
2. Prepare reviewed parallel fixtures. Use the same task and meaning across languages, reviewed by fluent readers. Preserve script, Unicode normalization, whitespace, punctuation, direction, and mixed-language cases relevant to the product.
3. Count only observable units. Keep source characters and a declared UI word rule separate from tokenizer tokens. Count the rendered request only if the runtime exposes it; otherwise report token count as unavailable.
4. Account for context components. Record template text, roles/control markers, prior turns, tool definitions, non-text input, and output reserve when observable. Do not treat visible user text as the entire model request.
5. Exercise boundary behavior. Run short, typical, and long fixtures with unique start/middle/end markers. Record accepted, rejected, truncated, or unobservable outcomes and distinguish input limits from output stopping.
6. Report distributions and limits. Use a stated fixture sampling method and preserve per-language results. Keep token evidence separate from memory, speed, translation quality, or device-support conclusions.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• No fixed word-to-token or character-to-token ratio is presented as an exact measure for the tested artifact.
• Every count is labeled by unit and source: UI words, Unicode characters, exposed tokenizer tokens, rendered request, or unavailable boundary.
• Context acceptance, output allowance, and memory observations are reported as separate outcomes for the exact fixture and runtime.

Decision table

Failure patterns to prevent

• Using a fixed words-to-tokens ratio
• Publishing one language as universal

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Google AI for Developers — Gemma 4 model card
• Google AI Edge — LiteRT-LM Kotlin API for Android
• SentencePiece authors — language-independent subword tokenizer paper

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions