Frame UTF-8 JSON Text Sequences and Report Uncertain Tails

Direct answer

A JSON text is one serialized value; concatenating multiple JSON texts does not by itself define a record boundary. RFC 7464 defines JSON Text Sequences using UTF-8 bytes, an ASCII Record Separator (RS, 0x1E) before each encoded JSON text, and a line feed (LF, 0x0A) after it. The framing is byte-oriented. A reader should locate RS boundaries in the UTF-8 stream first, then parse each candidate record, rather than split on arbitrary newline characters inside JSON strings or pretty-printed objects. The separator works because the RS control character must be escaped inside a JSON string and cannot otherwise appear as a raw JSON character. The LF after each encoder record is useful for detecting truncation of top-level numbers and literals such as `true`, `false`, and `null`, whose JSON spellings are not self-delimiting in the same way as an object, array, or string. A parser should require the specified whitespace boundary for those top-level forms and classify a possibly truncated record instead of accepting a valid-looking prefix. RFC 7464 deliberately distinguishes encoder and parser grammar so a parser can recover after damaged elements. When one record is malformed, a sequence parser can report that record and continue from the next RS. Multiple consecutive RS bytes do not create empty elements under the RFC parser grammar. Recovery is local to framing and syntax; it does not prove that a damaged record was unimportant, that sequence positions remain stable, or that missing records can be reconstructed. Do not use an array index as durable record identity where truncation or loss is possible. A JSON Text Sequence has no end-of-sequence marker. If the final bytes are missing, the reader may be unable to tell whether the producer intentionally stopped, the file was truncated, or one or more records were never written. The LF can help detect incomplete scalar records, but it is not a durable transaction commit, checksum, length prefix, or guarantee that every intended record reached storage. Record tail status separately: complete according to the observed framing, malformed record recovered, final record possibly truncated, or stream end unknown. If durability matters, add an application-level manifest or commit record with its own protocol and tests. Do framing on bytes with an explicit UTF-8 policy. UTF-8 continuation bytes cannot equal the ASCII RS byte, so scanning for 0x1E is unambiguous when the input is valid UTF-8; still validate UTF-8 for each record and do not silently replace invalid byte sequences. Preserve byte offsets for each RS and LF, raw record bytes where retention is allowed, parser result, and any recovery action. A normalized String alone loses exact source offsets and can hide encoding replacement. Bound the whole stream as well as each record. Set maximum file/stream bytes, record bytes, records, nesting depth, collection sizes, processing time, and cancellation behavior. A single huge record can consume memory before JSON schema validation begins. Use the selected parser’s streaming or size-limit interfaces when available; otherwise stage bounded record bytes and reject before materializing an oversized object. Do not invent guarantees from a parser option whose exact library/version has not been tested. Syntax, schema, and action are separate stages. A complete JSON text can still lack fields, contain wrong types, repeat object member names with implementation-dependent results, or carry values outside the app’s rules. Validate every accepted record against a versioned schema and current state. A record that proposes a function call, file change, or preference update is not authorized merely because its JSON parsed. Never execute partial parse events from a record that later fails, and commit record-level effects only after the chosen integrity and validation boundary. Test the exact writer and reader with objects, arrays, strings, numbers, booleans, null, embedded escaped newlines, escaped RS, multi-byte UTF-8, empty input, consecutive RS, malformed JSON between good records, truncation inside an object, truncation after a top-level number or literal, invalid UTF-8, oversized records, and cancellation at each byte boundary. Assert record byte offsets, continuation after errors, scalar truncation warnings, preserved good records, and an explicit uncertain-tail outcome. Avoid using sensitive real prompts or model outputs in fixtures. Do not confuse RFC 7464 JSON Text Sequences with newline-delimited JSON conventions, a single JSON array, or model token streaming. A model’s partial text chunks are runtime events with request identity and cancellation semantics; JSON-seq RS/LF are a storage or interchange framing contract. One does not imply the other, and a sequence parser does not provide model-output quality, transaction durability, ordering guarantees under missing records, or cryptographic integrity. This method does not claim Cove persists JSON-seq data.

A process stops while appending local JSON records

An Android worker appends small result objects to a local sequence. The process is killed after writing part of a record, then a later run encounters one malformed fragment followed by a valid RS-framed record. The reader must recover without claiming the final history is complete.

Decision

Use the RFC framing bytes, isolate parse status and byte offsets per record, continue only at a verified next RS, and mark the observed tail as uncertain unless an application-level completion marker proves otherwise.

Expected outcome

Earlier valid records remain individually reviewable, malformed records are reported, and the file is not represented as an atomic or complete transaction log.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Pinned UTF-8 RS/LF writer and parser
• Malformed-middle and scalar-truncation fixtures
• Byte offsets, limits, and completion-state outcomes

A completed byte stream is not enough. Final placement, integrity, runtime load, and recovery must also pass.

Implementation workflow

1. Name the sequence format and byte contract. Declare RFC 7464 rather than an informal newline convention. Encode each JSON text as UTF-8 with RS before and LF after; define accepted top-level values and preserve source byte offsets.
2. Frame before parsing and validate encoding. Scan the byte stream for raw RS boundaries, validate each candidate as UTF-8, and then pass bounded record bytes to the JSON parser. Do not split pretty-printed content by line or normalize invalid UTF-8 into replacement characters.
3. Recover per record without claiming completeness. On malformed JSON, record the byte range and error, then resume at the next RS under RFC parser behavior. Detect possibly truncated top-level numbers/true/false/null using the required whitespace boundary; distinguish recovered records from missing or unknown tail records.
4. Validate schema before any side effect. Apply a versioned record schema, size and range rules, and current-state authorization only after a complete JSON text parses. Do not execute partial events or treat framing as transaction commit, signature, ordering, or consent.
5. Exercise writer, reader, crashes, and bounds. Test valid value types, escaped newline/RS, multibyte UTF-8, consecutive separators, malformed middle records, truncation at each framing point, huge records, cancellation, and restart. Assert byte offsets, recovery decisions, and explicit uncertain-tail states.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• The producer follows UTF-8 RS + JSON text + LF framing, and the parser uses the declared RFC 7464 byte grammar rather than line splitting.
• Malformed middle records, consecutive RS bytes, invalid UTF-8, and truncated top-level numbers/literals have tested per-record results and offsets.
• A sequence without a completion marker is never claimed complete or atomic; a recoverable earlier record does not erase uncertainty about missing tail data.
• Record byte count, total stream size, record count, nesting, schema processing time, and cancellation have explicit limits and failure states.
• Parsing, schema validity, user authorization, and side effects remain separate; partial parse events cannot trigger actions.

Decision table

Failure patterns to prevent

• Splitting all records by newline
• Treating LF as a commit or end marker
• Executing partial records before schema validation

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Android privacy and security
• IETF RFC 7464 — JSON Text Sequences
• IETF RFC 8259 — The JSON Data Interchange Format
• IETF RFC 3629 — UTF-8, a transformation format of ISO 10646

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions