Add inline text ingestion support

[jioffe502]

Summary

• add texts() for ingesting raw Python strings without temporary files
• allow inline text to be ingested alongside files and buffers by reusing the existing modality planner and extraction branches
• reuse the normal text split, embed, VDB, and service upload paths across in-process, batch, and service modes
• preserve deterministic inline source identities, duplicates, validation, and empty-input behavior

Closes #2345.

Validation

• 295 focused ingestion, manifest, splitter, actor, and service tests passed
• local-Ray mixed file and inline-text ingestion test passed
• Flake8 and diff whitespace checks passed on the changed files

[greptile-apps]

Greptile Summary

This PR adds texts() support to all three run modes (inprocess, batch, service), allowing callers to pass raw Python strings directly into the text-split → embed → VDB pipeline without creating temporary files. Each inline string gets a deterministic inline:// source identifier, flows through the normal TxtSplitCPUActor, and can be mixed with file and buffer sources via the existing branch-extraction planner.

• New texts() / extract_txt() methods added to ingestor base class and implemented in GraphIngestor and ServiceIngestor; blank-only corpora short-circuit before touching Ray or the service.
• inline_text.py introduces normalize_inline_texts, inline_text_source_id, and is_inline_text_source as shared identity utilities; txt_bytes_to_chunks_df is refactored to delegate to the new text_to_chunks_df to avoid code duplication.
• Service mode encodes inline strings as InMemoryUpload (UTF-8 bytes + .txt classification filename), extending RetrieverServiceClient to accept Path | InMemoryUpload; the ingest router is updated to fast-path TEXT/HTML files to REALTIME without PDF page-counting.

Confidence Score: 5/5

Safe to merge. All three run modes produce correct results; blank-only corpora short-circuit cleanly; branch execution and schema unification across file and inline paths are coherent. The silent exception-swallowing issue from the prior review is now addressed with a logged warning.

The implementation is thorough and well-tested. The two inline observations — bytes sneaking through the Sequence guard and _has_mixed_inline_sources() returning True for an empty list — are edge-case quality issues, not runtime failures under normal use.

Files Needing Attention: inline_text.py (bytes guard) and the _has_mixed_inline_sources helpers in both graph_ingestor.py and service_ingestor.py (empty-list treatment).

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/common/inline_text.py	New module providing shared inline-text identity utilities. Well-typed, correctly handles scalars/sequences, but silently accepts bytes through the Sequence duck-type check, producing a confusing 'got int' error instead of 'got bytes'.
nemo_retriever/src/nemo_retriever/common/modality/txt/split.py	Refactored to extract text_to_chunks_df as the canonical single-pass splitter; txt_file_to_chunks_df and txt_bytes_to_chunks_df now delegate to it. Adds empty_text_chunks_df() to unify the empty-result schema (now includes the previously-missing content column). Well-documented.
nemo_retriever/src/nemo_retriever/ingestor/graph_ingestor.py	Core implementation of texts() for library mode. Blank-only early-exit, mixed-source branch detection, and inline dataset construction look correct. The _has_mixed_inline_sources() check treats _inline_texts=[] as active, which may surprise callers who pass a dynamically empty list.
nemo_retriever/src/nemo_retriever/ingestor/branch_extraction.py	Cleanly extended to split branch inputs into file paths and inline rows. Batch path creates two datasets per branch when both are present; they share the same output schema after TxtSplitCPUActor so union is safe.
nemo_retriever/src/nemo_retriever/service/service_ingestor.py	Service-mode inline text support via InMemoryUpload, blank-only short-circuit, and auto-mode routing for mixed sources. _collect_inputs return type is now list[UploadInput] (was list[Path]). Early-exit return tuples for return_failures/return_traces correctly match the normal-path contract.
nemo_retriever/src/nemo_retriever/service/client.py	Introduces InMemoryUpload NamedTuple and UploadInput union type; _upload_one / ingest_documents / aingest_documents_stream updated uniformly. Backward-compatible (existing Path usage unchanged).
nemo_retriever/src/nemo_retriever/operators/extract/txt/ray_data.py	Adds the text_to_chunks_df branch for inline-text rows alongside the existing bytes branch. Exception handler now logs source path with exc_info=True (previously flagged in prior review, now addressed). preprocess returns empty_text_chunks_df() for schema consistency.
nemo_retriever/src/nemo_retriever/service/routers/ingest.py	Routing change: TEXT and HTML file categories now fast-path to REALTIME pool (same as IMAGE), avoiding unnecessary PDF page-count parsing. Comment updated to match.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["User calls .texts(['a', 'b'])"] --> B{Any text strip?}
    B -- "No (all blank,\nno files/buffers)" --> C["Short-circuit:\nreturn empty_text_chunks_df()"]
    B -- "Yes or mixed\nwith files" --> D{Has mixed\ninline + files?}
    D -- No --> E["_execute_single_graph\n(inline_text_dataframe\nor inline_text_dataset)"]
    D -- Yes --> F["Build manifest\nplan_extraction_branches()"]
    F --> G["ExtractionBranchExecutor\n_partition_branch_inputs()"]
    G --> H["File paths -> build_dataset(paths)"]
    G --> I["Inline rows -> build_dataset(ray.data.from_items)"]
    H --> J["TxtSplitCPUActor\ntxt_bytes_to_chunks_df()"]
    I --> K["TxtSplitCPUActor\ntext_to_chunks_df()"]
    E --> K
    J --> L["normalize_ray_branch_datasets\nunion + to_pandas()"]
    K --> L
    L --> M["Embed -> VDB -> result DataFrame"]

Loading

_{Reviews (10): Last reviewed commit: "Merge branch 'main' into jioffe502/inlin..." | Re-trigger Greptile}