Production-scale multimodal retrieval-augmented generation for networkoperations: Evaluation and measurement pitfalls

Retrieval-augmented generation (RAG) grounds large language models for network operations, but published telecom RAG is evaluated offline on single, public, text-only corpora, leaving the production case unaddressed. We report a hybrid multimodal RAG platform in production at a global digital-infrastructure operator: dense-sparse retrieval with reciprocal rank fusion and cross-encoder reranking, 1536-dimensional multimodal embeddings, an authoritative asset-graph grounding layer, and asynchronous warm-standby geo-replication, over 334,818 vector points from eight sources of which 11.7% are images. We contribute a production-derived benchmark stratified by answerability, a context-aware LLM-judge protocol whose failure modes we measure directly, component ablations, and systems results.

We also report its most latency-sensitive consumer, an always-listening spoken-avatar assistant on two 2017-class edge GPUs that self-personalizes its own edge models. Our central finding concerns measurement. An answer-only LLM judge is systematically wrong on groundedness, scoring unanswerable questions above answerable ones.

A context-aware judge removes that shortcut but introduces another: a system that retrieves badly refuses rather than inventing, and a refusal is trivially well grounded, so judged groundedness runs backwards against gold-label retrieval quality (Spearman -0.89 across configurations spanning an elevenfold range in Recall@1). Groundedness cannot separate a correct refusal from a correct grounded answer, so refusal rates must accompany it. Rankings are also unstable across judge families (Kendall τ = 0.10), so we report the retrieval benefit we measure, +0.25 relevance over closed-book generation, as judge-specific.

Finally, a production corpus decays silently unless per-source ingestionfreshness is monitored, and wide-area round-trips are a small fraction of any interactive response.