PageIndex as RAG Alternative: A Comparative Evaluation of Vectorless Document Retrieval

I spent the past week building an evaluation harness to compare PageIndex—VectifyAI's "vectorless, reasoning-based RAG" system—against a traditional RAG pipeline on commercial loan documents. The hypothesis was intuitive: tree-structured retrieval that reasons about document organization should outperform similarity search on complex queries, even if it underperforms on simple lookups.

The hypothesis was wrong.

The Setup#

Traditional pipeline: PyMuPDF4LLM → 1,000-character chunks with 200 overlap → Voyage 3 Large embeddings → FAISS top-5 → Claude Sonnet 4

PageIndex pipeline: PDF → PageIndex Cloud API (tree generation) → reasoning-based traversal → Claude generation (server-side)

Same generation model. Same questions. Same ground truth. The only variable is the retrieval mechanism.

I ran 36 questions across commercial loan documents—credit memos, appraisals, the structurally heterogeneous stuff that makes RAG hard.

The Results#

Traditional RAG won on every metric that matters. Not by a little—by 25 percentage points on accuracy, with zero hallucinations versus one-in-four claims unsupported.

The only metric favoring PageIndex is raw cost per query ($0.0043 vs $0.0065), an advantage that evaporates when you normalize by quality.

Why the Gap?#

Recall drives accuracy. Traditional retrieval's top-5 chunks achieved 80% recall on source pages. PageIndex's tree traversal achieved 58%. When the generation model has the relevant context, it produces correct answers. When it doesn't, it either fails or hallucinates. PageIndex's "targeted" retrieval is too targeted — it misses content that broad similarity search reliably surfaces.

Latency is architectural. Vector search is sub-millisecond. Tree traversal requires sequential LLM reasoning at each level, plus polling overhead. The 5.8x latency gap isn't fixable with optimization; it's inherent to the approach.

Multi-location synthesis breaks both. Neither pipeline handled questions requiring information from multiple document sections (14% correct each). This is the shared failure mode that better architectures — iterative retrieval, query decomposition, graph-based navigation — might address.

But What About Mafin 2.5's 98.7%?#

VectifyAI claims their Mafin 2.5 system achieves 98.7% on FinanceBench. Three factors explain the gap:

System scope differs. Mafin 2.5 is a tuned application layer built on PageIndex, not the raw API. It likely includes orchestration, prompt engineering, and multi-step reasoning that the Cloud API doesn't expose.

Evaluation methodology differs. Their benchmark includes human re-annotation for ambiguous questions. The FinanceBench paper shows GPT-4-Turbo with retrieval at 19% under strict scoring; my traditional pipeline hit 47%, exactly matching published baselines.

Document characteristics differ. FinanceBench uses standardized SEC filings with predictable structure (Item 1, Item 1A, Item 7). Commercial loan documents are structurally heterogeneous. Tree traversal may work better on documents with consistent hierarchies.

None of this invalidates their claims or my findings. They're answering different questions.

When to Use What#

Favor traditional RAG when:

• Accuracy matters and errors carry risk
• You need sub-10-second responses
• Documents have heterogeneous structure
• Hallucination is unacceptable

Consider PageIndex when:

• Documents have highly consistent structure (standardized forms, regulatory filings)
• The full Mafin 2.5 system is available for your domain
• Latency tolerance exceeds 30 seconds
• Eliminating vector infrastructure is a priority and you'll accept the accuracy tradeoff

Consider neither when:

• Questions require synthesis across non-adjacent document sections
• You need explicit source citations in every answer

The Bottom Line#

PageIndex's architectural premise is sound: document structure carries information that chunking destroys, and reasoning about where to look should beat geometric similarity on complex queries. But the implementation—at least as exposed through the Cloud API—doesn't deliver on that promise. For commercial loan documents, traditional RAG is faster, more accurate, cheaper per correct answer, and produces zero hallucinations.

The evaluation harness and full methodology are available in the complete paper. If you're evaluating PageIndex for your own use case, I'd encourage you to run a similar comparison on your documents before committing.