Deepseek report on eval and extraction

Evaluation Framework for Accurate OCR/VLM Data Extraction in Financial Documents
Prepared for Accounting Team Proof-of-Concept (POC)

1. Introduction #

This report outlines evaluation techniques and novel strategies for ensuring high accuracy in structured data extraction from financial documents (invoices/expenses) using OCR/Vision-Language Models (VLMs). Given the critical nature of financial data, we prioritize methods that:

• Detect and correct extraction errors
• Flag low-confidence results
• Iteratively refine extractions
• Benchmark against traditional OCR & modern VLMs

We explore PyDantic Evals, Mozilla Transformer Lab, and novel multi-pass refinement techniques, alongside assessing whether fine-tuning a model is viable for this POC.

2. Evaluation Techniques for Structured Data Extraction #

Based on Transformer Lab’s Evaluation Methods, we adapt the following for financial document processing:

(A) Exact Match (EM) & Fuzzy Matching #

• Application: Compare extracted fields (invoice number, date, amount) against ground truth.
• Implementation:
  • Use PyDantic Evals to enforce schema validation (e.g., InvoiceSchema).
  • Apply fuzzy matching (Levenshtein distance) for minor OCR errors (e.g., "O" vs "0").

(B) Confidence Score Thresholding #

• Application: Flag low-confidence extractions (e.g., Gemini’s confidence_score < 0.8).
• Implementation:
  • Mozilla Transformer Lab’s "Uncertainty Estimation" module to identify unreliable predictions.
  • Route low-confidence extractions to a human review queue or secondary model (e.g., LayoutLMv3).

(C) Multi-Model Consensus Voting #

• Application: Run 2–3 models (Gemini, Mistral OCR, LayoutLM) and validate agreement.
• Implementation:
  • Use Transformer Lab’s "Ensemble Evaluation" to compare outputs.
  • Discrepancies trigger a re-extraction with stricter prompts (e.g., "Re-analyze with focus on amount totals").

(D) Template-Based Validation #

• Application: Enforce invoice-specific rules (e.g., total = subtotal + tax).
• Implementation:
  • PyDantic Evals to validate arithmetic/logical consistency.
  • Flag mismatches for reprocessing.

(E) Human-in-the-Loop (HITL) Sampling #

• Application: Randomly sample 5–10% of extractions for manual audit.
• Implementation:
  • Integrate Label Studio (open-source) for quick human verification.

3. Novel Strategies for Improving Extraction Accuracy #

(A) Multi-Pass Refinement Workflow #

1. First Pass: Fast extraction (Gemini 2.0 Flash for raw text).
2. Second Pass: Structured parsing (Mistral OCR + JSON schema).
3. Third Pass: Error correction (LayoutLMv3 for layout-aware validation).

Example: If the first pass misses a line item, the second pass cross-references bounding boxes, and the third pass validates totals.

(B) Hybrid OCR/VLM Pipelines #

• Toolkit: Combine traditional OCR (Tesseract) with VLMs (Gemini) for:
  • Text-heavy regions: Tesseract (higher precision for clean text).
  • Complex layouts: Gemini (context-aware extraction).

(C) Low-Confidence Auto-Correction #

• Implementation:
  • For low-confidence fields, use Mistral’s self-reflection ("Review this extraction: [text]. Is this correct? Fix if not.").
  • Pulse API (if budget allows) for automated template learning.

4. Should We Fine-Tune a Model? #

Pros: #

• Better accuracy for domain-specific invoices (e.g., healthcare vs. retail).
• Reduced reliance on prompt engineering.

Cons: #

• Cost: Requires ~1,000–5,000 labeled documents for decent results.
• Effort: Annotation + training pipeline setup (~2–4 weeks for POC).

Recommendation: #

• Start with zero-shot VLMs (Gemini/LayoutLM) and evaluate gaps.
• If errors are consistent (e.g., misreading vendor names), fine-tune LayoutLMv3 on 500–1,000 samples using Transformer Lab.

5. Tooling Recommendations #

6. Next Steps for POC #

1. Implement PyDantic Evals for schema validation.
2. Benchmark Gemini vs. Mistral OCR on 50 sample invoices.
3. Set up multi-pass refinement for low-confidence extractions.
4. Evaluate fine-tuning need after initial testing.

Budget-Friendly Priority: Start with multi-model consensus + HITL sampling before investing in fine-tuning.

Appendix: Relevant Tools from Hacker News

• OmniOCR: Strong multilingual support (benchmark if needed).
• olmOCR: Open-source, good for tables (test against invoices).
• JigsawStack vOCR: Claims superior handwriting support (verify).

Let me know if you'd like to dive deeper into any section!