FilingsIQ
Question answering over SEC 10-K filings that recomputes its own numbers and refuses to guess.
- Status
- Evaluated on a 40-question benchmark
- Role
- Solo — ingestion, retrieval, verification layer, evaluation
- Timeline
- 2026
At a glance
A QA system for SEC 10-K filings built around numerical accuracy and verifiability. Every answer cites source chunks, derived figures are recomputed in plain Python and checked, and the system declines to answer out-of-corpus questions. Runs entirely on a local GPU with zero paid API dependency.
- Pulls live 10-K filings from EDGAR and parses tables atomically, preserving row labels and multi-year columns.
- Fiscal-year reconciliation handles companies whose financial year is not the calendar year.
- Derived metrics — growth rates, margins, deltas — are recomputed in plain Python and validated, not trusted from the model.
- A retry loop regenerates failed claims once with feedback; out-of-corpus questions are confidently refused.
- The README documents real bugs found in testing: hash collisions silencing data, XBRL fiscal-year ambiguity, small-model subtotal confusion.
Key numbers
91.7%
Accuracy (answered)
40-question extractive benchmark, 3 companies, 2 years
8.3%
Hallucination rate
The other side of the same number
60%
Coverage
Deliberately low — strict refusal over guessing
~4.6 s avg
Latency
~12 s at p95
Tech stack
- Model
- Qwen2.5-3B-InstructLocal GPU by defaultbge-small-en-v1.5 embeddings
- Retrieval
- SQLite + BM25 lexical searchDense embeddingsHard metadata filtering by company & year
- Parsing
- Custom HTML → markdownTable-atomic parsingFiscal-year reconciliation
- Verification
- Provenance checks vs cited chunksArithmetic recomputation in PythonRefusal calibration
The problem
Financial QA is unforgiving. A RAG system that is 95% right on prose can still be dangerously wrong on a number, and a plausible-sounding wrong figure from a 10-K is worse than 'I don't know'.
FilingsIQ optimises for being right or silent, in that order.
Ingestion and retrieval
Filings are pulled live from EDGAR and converted from HTML to markdown with a custom parser that keeps tables atomic — row labels and multi-year columns stay intact. Retrieval combines BM25 lexical search with dense embeddings over SQLite, and hard metadata filters restrict to the right company and year before ranking.
The verification layer
This is the part that matters. Every claim is checked against the specific chunks it cites. Every derived number — a growth rate, a margin, a year-over-year delta — is recomputed in plain Python and compared. If a claim fails verification, the system regenerates it once with feedback; if it still fails, it is dropped. Questions outside the corpus are refused rather than answered.
Results and honesty
On a 40-question extractive benchmark across three companies and two years: 91.7% accuracy on answered questions, 8.3% hallucination, ~4.6 s average latency. Coverage is only 60% — a deliberate trade, because the strict refusal threshold prioritises correctness. The README transparently lists the bugs found during testing and how they were fixed.
Want the details?
The full source, READMEs, evaluation harnesses, and results are on GitHub.
github.com/dariyaDeepanshu/filingsiq