AskWarehouse
A text-to-SQL agent that self-corrects, asks for clarification, and refuses to guess.
- Status
- Evaluated on 109 gold questions
- Role
- Solo — agent, retrieval, safety layer, evaluation
- Timeline
- 2026
At a glance
A natural-language analytics agent over a 16M-row synthetic warehouse. It retrieves only the schema it needs, self-critiques its SQL, repairs errors using database feedback, asks for clarification on genuinely ambiguous questions, and returns charts alongside the query. Safety is enforced in code, not prompts.
- Schema retrieval via embeddings means the model sees the tables it needs, not the whole warehouse.
- A self-critique step catches SQL errors before execution; a repair loop reuses the database's own error messages for up to three retries.
- Detects genuinely ambiguous questions and asks for clarification instead of confidently answering the wrong one.
- Safety is code: read-only connections at the storage layer, AST guards that reject non-SELECT and PII access, forced LIMITs and statement timeouts.
- Ships a failure taxonomy and ablation studies showing where each accuracy gain comes from.
Key numbers
64.2%
Execution accuracy
On 109 gold questions over a 16M-row warehouse
90.8%
Valid SQL
Syntactically and semantically executable
95%
Ambiguity detection
On a 40-question ambiguity eval
1.6–1.7 s
Median latency
Single-call configurations
Tech stack
- Model
- Qwen2.5-Coder-7B-Instruct4-bit via transformers + bitsandbytesOptional OpenAI / Anthropic providers
- Data
- DuckDB (read-only)~16M-row synthetic warehousedbt semantic layer
- Retrieval
- Embeddings-based schema retrievalFuzzy-matched value index
- Safety
- sqlglot AST parsingEXPLAIN cost checksPII deny-listsForced LIMIT + timeoutsAudit logging
The problem
Text-to-SQL demos work on toy schemas and fall apart on a real warehouse: too many tables to fit in context, ambiguous business terms, and the constant risk that a generated query does something it should not.
AskWarehouse is built around those three failure modes.
How it answers a question
Relevant tables are retrieved with embeddings so the prompt stays small. The model drafts SQL, critiques its own draft, and — if execution fails — enters a repair loop that feeds the database error back in, up to three attempts. If the question is genuinely ambiguous, the agent stops and asks rather than guessing.
Results come back as a chart next to the generated SQL, so the answer is always inspectable.
Safety as code
Database connections are read-only at the storage layer. Every query is parsed into an AST with sqlglot and rejected if it is not a SELECT or if it touches PII deny-listed columns. EXPLAIN cost checks block expensive scans, LIMIT clauses are forced, statements time out, and everything is written to an audit log.
Evaluation
On 109 gold questions over the 16M-row warehouse: 64.2% execution accuracy and 90.8% valid SQL, at 1.6–1.7 s median latency for single-call configurations. On a separate 40-question ambiguity set the agent correctly identifies ambiguous questions 95% of the time. The repository includes a failure taxonomy and ablations attributing each improvement to its cause.
Want the details?
The full source, READMEs, evaluation harnesses, and results are on GitHub.
github.com/dariyaDeepanshu/AskWarehouse