SEBI Circular RAG: the product work behind it
These five artifacts show how SEBI Circular RAG was framed, measured and decided, not just how it was coded. Each one summarises the project's own documents and links to them, so you can check every line. Read the case study.
What problem does it solve, and for whom?
Problem
SEBI amends circulars and folds them into master circulars. An answer built on a superseded circular can look right and still be non-compliant, and generic chatbots invent circular numbers.
Users
Compliance officers and product managers at mutual fund AMCs, brokers, depositories, RTAs and fintechs; RegTech builders; Indian legal-NLP researchers.
Job to be done
"Find the circular that governs this, show the exact text, say whether it still applies, and make nothing up."
Constraints
- Local-first on Apple silicon; official SEBI publications are the only authority
- When evidence is insufficient, answer "I don’t know based on the available evidence"
- One pinned benchmark runtime, model versions and seeds, so results are reproducible
- Optimise only validated stages, and only for a measurable benefit (default bar: 10%)
Non-goals
- Legal advice or unsupported legal conclusions
- API-only proprietary models that would send SEBI queries to a third party
- Targeting the Apple Neural Engine (declined in ADR-003)
Source: project_context.md §1, §8, §9 ↗
How is quality defined and gated?
A 260-question adjudicated golden set, stratified by failure type, with six release floors. A change that drops any metric below its floor does not ship.
Test set
- golden_v7: 260 adjudicated questions; 41 should be declined; 15 ask for the rule as of a date
- Strata: 40 title lookups, 60 paraphrases, 30 numeric or table, 40 supersession lineage, 20 multi-hop, 20 repealed basis, 40 hard negatives, 10 far negatives
- Adjudicated with a local 35B model, with agreement checks and escalations
Metrics
| Metric | Measured | Bar |
|---|---|---|
| Abstention accuracy | 0.981 | 0.937 |
| Recall@10 | 0.943 | 0.840 |
| Context recall | 0.916 | 0.819 |
| Citation recall | 0.881 | 0.735 |
| nDCG@10 | 0.697 | 0.593 |
| Citation precision · Known weak spot | 0.194 | 0.147 |
What does success mean, and what must not break?
My framing of metrics the project already measures.
Questions answered correctly with a checkable citation, or honestly declined
In regulation a confident wrong answer costs more than no answer, so "honestly declined" counts as success.
- Find the right circular: Recall@10 0.943 · nDCG@10 0.697
- Use the right passage: Context recall 0.916
- Cite what was used: Citation recall 0.881
- Decline when evidence is missing: Abstention accuracy 0.981
- Don’t over-cite: Citation precision 0.194 (weak)
- Never cite a forbidden source: must_not_cite violation rate
What was decided, and why?
| Decision | Date | Status | What and why |
|---|---|---|---|
| D1 Hybrid retrieval is mandatory ↗ | 2 Jul 2026 | Amended | Dense (FAISS) plus sparse (BM25) retrieval fused with RRF; a new dense store only on 10% or better evidence. Why: The retrieval stage is fixed architecture; the model inside it is replaceable only on measured evidence. |
| ADR-002 Certainty bands, not probabilities ↗ | 2 Jul 2026 | Accepted | Every answer carries high, medium or low certainty and a reason when it declines; a labelled low-confidence draft only on explicit opt-in. Why: Raw model scores are not certainty (a hard negative scored 0.985), and 56 golden items were too few to fit a calibrator. |
| ADR-003 Apple Neural Engine declined ↗ | 22 Jul 2026 | Declined | Keep generation on MLX and embeddings and reranking on MPS/MLX. Why: The Neural Engine trades speed for energy efficiency and needs a Core ML conversion; a plugged-in server pipeline needs throughput, where MLX on the GPU wins. |
| D2 Proprietary API embedder rejected ↗ | 20 Aug 2026 | Rejected | Voyage models rejected as candidates, regardless of benchmark strength. Why: API-only service conflicts with local-first and would send SEBI queries to a third party. |
| ADR-004 New reranker adopted below the 10% bar ↗ | 24 Aug 2026 | Adopted | jina-reranker-v3 adopted by explicit owner override: recall@10 +2.42% and nDCG@10 +6.76% on SEBI data, no regression. Why: A consistent, non-regressive gain; the 10% bar was self-imposed, and the override and its reasoning are recorded. |
What shipped, what's weak, and what's next?
Status: Live demo on Hugging Face; open code and dataset
Shipped
- Index of 1,490 circulars and 83,752 chunks (15 Sep 2026)
- Answers with numbered citations, supersession status and successor circulars
- Certainty block and decline reason on every response
- Open dataset of SEBI circulars on Hugging Face (CC BY 4.0)
Known limits
- Citation precision 0.194: answers cite more passages than they need
- The armed gate file has no stack fingerprint, so drift is flagged as "unverifiable" until floors are re-derived
- English circulars only
Next
- Add a stack fingerprint to the gate and re-derive floors on the current chunker
- Improve citation precision without losing citation recall
Source: Evaluation log (status.md) ↗ · All product artifacts
Open it yourself: Live demo ↗ · GitHub ↗
What else should you know about these artifacts?
How is SEBI Circular RAG evaluated?
A 260-question adjudicated golden set, stratified by failure type, with six release floors. A change that drops any metric below its floor does not ship. Floors belong to the stack, not the corpus: changing the generator, embedder, corpus size, chunker version or citation margin invalidates them, and they must be re-derived before comparing anything.
Which decisions shaped SEBI Circular RAG?
Hybrid retrieval is mandatory (amended, 2 Jul 2026): The retrieval stage is fixed architecture; the model inside it is replaceable only on measured evidence. Certainty bands, not probabilities (accepted, 2 Jul 2026): Raw model scores are not certainty (a hard negative scored 0.985), and 56 golden items were too few to fit a calibrator. Apple Neural Engine declined (declined, 22 Jul 2026): The Neural Engine trades speed for energy efficiency and needs a Core ML conversion; a plugged-in server pipeline needs throughput, where MLX on the GPU wins. Proprietary API embedder rejected (rejected, 20 Aug 2026): API-only service conflicts with local-first and would send SEBI queries to a third party. New reranker adopted below the 10% bar (adopted, 24 Aug 2026): A consistent, non-regressive gain; the 10% bar was self-imposed, and the override and its reasoning are recorded.
What are the known limits of SEBI Circular RAG?
Citation precision 0.194: answers cite more passages than they need. The armed gate file has no stack fingerprint, so drift is flagged as "unverifiable" until floors are re-derived. English circulars only.