Rules change underneath you
SEBI regularly amends circulars and folds them into master circulars. An answer based on a superseded circular can look right and still be non-compliant.
AINPM.in / Case study
MVP · Live · Open source
SEBI Circular RAG: trustworthy answers over Indian securities regulation
Compliance and product teams at AMCs, brokers and fintechs work from hundreds of SEBI circulars, many of them amended, consolidated into master circulars, or superseded. I built an AI-native assistant that finds the right circular, cites it, tells you whether it's still in force, and says so when it doesn't know.
- Role: product owner & builder
- Jul–Sep 2026
- Python · FAISS · BM25 · BGE-M3
- Gradio on Hugging Face
- Data: CC-BY-4.0
1,490circulars · 83,752 chunks
0.943Recall@10 (full pipeline)
0.881Citation recall
0.981Abstention accuracy
Full evaluation on 15 Aug 2026 over golden_v7 (260 adjudicated queries), measured when the corpus had 728 circulars. On 15 Sep 2026 the pipeline was re-run on the 1,490-circular index and still cleared all six quality gates.
01 · The problem
In regulation, a confident wrong answer is worse than no answer
"What applied then?" matters
Audits, disputes and product reviews often need the rule as it stood on a specific date, not today's version.
Generic chatbots guess
General LLMs hallucinate circular numbers and rarely admit gaps. Every answer has to be checkable against the source.
Who it's for
Compliance officers and product managers at mutual fund AMCs, brokers, depositories, RTAs and fintechs; RegTech builders; researchers working on Indian legal NLP.
Job to be done
"Help me find the circular that governs this, show me the exact text, tell me whether it still applies, and don't make anything up."
02 · Product principles
Trust was the product requirement, not a feature
These six rules shaped every design and model decision.
Cite everything
Each claim links to a numbered circular and the exact passage it came from.
Abstain when unsure
A calibrated gate refuses out-of-scope or weakly supported questions, such as RBI or GST topics, instead of stretching SEBI text to fit.
Know what's superseded
A lineage graph of 4,577 supersession and amendment links demotes superseded circulars and names the one that replaced them.
Answer "as of" a date
Pick a date and retrieval scores against the rules in force then, excluding circulars issued later.
Check its own work
A faithfulness check flags any cited circular that wasn't in the retrieved context and adds a caution.
Stay honest about limits
Not legal advice, not SEBI-endorsed, and the coverage caveats are published with the data.
03 · How it works
From SEBI PDFs to a grounded, checkable answer
Ingest
Scrape SEBI circulars and master circulars, extract PDF text, and validate corpus integrity.
Section-aware chunking
Split along the circular's own structure; prefix each chunk with circular number, subject and section.
Lineage graph
Mine references to build supersedes / amends edges and link circulars to the regulations they rely on.
Hybrid retrieval
BGE-M3 dense vectors (FAISS) plus BM25 keywords, fused with reciprocal rank fusion.
Rerank
A bge-reranker-v2-m3 cross-encoder re-scores the top 50 candidates.
Supersession & as-of
Demote superseded circulars; for dated questions, filter to what was in force on that date.
Abstention gate
Topic-similarity thresholds decide whether there's enough evidence to answer at all.
Grounded generation
A small Qwen2.5 model writes the answer using only the retrieved passages.
Citation filter & faithfulness
Keep only citations the answer actually uses; flag any unsupported ones.
Retrieval & generationTrust controls
Runs local-first on Apple Silicon (MLX), with a FastAPI service and API keys. The public demo runs the same retrieval on a free Hugging Face CPU Space using a prebuilt index, so results match the local system.
04 · Evals as the spec
260 adjudicated questions decide what ships
Instead of a PRD with "should be accurate", the spec is a golden set. It is stratified by the ways regulatory search fails, reviewed row by row, and wired into gates that block regressions.
- 40 title lookups
- 60 paraphrased body text
- 30 numbers & tables
- 40 lineage / supersession
- 20 multi-hop
- 20 repealed legal basis
- 40 hard negatives
- 10 far negatives
| Metric | Result | What it tells a PM |
|---|---|---|
| Recall@10 | 0.943 | The right circular is in the top 10 for 94% of answerable questions. |
| Context recall | 0.916 | The passages sent to the model contain the needed evidence. |
| nDCG@10 | 0.697 | Ranking quality: how high the right circular appears. |
| Citation recall | 0.881 | Answers cite the circulars they should. |
| Citation precision | 0.194 | Still extra citations per answer. The known weak spot and the next target. |
| Abstention accuracy | 0.981 | Correctly refuses or answers across both answerable and out-of-scope questions. |
| Warm answer latency | ~2.1 s | Local generation on Apple Silicon. |
Source: project status log, full evaluation 15 Aug 2026 (728-circular corpus). Faithfulness is checked per answer but not yet reported as an aggregate metric. The test suite had 1,131 passing tests on 16 Sep 2026.
05 · PM decisions & trade-offs
What I shipped, and what I deliberately didn't
Every change was an experiment measured against the golden set. Saying no to plausible ideas was as important as saying yes.
| Decision | Evidence | Call |
|---|---|---|
| Enrich chunks with circular number, subject & section | Citation precision 0.60 → 0.74 on the earlier eval set | Shipped |
| Retrieve 10 contexts instead of 5 | Citation recall 0.772 → 0.888, but precision 0.177 → 0.119. Missing a governing circular costs more than an extra citation. | Shipped |
| Post-hoc citation filter (cross-encoder, margin 0.35) | Prompting the 1.5B model to cite selectively was a no-op. It emitted no parseable citations. A model-agnostic filter raised citation precision 88% at the chosen margin. | Shipped |
| Fix mislabeled eval rows before tuning | Re-labeling mislabeled abstain rows moved abstention accuracy 0.849 → 0.973. The model was right; the test was wrong. | Shipped |
| HyDE (hypothetical-answer retrieval) | Recall@10 −2.3 pp (not significant) at 41× latency | Rejected |
| SPLADE sparse retrieval leg | Looked promising (nDCG +0.029) but failed a pre-registered held-out confirmation | Rejected |
| Qwen3-Reranker-0.6B instead of bge-reranker | Lower AUROC (0.799 vs 0.812), worse precision, 2× latency | Rejected |
| Fine-tuning BGE-M3 on SEBI text | No significant nDCG@10 gain on any arm, so the base model stays | Rejected |
The PM lesson: the biggest quality jump came from fixing evaluation data, not from a new model. In AI-native products, the eval set is the product spec, so it deserves the same review rigour as code.
06 · Live demo
Try it: ask a question about SEBI circulars
Example questions: "How many nominees can an investor add to a demat account or mutual fund folio?" · "Which circular governs mutual fund nominations, and is it still in force?" · Try an out-of-scope question like GST e-invoicing to see it abstain.
Modes: rag (full answer with citations), retrieval_only (no LLM, still shows citations and lineage), and an optional as-of date. Answers are informational, not legal advice. Always verify on sebi.gov.in.
07 · Open data
Published so others can build and benchmark
The corpus, annotations and a ready-to-use index are public on Hugging Face. Annotations are CC-BY-4.0; the underlying SEBI circulars are government works with source links on every record.
sebi-circulars dataset ↗
Snapshot v2026.08 · 728 circulars (2010–2026), including SEBI's master circulars.
- 728 corpus
- 78,585 chunks
- 4,577 lineage edges
- 2,769 supersession pairs
- 8,903 normalized citations
- 56 eval queries
sebi-circulars-index ↗
The prebuilt index the live demo uses: 1,490 circulars, 83,752 chunks, 1024-dim BGE-M3 vectors in FAISS, a BM25 index and the lineage graph.
- 1,490 circulars
- 83,752 chunks
- FAISS + BM25
- lineage.json
08 · What's next
Roadmap
Raise citation precision
Cut extra citations without losing recall. This is the clearest quality gap today.
Measure faithfulness end-to-end
Report faithfulness as a gated aggregate metric, not just a per-answer flag.
Refresh the public dataset
Publish the 1,490-circular snapshot and expand the adjudicated golden set.