Rules change underneath you
SEBI regularly amends circulars and folds them into master circulars. An answer based on a superseded circular can look right and still be non-compliant.
SEBI Circular RAG is an open-source assistant that answers questions over 1,490 SEBI circulars with citations, supersession checks and abstention. Compliance and product teams at AMCs, brokers and fintechs work from hundreds of SEBI circulars, many of them amended, consolidated into master circulars, or superseded. I built an AI-native assistant that finds the right circular, cites it, tells you whether it's still in force, and says so when it doesn't know.

Tick = release floor. Every metric clears its floor; citation precision is the known weak spot. Source: Evaluation log ↗
In regulation, a confident wrong answer is worse than no answer.
SEBI regularly amends circulars and folds them into master circulars. An answer based on a superseded circular can look right and still be non-compliant.
Audits, disputes and product reviews often need the rule as it stood on a specific date, not today's version.
General LLMs hallucinate circular numbers and rarely admit gaps. Every answer has to be checkable against the source.
Compliance officers and product managers at mutual fund AMCs, brokers, depositories, RTAs and fintechs; RegTech builders; researchers working on Indian legal NLP.
"Help me find the circular that governs this, show me the exact text, tell me whether it still applies, and don't make anything up."
Trust was the product requirement, not a feature. These six rules shaped every design and model decision.
Each claim links to a numbered circular and the exact passage it came from.
A calibrated gate refuses out-of-scope or weakly supported questions, such as RBI or GST topics, instead of stretching SEBI text to fit.
A lineage graph of 4,577 supersession and amendment links demotes superseded circulars and names the one that replaced them.
Pick a date and retrieval scores against the rules in force then, excluding circulars issued later.
A faithfulness check flags any cited circular that wasn't in the retrieved context and adds a caution.
Not legal advice, not SEBI-endorsed, and the coverage caveats are published with the data.
Nine steps take a SEBI PDF to a grounded, checkable answer.
Scrape SEBI circulars and master circulars, extract PDF text, and validate corpus integrity.
Split along the circular's own structure; prefix each chunk with circular number, subject and section.
Mine references to build supersedes / amends edges and link circulars to the regulations they rely on.
BGE-M3 dense vectors (FAISS) plus BM25 keywords, fused with reciprocal rank fusion.
A bge-reranker-v2-m3 cross-encoder re-scores the top 50 candidates.
Demote superseded circulars; for dated questions, filter to what was in force on that date.
Topic-similarity thresholds decide whether there's enough evidence to answer at all.
A small Qwen2.5 model writes the answer using only the retrieved passages.
Keep only citations the answer actually uses; flag any unsupported ones.
Runs local-first on Apple Silicon (MLX), with a FastAPI service and API keys. The public demo runs the same retrieval on a free Hugging Face CPU Space using a prebuilt index, so results match the local system.
260 adjudicated questions decide what ships. Instead of a PRD with "should be accurate", the spec is a golden set. It is stratified by the ways regulatory search fails, reviewed row by row, and wired into gates that block regressions.
| Metric | Result | What it tells a PM |
|---|---|---|
| Recall@10 | 0.943 | The right circular is in the top 10 for 94% of answerable questions. |
| Context recall | 0.916 | The passages sent to the model contain the needed evidence. |
| nDCG@10 | 0.697 | Ranking quality: how high the right circular appears. |
| Citation recall | 0.881 | Answers cite the circulars they should. |
| Citation precision | 0.194 | Still extra citations per answer. The known weak spot and the next target. |
| Abstention accuracy | 0.981 | Correctly refuses or answers across both answerable and out-of-scope questions. |
| Warm answer latency | ~2.1 s | Local generation on Apple Silicon. |
Source: project status log, full evaluation 15 Aug 2026 (728-circular corpus). Faithfulness is checked per answer but not yet reported as an aggregate metric. The test suite had 1,131 passing tests on 16 Sep 2026.
Every change was an experiment measured against the golden set. Saying no to plausible ideas was as important as saying yes.
| Decision | Evidence | Call |
|---|---|---|
| Enrich chunks with circular number, subject & section | Citation precision 0.60 → 0.74 on the earlier eval set | Shipped |
| Retrieve 10 contexts instead of 5 | Citation recall 0.772 → 0.888, but precision 0.177 → 0.119. Missing a governing circular costs more than an extra citation. | Shipped |
| Post-hoc citation filter (cross-encoder, margin 0.35) | Prompting the 1.5B model to cite selectively was a no-op. It emitted no parseable citations. A model-agnostic filter raised citation precision 88% at the chosen margin. | Shipped |
| Fix mislabeled eval rows before tuning | Re-labeling mislabeled abstain rows moved abstention accuracy 0.849 → 0.973. The model was right; the test was wrong. | Shipped |
| HyDE (hypothetical-answer retrieval) | Recall@10 −2.3 pp (not significant) at 41× latency | Rejected |
| SPLADE sparse retrieval leg | Looked promising (nDCG +0.029) but failed a pre-registered held-out confirmation | Rejected |
| Qwen3-Reranker-0.6B instead of bge-reranker | Lower AUROC (0.799 vs 0.812), worse precision, 2× latency | Rejected |
| Fine-tuning BGE-M3 on SEBI text | No significant nDCG@10 gain on any arm, so the base model stays | Rejected |
Example questions: "How many nominees can an investor add to a demat account or mutual fund folio?" · "Which circular governs mutual fund nominations, and is it still in force?" · Try an out-of-scope question like GST e-invoicing to see it abstain.
Modes: rag (full answer with citations), retrieval_only (no LLM, still shows citations and lineage), and an optional as-of date. Answers are informational, not legal advice. Always verify on sebi.gov.in.
The corpus, annotations and a ready-to-use index are public on Hugging Face. Annotations are CC-BY-4.0; the underlying SEBI circulars are government works with source links on every record.
Snapshot v2026.08 · 728 circulars (2010–2026), including SEBI's master circulars.
The prebuilt index the live demo uses: 1,490 circulars, 83,752 chunks, 1024-dim BGE-M3 vectors in FAISS, a BM25 index and the lineage graph.
Employers trust skills they can verify. Each claim below links to its place in the skills matrix, and every piece of evidence is public.
See the product artifacts: brief, eval spec, metrics tree, decisions
Last public activity 16 September 2026 · checked 5 November 2026
SEBI Circular RAG is an open-source retrieval-augmented generation system built by Ian Pinto that answers questions over 1,490 SEBI circulars. It combines FAISS and BM25 hybrid retrieval, cross-encoder reranking, supersession-aware citations, an abstention gate and a faithfulness check.
Yes. The live demo on Hugging Face Spaces is free, the code is public on GitHub, and the dataset annotations are published under CC-BY-4.0. The underlying SEBI circulars are Indian government works with source links on every record.
On the full evaluation of 15 August 2026 over 260 adjudicated queries, it scored 0.943 recall@10, 0.916 context recall, 0.881 citation recall, 0.194 citation precision and 0.981 abstention accuracy. That run used a 728-circular corpus; a re-run on the 1,490-circular index on 15 September 2026 cleared all six quality gates.
No. It is an independent research demo, not legal advice, and it is not affiliated with or endorsed by SEBI. Answers can abstain, flag superseded circulars and mark unsupported citations, but you should always verify against sebi.gov.in.
It means the system knows when a circular has been replaced or amended by a later one. It uses a lineage graph of 4,577 supersession and amendment links to demote superseded circulars in results and to show which circular replaced them.
It uses BAAI/bge-m3 for embeddings, BAAI/bge-reranker-v2-m3 for reranking and a small Qwen2.5 instruct model for generation. It runs locally on Apple Silicon with MLX, and the public demo runs the same retrieval on CPU.
The dataset is on Hugging Face at opnsrcntrbtrian/sebi-circulars. Snapshot v2026.08 has 728 circulars, 78,585 retrieval chunks, 4,577 lineage edges, 2,769 supersession pairs, 8,903 normalized citations and 56 evaluation queries. A prebuilt index for 1,490 circulars is at opnsrcntrbtrian/sebi-circulars-index.
Cut extra citations without losing recall. This is the clearest quality gap today.
Report faithfulness as a gated aggregate metric, not just a per-answer flag.
Publish the 1,490-circular snapshot and expand the adjudicated golden set.
A zero-knowledge, offline-first password manager with passkey unlock and on-device AI.
Proves Privacy and security architecture · On-device and local AI · Technical program delivery
Last public activity
Passkey-derived vault key wrapping as an npm library and a research preprint.
Proves Privacy and security architecture · Communicating AI quality honestly
Last public activity
Online lectures into transcripts, slide-linked notes and flashcards, entirely on your Mac.
Proves On-device and local AI · AI evaluation design · AI product strategy and MVP scoping
Last public activity
Search Python code in plain English, then get a one-line explanation of each function.
Proves AI evaluation design · Retrieval (RAG) product design · Communicating AI quality honestly
Last public activity