AINPM.in / SEBI Circular RAG case study
MVP · Live · Open source

AI-native PM case studySEBI Circular RAG: trustworthy answers over Indian securities regulation

SEBI Circular RAG is an open-source assistant that answers questions over 1,490 SEBI circulars with citations, supersession checks and abstention. Compliance and product teams at AMCs, brokers and fintechs work from hundreds of SEBI circulars, many of them amended, consolidated into master circulars, or superseded. I built an AI-native assistant that finds the right circular, cites it, tells you whether it's still in force, and says so when it doesn't know.

  • Role: product owner & builder
  • Jul–Sep 2026
  • Python · FAISS · BM25 · BGE-M3
  • Gradio on Hugging Face
  • Data: CC-BY-4.0
Citations with in-force / superseded status
SEBI Circular RAG interface showing numbered citations with circular numbers, supersession status, successor circulars and underlying regulations
A cited answer. Superseded circulars carry a warning and point to their successor.
Quality on 260 adjudicated questionsFull evaluation 15 Aug 2026 · gate floors derived 2 Sep 2026
  • Abstention accuracy0.981
  • Recall@100.943
  • Context recall0.916
  • Citation recall0.881
  • nDCG@100.697
  • Citation precision0.194

Tick = release floor. Every metric clears its floor; citation precision is the known weak spot. Source: Evaluation log ↗

01 · The problem

Why is regulatory search hard for AI?

In regulation, a confident wrong answer is worse than no answer.

Rules change underneath you

SEBI regularly amends circulars and folds them into master circulars. An answer based on a superseded circular can look right and still be non-compliant.

"What applied then?" matters

Audits, disputes and product reviews often need the rule as it stood on a specific date, not today's version.

Generic chatbots guess

General LLMs hallucinate circular numbers and rarely admit gaps. Every answer has to be checkable against the source.

Who it's for

Compliance officers and product managers at mutual fund AMCs, brokers, depositories, RTAs and fintechs; RegTech builders; researchers working on Indian legal NLP.

Job to be done

"Help me find the circular that governs this, show me the exact text, tell me whether it still applies, and don't make anything up."

02 · Product principles

What makes the answers trustworthy?

Trust was the product requirement, not a feature. These six rules shaped every design and model decision.

Cite everything

Each claim links to a numbered circular and the exact passage it came from.

Abstain when unsure

A calibrated gate refuses out-of-scope or weakly supported questions, such as RBI or GST topics, instead of stretching SEBI text to fit.

Know what's superseded

A lineage graph of 4,577 supersession and amendment links demotes superseded circulars and names the one that replaced them.

Answer "as of" a date

Pick a date and retrieval scores against the rules in force then, excluding circulars issued later.

Check its own work

A faithfulness check flags any cited circular that wasn't in the retrieved context and adds a caution.

Stay honest about limits

Not legal advice, not SEBI-endorsed, and the coverage caveats are published with the data.

03 · How it works

How does SEBI Circular RAG work?

Nine steps take a SEBI PDF to a grounded, checkable answer.

  1. Ingest

    Scrape SEBI circulars and master circulars, extract PDF text, and validate corpus integrity.

  2. Section-aware chunking

    Split along the circular's own structure; prefix each chunk with circular number, subject and section.

  3. Lineage graph

    Mine references to build supersedes / amends edges and link circulars to the regulations they rely on.

  4. Hybrid retrieval

    BGE-M3 dense vectors (FAISS) plus BM25 keywords, fused with reciprocal rank fusion.

  5. Rerank

    A bge-reranker-v2-m3 cross-encoder re-scores the top 50 candidates.

  6. Supersession & as-of

    Demote superseded circulars; for dated questions, filter to what was in force on that date.

  7. Abstention gate

    Topic-similarity thresholds decide whether there's enough evidence to answer at all.

  8. Grounded generation

    A small Qwen2.5 model writes the answer using only the retrieved passages.

  9. Citation filter & faithfulness

    Keep only citations the answer actually uses; flag any unsupported ones.

Retrieval & generationTrust controls

Runs local-first on Apple Silicon (MLX), with a FastAPI service and API keys. The public demo runs the same retrieval on a free Hugging Face CPU Space using a prebuilt index, so results match the local system.

04 · Evals as the spec

How accurate is it?

260 adjudicated questions decide what ships. Instead of a PRD with "should be accurate", the spec is a golden set. It is stratified by the ways regulatory search fails, reviewed row by row, and wired into gates that block regressions.

  • 40 title lookups
  • 60 paraphrased body text
  • 30 numbers & tables
  • 40 lineage / supersession
  • 20 multi-hop
  • 20 repealed legal basis
  • 40 hard negatives
  • 10 far negatives
MetricResultWhat it tells a PM
Recall@100.943The right circular is in the top 10 for 94% of answerable questions.
Context recall0.916The passages sent to the model contain the needed evidence.
nDCG@100.697Ranking quality: how high the right circular appears.
Citation recall0.881Answers cite the circulars they should.
Citation precision0.194Still extra citations per answer. The known weak spot and the next target.
Abstention accuracy0.981Correctly refuses or answers across both answerable and out-of-scope questions.
Warm answer latency~2.1 sLocal generation on Apple Silicon.

Source: project status log, full evaluation 15 Aug 2026 (728-circular corpus). Faithfulness is checked per answer but not yet reported as an aggregate metric. The test suite had 1,131 passing tests on 16 Sep 2026.

05 · PM decisions & trade-offs

What was shipped, and what was rejected?

Every change was an experiment measured against the golden set. Saying no to plausible ideas was as important as saying yes.

DecisionEvidenceCall
Enrich chunks with circular number, subject & sectionCitation precision 0.60 → 0.74 on the earlier eval setShipped
Retrieve 10 contexts instead of 5Citation recall 0.772 → 0.888, but precision 0.177 → 0.119. Missing a governing circular costs more than an extra citation.Shipped
Post-hoc citation filter (cross-encoder, margin 0.35)Prompting the 1.5B model to cite selectively was a no-op. It emitted no parseable citations. A model-agnostic filter raised citation precision 88% at the chosen margin.Shipped
Fix mislabeled eval rows before tuningRe-labeling mislabeled abstain rows moved abstention accuracy 0.849 → 0.973. The model was right; the test was wrong.Shipped
HyDE (hypothetical-answer retrieval)Recall@10 −2.3 pp (not significant) at 41× latencyRejected
SPLADE sparse retrieval legLooked promising (nDCG +0.029) but failed a pre-registered held-out confirmationRejected
Qwen3-Reranker-0.6B instead of bge-rerankerLower AUROC (0.799 vs 0.812), worse precision, 2× latencyRejected
Fine-tuning BGE-M3 on SEBI textNo significant nDCG@10 gain on any arm, so the base model staysRejected
The PM lesson: the biggest quality jump came from fixing evaluation data, not from a new model. In AI-native products, the eval set is the product spec, so it deserves the same review rigour as code.
06 · Live demo

Try it: ask a question about SEBI circulars

Example questions: "How many nominees can an investor add to a demat account or mutual fund folio?" · "Which circular governs mutual fund nominations, and is it still in force?" · Try an out-of-scope question like GST e-invoicing to see it abstain.

Hugging Face Space · free CPU hardware. The first question can take a few minutes while models load.Open full screen ↗

The demo loads on request to keep this page fast.

Modes: rag (full answer with citations), retrieval_only (no LLM, still shows citations and lineage), and an optional as-of date. Answers are informational, not legal advice. Always verify on sebi.gov.in.

07 · Open data

Published so others can build and benchmark

The corpus, annotations and a ready-to-use index are public on Hugging Face. Annotations are CC-BY-4.0; the underlying SEBI circulars are government works with source links on every record.

sebi-circulars dataset ↗

Snapshot v2026.08 · 728 circulars (2010–2026), including SEBI's master circulars.

  • 728 corpus
  • 78,585 chunks
  • 4,577 lineage edges
  • 2,769 supersession pairs
  • 8,903 normalized citations
  • 56 eval queries

sebi-circulars-index ↗

The prebuilt index the live demo uses: 1,490 circulars, 83,752 chunks, 1024-dim BGE-M3 vectors in FAISS, a BM25 index and the lineage graph.

  • 1,490 circulars
  • 83,752 chunks
  • FAISS + BM25
  • lineage.json
Proof of skill

What does this project prove, and how can you check it?

Employers trust skills they can verify. Each claim below links to its place in the skills matrix, and every piece of evidence is public.

Skills demonstrated

Evidence you can open

See the product artifacts: brief, eval spec, metrics tree, decisions

Last public activity 16 September 2026 · checked 5 November 2026

08 · FAQ

Frequently asked questions

What is SEBI Circular RAG?

SEBI Circular RAG is an open-source retrieval-augmented generation system built by Ian Pinto that answers questions over 1,490 SEBI circulars. It combines FAISS and BM25 hybrid retrieval, cross-encoder reranking, supersession-aware citations, an abstention gate and a faithfulness check.

Is SEBI Circular RAG free to use?

Yes. The live demo on Hugging Face Spaces is free, the code is public on GitHub, and the dataset annotations are published under CC-BY-4.0. The underlying SEBI circulars are Indian government works with source links on every record.

How accurate is SEBI Circular RAG?

On the full evaluation of 15 August 2026 over 260 adjudicated queries, it scored 0.943 recall@10, 0.916 context recall, 0.881 citation recall, 0.194 citation precision and 0.981 abstention accuracy. That run used a 728-circular corpus; a re-run on the 1,490-circular index on 15 September 2026 cleared all six quality gates.

Can I rely on it for compliance decisions?

No. It is an independent research demo, not legal advice, and it is not affiliated with or endorsed by SEBI. Answers can abstain, flag superseded circulars and mark unsupported citations, but you should always verify against sebi.gov.in.

What does supersession-aware mean?

It means the system knows when a circular has been replaced or amended by a later one. It uses a lineage graph of 4,577 supersession and amendment links to demote superseded circulars in results and to show which circular replaced them.

Which models does it use?

It uses BAAI/bge-m3 for embeddings, BAAI/bge-reranker-v2-m3 for reranking and a small Qwen2.5 instruct model for generation. It runs locally on Apple Silicon with MLX, and the public demo runs the same retrieval on CPU.

Where can I get the SEBI circulars dataset?

The dataset is on Hugging Face at opnsrcntrbtrian/sebi-circulars. Snapshot v2026.08 has 728 circulars, 78,585 retrieval chunks, 4,577 lineage edges, 2,769 supersession pairs, 8,903 normalized citations and 56 evaluation queries. A prebuilt index for 1,490 circulars is at opnsrcntrbtrian/sebi-circulars-index.

09 · What's next

Roadmap

Raise citation precision

Cut extra citations without losing recall. This is the clearest quality gap today.

Measure faithfulness end-to-end

Report faithfulness as a gated aggregate metric, not just a per-answer flag.

Refresh the public dataset

Publish the 1,490-circular snapshot and expand the adjudicated golden set.

More work

Other case studies