Cited answers over 1,490 SEBI circulars, or an honest abstention.
- RAG
- Evals
- SEBI
Proves AI evaluation design · Retrieval (RAG) product design · Trust and compliance by design (India) · Communicating AI quality honestly
Last public activity
CodeSearchNet Find & Explain retrieves Python functions that match a plain-English query and writes a one-line explanation of each one. Fine-tuning a small encoder raised mean reciprocal rank from 0.527 (BM25) to 0.815 on 1,000 test queries, and the report publishes the weak spots alongside the wins.
Tick = baseline (BM25 or zero-shot). Queries are author-written docstrings, which likely overstates real-world search quality. Source: REPORT.md ↗
Keyword search fails when a developer describes intent ("validate input") and the code says something else (check_args). Docstrings and code share little vocabulary, so lexical search is weak.
Official CodeSearchNet Python test split: 1,000 queries, each scored against its gold function plus 999 distractors.
| Run | MRR | Recall@1 | Recall@10 |
|---|---|---|---|
| BM25 (lexical) | 0.527 | 0.407 | 0.747 |
| MiniLM, zero-shot | 0.687 | 0.567 | 0.898 |
| MiniLM, fine-tuned on 50K pairs | 0.794 | 0.701 | 0.945 |
| MiniLM, fine-tuned on 393K pairs | 0.815 | 0.725 | 0.961 |
| Limit | Evidence |
|---|---|
| Docstring-as-query bias | Queries are written by the same author as the code, which likely overstates real-world search quality. |
| Bigger search space, lower top-1 | MRR falls from 0.882 to 0.755 as distractors grow from 99 to 4,999; Recall@10 stays above 0.92. |
| Small summary sample | Summaries were scored on 50 examples; one empty output was replaced with a documented sentinel rather than dropped. |
| Scope | Python only; one training epoch for every model. |
Employers trust skills they can verify. Each claim below links to its place in the skills matrix, and every piece of evidence is public.
See the product artifacts: brief, eval spec, metrics tree, decisions
Last public activity 13 August 2026 · checked 5 November 2026
An open-source two-stage pipeline by Ian Pinto: a fine-tuned sentence-transformer finds Python functions that match a plain-English query, and a fine-tuned flan-t5-base model writes a one-line explanation of each result. A live demo runs on Hugging Face Spaces.
On 1,000 queries from the official CodeSearchNet Python test split, each scored against its gold function plus 999 distractors, the full-corpus model reaches MRR 0.815, Recall@1 0.725 and Recall@10 0.961. BM25 reaches MRR 0.527 and the untuned encoder 0.687.
Yes, but small. Training on all 393,427 pairs instead of 50,000 adds 0.022 MRR. A paired bootstrap (10,000 resamples) gives a 95% interval of 0.012 to 0.032, and a paired t-test gives p < 0.001.
Queries are author-written docstrings, which likely overstates real-world search quality. MRR falls from 0.882 to 0.755 as the distractor pool grows from 99 to 4,999. It covers Python only, and every model was trained for one epoch.
Cited answers over 1,490 SEBI circulars, or an honest abstention.
Proves AI evaluation design · Retrieval (RAG) product design · Trust and compliance by design (India) · Communicating AI quality honestly
Last public activity
A zero-knowledge, offline-first password manager with passkey unlock and on-device AI.
Proves Privacy and security architecture · On-device and local AI · Technical program delivery
Last public activity
Passkey-derived vault key wrapping as an npm library and a research preprint.
Proves Privacy and security architecture · Communicating AI quality honestly
Last public activity
Online lectures into transcripts, slide-linked notes and flashcards, entirely on your Mac.
Proves On-device and local AI · AI evaluation design · AI product strategy and MVP scoping
Last public activity