AINPM.in / Work / CodeSearchNet Find & Explain
Case study · Retrieval · Fine-tuning

Semantic code search with measured resultsFind Python code in plain English, then explain it

CodeSearchNet Find & Explain retrieves Python functions that match a plain-English query and writes a one-line explanation of each one. Fine-tuning a small encoder raised mean reciprocal rank from 0.527 (BM25) to 0.815 on 1,000 test queries, and the report publishes the weak spots alongside the wins.

  • sentence-transformers
  • MiniLM-L6
  • flan-t5-base
  • Gradio
  • MIT
Retrieval on 1,000 test queries (1 gold + 999 distractors)Project report, Aug 2026 · official CodeSearchNet Python test split
  • MRR, fine-tuned (BM25 0.527)0.815
  • Recall@10.725
  • Recall@100.961
  • Summary BERTScore F1 (zero-shot 0.747)0.846

Tick = baseline (BM25 or zero-shot). Queries are author-written docstrings, which likely overstates real-world search quality. Source: REPORT.md ↗

01 · The problem

What job does it do?

Keyword search fails when a developer describes intent ("validate input") and the code says something else (check_args). Docstrings and code share little vocabulary, so lexical search is weak.

Job to be done: when I need a function that already exists, I want to describe what it does and see the best matches with a plain explanation, so I can judge relevance without reading every body.
02 · Results

How much better is it than the baselines?

Official CodeSearchNet Python test split: 1,000 queries, each scored against its gold function plus 999 distractors.

RunMRRRecall@1Recall@10
BM25 (lexical)0.5270.4070.747
MiniLM, zero-shot0.6870.5670.898
MiniLM, fine-tuned on 50K pairs0.7940.7010.945
MiniLM, fine-tuned on 393K pairs0.8150.7250.961
Was the extra data worth it? Training on 7.9 times more data added only 0.022 MRR. It is a real gain (paired bootstrap 95% interval 0.012 to 0.032; t-test p < 0.001), but a small one. The next experiment is more epochs, not more data.
03 · How it works

How was it built and evaluated?

Data

  • CodeSearchNet Python (412,178 training rows) joined on GitHub permalink to a dataset with curated one-line summaries: 100% join rate on a 20,000-row sample
  • Official splits kept, which are already separated by repository
  • Docstrings stripped from the code side so the task is not trivially lexical; duplicates and boilerplate removed, leaving 393,427 pairs

Models

  • Retrieval: all-MiniLM-L6-v2 fine-tuned with in-batch negatives (MultipleNegativesRankingLoss), exact search over normalised embeddings
  • Explanation: flan-t5-base (250M parameters) fine-tuned on the same corpus: BERTScore F1 0.747 zero-shot to 0.846 fine-tuned, on 50 test examples
  • An API model can stand in for the local summariser behind the same interface and prompt
04 · Limits

Where does it fall short?

LimitEvidence
Docstring-as-query biasQueries are written by the same author as the code, which likely overstates real-world search quality.
Bigger search space, lower top-1MRR falls from 0.882 to 0.755 as distractors grow from 99 to 4,999; Recall@10 stays above 0.92.
Small summary sampleSummaries were scored on 50 examples; one empty output was replaced with a documented sentinel rather than dropped.
ScopePython only; one training epoch for every model.
Proof of skill

What does this project prove, and how can you check it?

Employers trust skills they can verify. Each claim below links to its place in the skills matrix, and every piece of evidence is public.

Skills demonstrated

Evidence you can open

See the product artifacts: brief, eval spec, metrics tree, decisions

Last public activity 13 August 2026 · checked 5 November 2026

05 · FAQ

Frequently asked questions

What is CodeSearchNet Find & Explain?

An open-source two-stage pipeline by Ian Pinto: a fine-tuned sentence-transformer finds Python functions that match a plain-English query, and a fine-tuned flan-t5-base model writes a one-line explanation of each result. A live demo runs on Hugging Face Spaces.

How good is the retrieval?

On 1,000 queries from the official CodeSearchNet Python test split, each scored against its gold function plus 999 distractors, the full-corpus model reaches MRR 0.815, Recall@1 0.725 and Recall@10 0.961. BM25 reaches MRR 0.527 and the untuned encoder 0.687.

Is the gain from more training data real?

Yes, but small. Training on all 393,427 pairs instead of 50,000 adds 0.022 MRR. A paired bootstrap (10,000 resamples) gives a 95% interval of 0.012 to 0.032, and a paired t-test gives p < 0.001.

What are the limits?

Queries are author-written docstrings, which likely overstates real-world search quality. MRR falls from 0.882 to 0.755 as the distractor pool grows from 99 to 4,999. It covers Python only, and every model was trained for one epoch.

More work

Other case studies