AINPM.in / Work / Product artifacts / CodeSearchNet Find & Explain
Product artifacts · Checked 8 Nov 2026

CodeSearchNet Find & Explain: the product work behind it

These five artifacts show how CodeSearchNet Find & Explain was framed, measured and decided, not just how it was coded. Each one summarises the project's own documents and links to them, so you can check every line. Read the case study.

01 · Problem brief

What problem does it solve, and for whom?

Problem

Keyword search fails when a developer describes intent and the code uses different words; docstrings and code share little vocabulary.

Users

Developers searching unfamiliar Python codebases.

Job to be done

"Describe what a function does, see the best matches, and understand each one without reading its body."

Constraints

  • Small enough to fine-tune on a free Colab T4 GPU
  • Official CodeSearchNet splits, already separated by repository, so test code never leaks into training
  • Exact search, not approximate, so index error cannot confound the metrics
  • Both summariser backends get the identical prompt, so they are comparable

Non-goals

  • Languages other than Python
  • Approximate-nearest-neighbour indexing at this corpus size
  • Real developer query logs (none were available)

Source: REPORT.md §1, §3, §4 ↗

02 · Evaluation spec

How is quality defined and gated?

Each of 1,000 test queries is ranked against its gold function plus 999 distractors; lexical and zero-shot baselines set the bar, and gains must survive significance tests.

Test set

  • Official CodeSearchNet Python test split; 393,427 training pairs after filtering and de-duplication
  • A gold function never ranked counts as a miss (0), not dropped
  • Summaries scored on 50 test examples with BLEU and BERTScore

Metrics

MetricMeasuredBar
MRR0.8150.527 (BM25)
Recall@10.7250.407 (BM25)
Recall@100.9610.747 (BM25)
nDCG@100.8500.574 (BM25)
Summary BERTScore F10.8460.747 (zero-shot)
Rule: A claimed gain needs a paired bootstrap (10,000 resamples) and a paired t-test; the 50K-to-393K data gain of +0.022 MRR passed both (p < 0.001), so it is reported as real but small.

Source: REPORT.md §4.3, §5 ↗

03 · Metrics tree

What does success mean, and what must not break?

My framing of metrics the project already measures.

North star

The right function appears in the first results and is understood without reading it

Search only helps if the answer is near the top and the developer can judge it quickly.

Drivers
  • Right function near the top: Recall@10 0.961 · MRR 0.815
  • Explanation matches meaning: BERTScore F1 0.846
Guardrails
  • Holds up in a bigger search space: MRR 0.755 at 4,999 distractors (0.882 at 99)
  • No empty explanations: 1 of 50 empty, replaced with a documented sentinel
  • Realistic queries: Docstring queries flagged as likely optimistic
04 · Decision log

What was decided, and why?

DecisionDateStatusWhat and why
Data Join two datasets on the GitHub permalink ↗10 Aug 2026AdoptedUse CodeSearchNet plus a dataset with curated one-line summaries; 100% join rate on a 20,000-row sample. Why: The simpler alternative lacked repository and language metadata, so Python filtering and repo-disjoint splits were impossible.
Prep Strip docstrings from the code side ↗10 Aug 2026AdoptedRemove docstrings from code and truncate query docstrings at the first structured section. Why: Leaving them in makes retrieval trivially lexical.
Index Exact search instead of approximate ↗10 Aug 2026AdoptedBrute-force search over normalised embeddings. Why: Approximation error would confound the metrics at this corpus size.
Model Smaller summariser rejected ↗10 Aug 2026Rejectedflan-t5-small (80M) dropped for flan-t5-base (250M). Why: It could not follow the prompt and degenerated into repeated unknown tokens.
Next More epochs before more data ↗10 Aug 2026AcceptedThe next experiment is a 3-epoch run, not a larger corpus. Why: 7.9 times more data added only 0.022 MRR.
05 · Launch notes

What shipped, what's weak, and what's next?

Status: Live demo on Hugging Face Spaces; models and code public

Shipped

  • Gradio demo with retrieval-only and find-and-explain modes
  • Two public retrieval models (50K and full corpus)
  • Full report with ablation, pool-size sensitivity and error analysis

Known limits

  • Docstring-as-query bias likely overstates real-world quality
  • Python only; one epoch for every model
  • Summaries scored on 50 examples

Next

  • 3-epoch fine-tune on the full corpus
  • Evaluation on real developer queries
  • Other languages and an approximate index for larger corpora

Source: REPORT.md §7, §8 ↗ · All product artifacts

Open it yourself: Live demo ↗ · GitHub ↗ · Model ↗

FAQ

What else should you know about these artifacts?

How is CodeSearchNet Find & Explain evaluated?

Each of 1,000 test queries is ranked against its gold function plus 999 distractors; lexical and zero-shot baselines set the bar, and gains must survive significance tests. A claimed gain needs a paired bootstrap (10,000 resamples) and a paired t-test; the 50K-to-393K data gain of +0.022 MRR passed both (p < 0.001), so it is reported as real but small.

Which decisions shaped CodeSearchNet Find & Explain?

Join two datasets on the GitHub permalink (adopted, 10 Aug 2026): The simpler alternative lacked repository and language metadata, so Python filtering and repo-disjoint splits were impossible. Strip docstrings from the code side (adopted, 10 Aug 2026): Leaving them in makes retrieval trivially lexical. Exact search instead of approximate (adopted, 10 Aug 2026): Approximation error would confound the metrics at this corpus size. Smaller summariser rejected (rejected, 10 Aug 2026): It could not follow the prompt and degenerated into repeated unknown tokens. More epochs before more data (accepted, 10 Aug 2026): 7.9 times more data added only 0.022 MRR.

What are the known limits of CodeSearchNet Find & Explain?

Docstring-as-query bias likely overstates real-world quality. Python only; one epoch for every model. Summaries scored on 50 examples.