AINPM.in / Work / Lecture Lens
Case study · Local AI · macOS

Private study tool for online coursesLecture Lens: lectures into notes, on your Mac

Lecture Lens is an open-source macOS app that turns the online lectures you attend into a timecoded transcript, slide-linked notes and flashcards, using models that run on your own Mac. Nothing is uploaded, copy-protected video is never recorded, and every prompt or model change can be checked with a local evaluation.

  • SwiftUI app
  • zsh + Python toolkit
  • screenpipe capture
  • Local models
  • Apache-2.0
opnsrcntrbtr.github.io/lecture-lens
Lecture Lens documentation site overview page describing the toolkit, the macOS app and who it is for
Documentation site built with Docusaurus.
Quality, limits and costsPrototype full eval run, 14 Oct 2026 · project documentation, Oct 2026
  • Answers faithful (43 of 43)1.000
  • Answers relevant (41 of 43)0.953
  • Flashcard quality (10 of 15)0.667
  • Words captured (one class)~75%
Bytes uploaded
0
Build time, 90-min class
~10 min

Tick = the 3 Oct baseline, before the judge scored the answer body only; same generator and prompt versions. Eval run summaries are in the private prototype; the harness and statistics are public. Transcript completeness is from one measured live class. Source: Evaluation guide ↗

How it started: from the private Screenpipe Study prototype to Lecture Lens · How it is built: architecture and decisions

01 · The problem

Who is it for, and what job does it do?

A working professional in a part-time online course, with recorded lectures during the week, a live class at the weekend, and little time to re-watch either.

Job to be done: when I attend a class, I want it captured and turned into material I can trust and will remember, without watching the tooling.
You doLecture Lens does
Watch a recorded lecture in the browserRecords the audio, pauses screen capture on copy-protected players, logs which page you were on
Sit in a live class in a meeting appCaptures slides and audio, picks the distinct slides, describes them with a vision model
Click BuildWrites a timecoded transcript, notes with slide and time citations, and flashcards
Ask a questionAnswers from your own captures, with sources
Change a prompt or a modelRuns a local evaluation so you can see whether quality moved
Product bets

What did I bet on, and what did the evidence say?

Each bet had a measure before it had a feature. Numbers are pass counts from the prototype's eval runs, with 95% Wilson intervals because the samples are small.

BetMeasureResultDecision
Fix the measurement before the modelAnswer relevancy and faithfulness, 43 questions, same generator and promptsRelevancy 31 → 41 of 43 (72% → 95%; intervals 57–83% and 85–99% don't overlap). Faithfulness 37 → 43 of 43, 3 Oct to 14 OctKept: the judge scores the answer body, not its source list. Errored judge calls are counted apart from failures (10 Oct)
Notes you test yourself on beat summariesNotes coverage and faithfulness, 15 cases15 of 15 on both (3 Oct and 14 Oct)Shipped recall-first notes: three questions before the notes
Flashcards worth reviewingG-Eval card quality, 15 cases11 then 10 of 15 (67%, interval 42–85%)Weak spot. Near-duplicate cards dropped (10 Oct); quality is the next prompt target
Say "not covered yet" instead of guessingQuestions the class had not answered yet, 8 casesHedging rose from 2 to 7 of 8, but agreement with the reference fell from 6 to 1 of 8No winner on 8 cases. The suite exposes a real trade-off, so it stays and grows
Questions to ask in a live class24 suggested questionsTeaching value 24 of 24; answerable from the class so far 10 of 24 (42%)Open: good questions, but too many ask about material not yet shown
A model inside Chrome instead of a model serverSame suite, Gemini Nano, 36 questions and 12 card sets (5 Nov)Correctness 32 of 36 (89%) vs 39 of 43 (91%) locally; notes 12 of 12; card quality 3 of 12Studio ships for answers and notes; cards stay on the local model for now (ADR-006)
Fast slide reading without "thinking"Slide-vision checks on dense, high-resolution slidesInvented content in 0 of 5 hallucination checks with thinking off, 3 of 5 with thinking onThinking off for slides (ADR-001)

Riskiest assumptions, and how they are tested

  • The local judge is good enough to gate a release. It scored 100% on the first judge benchmark, which more likely means the benchmark was easy. A blind holdout gave 80% agreement and kappa 0.44, so the judge informs decisions but does not block them yet. How the judge was measured
  • The recorder stays stable. Lecture Lens depends on a capture product it does not control, so the app checks frames written, not what processes report.
  • I will actually review the cards. The review log exists to find out; if it stays empty, cards are the wrong output.

Design review of my own app

I reviewed the six SwiftUI screens against usability heuristics and found 11 issues: 2 blockers, 5 major and 4 minor. The worst was a green "frames arriving" light driven by process status while nothing was being saved.

Three fixes (light driven by frames saved, no eval runs during a live class, show the settings in effect) cover every fault seen in the first live class.

Read the design review

02 · How it works

How does Lecture Lens fit together?

Lecture Lens architectureThree layers. The SwiftUI app starts and stops capture and shows lectures, questions and evaluations. It calls the zsh and Python toolkit with arguments and reads JSON back. The toolkit talks over HTTP on localhost to screenpipe and a local model server, and reads local files.Lecture Lens.app (SwiftUI)Start/stop capture · health lights · lectures · ask · evaluationsToolkit (zsh + Python)sp-start · sp-mode · lecture build · study ask · evaluation runner — argv in, JSON outOn your Macscreenpipe capture · local OpenAI-compatible model server · your files
Everything runs on 127.0.0.1 and local files. Highlighted: components the user installs and controls.
Output per lectureWhat it is
transcript.mdTimecoded speech-to-text
slides.mdDistinct slides with a description and on-screen text
notes.mdNotes with time and slide citations, action items and open questions
cards.tsvFlashcards for a spaced-repetition app
manifest.jsonWhat was built, from what, with which models and prompt versions
03 · Evaluation

How do you know the notes are right?

Every prompt or model change can be measured locally. The judge is a different model from the generator, and the judge itself is benchmarked.

SuiteWhat it checksTime
Smoke6 fixed cases: 2 questions, 2 notes, 2 card sets~12 min
FullAll cases, scored with DeepEval faithfulness, answer relevancy and G-Eval criteria1 h+
Judge benchmarkLabelled claim/context pairs: how often is the judge right?~10 min
Slide readingSynthetic slides with exact answersminutes
Lesson learned: in the first full baseline, most failed question-answering cases were the judge penalising the answer's source list, which it couldn't see. The harness now scores the answer body only. Check what a failure is before fixing the model.
The native app

What does a finished lecture look like in the app?

Screenshots of the macOS app (11 Nov 2026) showing a short demo lecture on how tides work. The slides, notes and cards were made up for these screenshots and are labelled as invented, so no course material appears. Select a screenshot to open it at full size.

Animated walkthrough of Lecture Lens with an invented demo lecture: pick a built lecture, read recall-first notes with citations, browse timed slide descriptions, the transcript, questions answered in class, and flashcards with answers revealed and rated
A 20-second walkthrough of one built lecture (11 and 12 Nov 2026). Shown as a still image if you prefer reduced motion.
Lecture Lens Notes tab for a demo lecture: recall questions, key ideas with timestamps linking to the transcript, and self-check questions
Notes: recall prompts first, then key ideas with timestamps and slide references.
Lecture Lens Slides tab: each captured slide with its timestamp, a one-line description and a thumbs-up or thumbs-down feedback control
Slides: each captured slide with its time and a description, plus "Was this right?" feedback.
Lecture Lens Cards tab: five flashcard questions with Show answer links and an Open for Anki import button
Cards: flashcards to review, ready to import into Anki.
New · 5 Nov 2026

What is Lecture Lens Studio?

The same notes, flashcards, answers and live-class questions, made by a model running inside Chrome instead of a local model server.

opnsrcntrbtr.github.io/lecture-lens/studio
Lecture Lens Studio page with engine choice (Gemini Nano or WebGPU models), a lectures folder picker and tabs for after class, ask, live class, recap and review
Studio before a lectures folder is chosen; no course content shown.

Why it matters

  • No model server and no 20 to 40 GB of weights: Gemini Nano runs in Chrome, with WebGPU models (such as Qwen3 4B at about 2.6 GB) as fallback (ADR-006)
  • Ships as a static page and an unpacked extension; nothing leaves the browser and the lectures folder
  • Live Q&A parity and in-browser transcription added on 5 Nov 2026; silent audio chunks skipped before transcription on 6 Nov 2026
  • The local judge can score Studio's outputs too, so both engines are measured the same way

Open Lecture Lens Studio

04 · Decisions & trade-offs

What was decided, and what was dropped?

DecisionReasonCall
Pause screen capture on DRM players, keep audioProtected video goes black while screen capture runs. The protection is respected; transcripts come from audio (ADR-004).Shipped
Three-layer content guardA public repo about a private activity: ignore rules, a hashed blocklist scan on every commit, and a guarded publish script.Shipped
Evals refuse to run during a live classEvaluation swaps the model and would disrupt capture.Shipped
Record everything, filter laterBreaks the privacy rule and the protected-video constraint.Dropped
Longer notes and weekly digestsFeels productive; the evidence says summaries you didn't write do little.Dropped
Hosted evaluation dashboardsCaptured text would leave the machine.Dropped
05 · Limits & roadmap

What are the limits, and what's next?

Known limits

  • macOS on Apple silicon only
  • In one measured live class, about a quarter of spoken words were missing compared with the meeting app's own transcript
  • Local models are slow: a full evaluation takes over an hour
  • Depends on screenpipe, a separate product it does not control

Next

  • Capture guard that alerts within 60 seconds when nothing is being saved
  • Cancellable evaluations that show the change against the previous run
  • Recall-first notes and a flashcard review log
  • A read-only local MCP server so an assistant can list, build and ask about lectures
Proof of skill

What does this project prove, and how can you check it?

Employers trust skills they can verify. Each claim below links to its place in the skills matrix, and every piece of evidence is public.

Skills demonstrated

Evidence you can open

See the product artifacts: brief, eval spec, metrics tree, decisions

Last public activity 6 November 2026 · checked 5 November 2026

06 · FAQ

Frequently asked questions

What is Lecture Lens?

Lecture Lens is an open-source macOS app and toolkit by Ian Pinto that turns online lectures into a timecoded transcript, slide-linked notes with citations, and flashcards. It runs entirely on an Apple silicon Mac with local models, and nothing is uploaded.

What do I need to run Lecture Lens?

macOS 15 or later on Apple silicon, with 32 GB of memory or more being realistic for the models; screenpipe installed separately for capture; a local OpenAI-compatible model server with a text model, a vision-capable model and a second model as evaluation judge; Python 3.11+, ffmpeg and Xcode command line tools.

Does Lecture Lens record copy-protected video?

No. On DRM-protected course players, screen capture pauses and only audio is kept, so the video stays watchable. Notes and flashcards for those lectures are built from the audio transcript.

How is the quality of notes and answers checked?

A local evaluation harness generates answers, notes and cards, then swaps in a different local model as judge and scores them with DeepEval metrics such as faithfulness and answer relevancy. A separate judge benchmark measures how often the judge itself is right.

How does it keep course material out of a public repository?

Three layers: .gitignore excludes capture output and recordings; a content guard scans every commit for blocked names stored as hashes, forbidden file types, secrets and personal data; and plain git push is disabled in favour of a publish script that runs the guard, which CI runs again.

More work

Other case studies