Lecture Lens: the product work behind it
These five artifacts show how Lecture Lens was framed, measured and decided, not just how it was coded. Each one summarises the project's own documents and links to them, so you can check every line. Read the case study.
What problem does it solve, and for whom?
Problem
Working professionals in part-time online courses attend recorded lectures and live classes with little time to re-watch either, and cloud note-takers upload course material.
Users
A working professional in a part-time online course, on an Apple silicon Mac.
Job to be done
"When I attend a class, I want it captured and turned into material I can trust and will remember, without watching the tooling."
Constraints
- Everything runs on the Mac; nothing is uploaded
- Copy-protected video is never recorded; audio is kept
- One user, no Apple Developer certificate
- Evaluation must never disrupt a live class
Non-goals
- Recording everything and filtering later
- Longer notes and weekly digests
- Hosted evaluation dashboards
Source: Roadmap and ADR-002 ↗
How is quality defined and gated?
Every prompt or model change can be measured locally, with a judge model that differs from the generator and is itself benchmarked.
Test set
- Smoke: 6 fixed cases (2 questions, 2 notes, 2 card sets), about 12 minutes
- Full: all cases, scored with DeepEval faithfulness, answer relevancy and G-Eval criteria, over an hour
- Judge benchmark on labelled claim and context pairs; synthetic slides with exact answers
Metrics
| Metric | Measured | Bar |
|---|---|---|
| Words captured, one measured class · Weak spot | ~75% | — |
| Bytes uploaded | 0 | — |
| Build time for a 90-minute class | ~10 min | — |
| Smoke-eval cases | 6 | — |
| Notes cases passed (prototype, 3 Oct 2026) | 15 of 15 | Faithfulness ≥ 0.85 |
| Answer cases passing every metric (3 Oct baseline) | 26 of 43 | Faithfulness ≥ 0.8 |
| Answers faithful / relevant after the judge fix (14 Oct) | 43 and 41 of 43 | Faithfulness ≥ 0.8, relevancy ≥ 0.7 |
| Flashcard cases passing every metric (3 Oct) · Weak spot | 9 of 15 | Faithfulness ≥ 0.85 |
| Flashcard quality (14 Oct) · Weak spot | 10 of 15 | G-Eval ≥ 0.7 |
| Judge agreement with labels, blind holdout | 80%, kappa 0.44 | — |
Source: Evaluation guide ↗
What does success mean, and what must not break?
My framing of metrics the project already measures.
Classes captured and turned into notes and cards the learner trusts and reviews
Capture that misses the class, or notes the learner cannot trust, make the rest useless.
- Capture completeness: Words captured ~75% in one measured class
- Notes stay faithful: DeepEval faithfulness and answer relevancy
- Cards get reviewed: Flashcard review log (planned)
- Privacy: 0 bytes uploaded
- Respect protected video: Screen capture pauses on DRM players
- Don’t break a live class: Evaluations refuse to run during capture
What was decided, and why?
| Decision | Date | Status | What and why |
|---|---|---|---|
| Vision Slide descriptions chosen on realistic frames ↗ | 29 Sept 2026 | Accepted | Use a 35B multimodal model with thinking off, crop to the lecture player, and discard any description where the image provably never reached the model. Why: Thinking on looked safer on clean slides but passed only 15 of 21 realistic ones; production-style input passed 21 of 21 (prototype ADR-001). |
| ADR-002 Native SwiftUI app launches the toolkit ↗ | 29 Sept 2026 | Accepted | A menu-bar app starts capture itself and calls the tools through a JSON contract. Why: macOS grants screen recording to the app that launches capture; terminal launches produced 0 frames. |
| ADR-004 Pause screen capture on protected video, keep audio ↗ | 30 Sept 2026 | Accepted | Pause capture on listed DRM player pages and build sessions from page markers and the transcript. Why: Protected video went black while capture ran; the protection is respected and the transcript is what matters. |
| ADR-005 Confirm repeated slides with structural similarity ↗ | 4 Nov 2026 | Accepted | Keep the cheap hash test, then confirm a repeat only if at most 5 of 36 grid cells changed. Why: Slides from one template look alike to the hash, so the old test missed most of them (recall 0.17 to 0.58 on synthetic decks); counting changed cells reached recall 1.00. Accepted on synthetic evidence, to confirm on a real build. |
| ADR-006 Run the language model inside Chrome ↗ | 5 Nov 2026 | Accepted | Add a browser provider: Gemini Nano first, a WebGPU model as fallback; the local model server stays the default. Why: The local server costs 20 to 40 GB of weights and a capable Mac; the browser needs neither. |
What shipped, what's weak, and what's next?
Status: Early, open source; Lecture Lens Studio in MVP
Shipped
- SwiftUI menu-bar app over a zsh and Python toolkit
- Local evaluation harness with a calibrated judge
- Six-layer content guard for a public repo
- Studio: notes, cards, answers, live Q&A and in-browser transcription inside Chrome (5 Nov 2026)
Known limits
- macOS on Apple silicon only
- About a quarter of spoken words missing in one measured class
- A full evaluation takes over an hour
- Depends on screenpipe, which it does not control
Next
- Capture guard that alerts within 60 seconds when nothing is saved
- Cancellable evaluations that show the change against the last run
- Eval checks drawn from real failures
Source: Roadmap ↗ · All product artifacts
What else should you know about these artifacts?
How is Lecture Lens evaluated?
Every prompt or model change can be measured locally, with a judge model that differs from the generator and is itself benchmarked. Check what a failure is before fixing the model: the first full baseline failed mostly because the judge penalised a source list it could not see, so only the answer body is scored now.
Which decisions shaped Lecture Lens?
Slide descriptions chosen on realistic frames (accepted, 29 Sept 2026): Thinking on looked safer on clean slides but passed only 15 of 21 realistic ones; production-style input passed 21 of 21 (prototype ADR-001). Native SwiftUI app launches the toolkit (accepted, 29 Sept 2026): macOS grants screen recording to the app that launches capture; terminal launches produced 0 frames. Pause screen capture on protected video, keep audio (accepted, 30 Sept 2026): Protected video went black while capture ran; the protection is respected and the transcript is what matters. Confirm repeated slides with structural similarity (accepted, 4 Nov 2026): Slides from one template look alike to the hash, so the old test missed most of them (recall 0.17 to 0.58 on synthetic decks); counting changed cells reached recall 1.00. Accepted on synthetic evidence, to confirm on a real build. Run the language model inside Chrome (accepted, 5 Nov 2026): The local server costs 20 to 40 GB of weights and a capable Mac; the browser needs neither.
What are the known limits of Lecture Lens?
macOS on Apple silicon only. About a quarter of spoken words missing in one measured class. A full evaluation takes over an hour. Depends on screenpipe, which it does not control.