Lecture Lens architecture: a class becomes study material without leaving the Mac
Lecture Lens is a pipeline of four local parts: a recorder that captures screen and audio, a Python toolkit that builds each lecture, a model server with one slot shared by the generator and the judge, and a SwiftUI app that starts and watches them. Every service listens on 127.0.0.1, so the only traffic is the lecture streaming in.
Where does the data go?
Two ways in, one build, five files out. On narrow screens, scroll the diagram sideways.
Which decisions shape it?
Each one has a written record with the evidence and the alternatives considered.
| Decision | Why | Record |
|---|---|---|
| The app launches the recorder | macOS gives recording permission to the app that starts the recorder. Started from a terminal, it inherits no grants and waits on a prompt nobody sees. | ADR-002 |
| The app calls the toolkit; it never re-implements it | One tested implementation of each capability. Every tool takes arguments, prints one JSON document and logs to stderr; spawning costs 100 to 300 ms per call. | System design |
| Pause screen capture on protected pages, keep the audio | Any open screen stream blanks DRM video; audio never does. The lecture stays watchable and the transcript still gets built. | ADR-004 |
| Hash first, then structural similarity, to find distinct slides | Slides from one template look alike to a hash. Counting changed grid cells raised recall on synthetic decks from 0.17–0.58 to 1.00. | ADR-005 |
| Describe slides with thinking off | On dense, high-resolution slides, thinking mode invented content in 3 of 5 checks; thinking off in none. | ADR-001 (prototype) |
| One model slot shared by generator and judge | The model server holds one large model. The eval runner swaps the judge in and always swaps back; evals refuse to start during a live class. | Architecture |
| A model inside Chrome as a second engine | No model server or 20 to 40 GB of weights. Gemini Nano matched the local model on answers (89% vs 91% correct) but not on flashcards. | ADR-006 |
| Rewrite mode-owned settings before every start | The recorder keeps flags between runs. A site allowlist left from portal mode once made it skip every window in live mode. | Architecture |
How do the app and the toolkit talk?
Arguments in, exactly one JSON document out on stdout, progress on stderr. Exit code 0 is success, 1 a domain failure (JSON still printed, with "ok": false), 2 a usage error.
| Command | Returns |
|---|---|
lecture doctor --json | Checks for the database, frames, app audio, ffmpeg, the vision model and the vault, each with a fix |
lecture list --json | Sessions with start, end, minutes, frames and whether they are built |
lecture build <id> --json | The lecture folder and its manifest: inputs, models and prompt versions |
study ask "…" --json | The answer, its numbered sources with timestamps, the model and the latency |
What does each choice cost, and when would it change?
| Chosen | Cost | Revisit when |
|---|---|---|
| Tools called by the app, not rewritten in Swift | 100–300 ms process spawn per call | A call becomes latency-critical (then add a daemon) |
| Polling health every 5–60 s, not push | A small CPU cost | The recorder exposes the events needed |
| Append-only JSONL feedback, not SQLite | A full scan to read | Feedback passes about 50,000 rows |
| Unsandboxed app | No App Store | Never, for this use: it launches processes and reads the recorder database |
How well it works is on the case study's product bets, and how the judge itself was checked is in this builder's note.
Frequently asked questions
How does Lecture Lens keep everything on the Mac?
Every service it talks to listens on 127.0.0.1: the screenpipe recorder on port 3030 and an OpenAI-compatible local model server on 8001. The app sends no telemetry, and course content never leaves the machine. The only network traffic is the lecture itself streaming in.
Why does the SwiftUI app shell out to Python and zsh tools?
So each capability has one tested implementation. The app runs a tool with arguments, reads exactly one JSON document from stdout and streams stderr into a job log. The cost is 100 to 300 ms of process start-up per call, which is acceptable for builds and evals.
Can Lecture Lens run without a local model server?
Partly. Lecture Lens Studio runs Gemini Nano inside Chrome, with WebGPU models as a fallback. On the same evaluation suite it was close to the local model on answers and notes, but weaker on flashcards, so cards still use the local model.