AINPM.in / Work / Lecture Lens / Architecture
Case study · Architecture · 12 Nov 2026

Lecture Lens architecture: a class becomes study material without leaving the Mac

Lecture Lens is a pipeline of four local parts: a recorder that captures screen and audio, a Python toolkit that builds each lecture, a model server with one slot shared by the generator and the judge, and a SwiftUI app that starts and watches them. Every service listens on 127.0.0.1, so the only traffic is the lecture streaming in.

01 · Pipeline

Where does the data go?

Two ways in, one build, five files out. On narrow screens, scroll the diagram sideways.

Lecture Lens pipeline, from a class to notes and flashcardsVideo and audio stream in from the course portal or meeting service. Everything else happens inside a boundary on the Mac where traffic stays on 127.0.0.1. The browser lecture sends only audio to the screenpipe recorder because screen capture pauses on protected pages; the meeting app sends frames and audio. The lecture build reads the recorder's database, selects the session, transcribes, picks distinct slides, describes them and writes notes, cards and answers with citations, writing five files per lecture. It calls one local model server slot, or Gemini Nano in Chrome instead. The eval runner scores the outputs and swaps a judge model into the same slot, then back. The SwiftUI app launches the recorder, so permissions belong to it, and runs the build and evals.Course portalinternetMeeting serviceinternetYour Mac · traffic stays on 127.0.0.1 · nothing is uploadedstreamcallBrowserprotected playerMeeting applive classaudio onlyframes, audioscreenpipe recorderscreen frames + OCRApp Tap: one app's audioProtected page visible:screen paused, audio keptSQLite · :3030readslecture build1 select session2 transcribe audio3 pick distinct slides4 describe slides5 notes, cards, answersPython toolkitwritesper lecturetranscript.mdslides.mdnotes.mdcards.tsvmanifest.jsonprompts, outputsLocal model serverone slot: generatoror judge · :8001StudioGemini Nanoinside Chromeorscoreseval runnersummary.jsoneval runner swaps the judge in, then backLecture Lens.app (SwiftUI)launches the recorder, so macOS permissions belong to the app · runs every tool with arguments in, JSON outhealth lights come from frames actually written, not from what processes reportlaunchesruns buildruns evals
From a class to study material. The only arrows that cross the dashed boundary point in. Screen capture pauses on protected pages; the judge and the generator share one model slot.
02 · Decisions

Which decisions shape it?

Each one has a written record with the evidence and the alternatives considered.

DecisionWhyRecord
The app launches the recordermacOS gives recording permission to the app that starts the recorder. Started from a terminal, it inherits no grants and waits on a prompt nobody sees.ADR-002
The app calls the toolkit; it never re-implements itOne tested implementation of each capability. Every tool takes arguments, prints one JSON document and logs to stderr; spawning costs 100 to 300 ms per call.System design
Pause screen capture on protected pages, keep the audioAny open screen stream blanks DRM video; audio never does. The lecture stays watchable and the transcript still gets built.ADR-004
Hash first, then structural similarity, to find distinct slidesSlides from one template look alike to a hash. Counting changed grid cells raised recall on synthetic decks from 0.17–0.58 to 1.00.ADR-005
Describe slides with thinking offOn dense, high-resolution slides, thinking mode invented content in 3 of 5 checks; thinking off in none.ADR-001 (prototype)
One model slot shared by generator and judgeThe model server holds one large model. The eval runner swaps the judge in and always swaps back; evals refuse to start during a live class.Architecture
A model inside Chrome as a second engineNo model server or 20 to 40 GB of weights. Gemini Nano matched the local model on answers (89% vs 91% correct) but not on flashcards.ADR-006
Rewrite mode-owned settings before every startThe recorder keeps flags between runs. A site allowlist left from portal mode once made it skip every window in live mode.Architecture
03 · Tool contract

How do the app and the toolkit talk?

Arguments in, exactly one JSON document out on stdout, progress on stderr. Exit code 0 is success, 1 a domain failure (JSON still printed, with "ok": false), 2 a usage error.

CommandReturns
lecture doctor --jsonChecks for the database, frames, app audio, ffmpeg, the vision model and the vault, each with a fix
lecture list --jsonSessions with start, end, minutes, frames and whether they are built
lecture build <id> --jsonThe lecture folder and its manifest: inputs, models and prompt versions
study ask "…" --jsonThe answer, its numbered sources with timestamps, the model and the latency
Failure must be visible. Three of the worst faults were silent: frames skipped while the light stayed green, frame extraction returning nothing, and a restart waiting on a hidden permission prompt. The rule since then: compare attempts with results, and warn instead of returning an empty value.
04 · Trade-offs

What does each choice cost, and when would it change?

ChosenCostRevisit when
Tools called by the app, not rewritten in Swift100–300 ms process spawn per callA call becomes latency-critical (then add a daemon)
Polling health every 5–60 s, not pushA small CPU costThe recorder exposes the events needed
Append-only JSONL feedback, not SQLiteA full scan to readFeedback passes about 50,000 rows
Unsandboxed appNo App StoreNever, for this use: it launches processes and reads the recorder database

How well it works is on the case study's product bets, and how the judge itself was checked is in this builder's note.

05 · FAQ

Frequently asked questions

How does Lecture Lens keep everything on the Mac?

Every service it talks to listens on 127.0.0.1: the screenpipe recorder on port 3030 and an OpenAI-compatible local model server on 8001. The app sends no telemetry, and course content never leaves the machine. The only network traffic is the lecture itself streaming in.

Why does the SwiftUI app shell out to Python and zsh tools?

So each capability has one tested implementation. The app runs a tool with arguments, reads exactly one JSON document from stdout and streams stderr into a job log. The cost is 100 to 300 ms of process start-up per call, which is acceptable for builds and evals.

Can Lecture Lens run without a local model server?

Partly. Lecture Lens Studio runs Gemini Nano inside Chrome, with WebGPU models as a fallback. On the same evaluation suite it was close to the local model on answers and notes, but weaker on flashcards, so cards still use the local model.