Notes from Jeff Dean and Diana Hu on Building AI Systems

Watching videos with an LLM as a thinking companion

I’m experimenting with a simple learning workflow: watching long-form videos while keeping an LLM chat session open—currently Codex 5.6 Sol on High.

Instead of waiting until the end and asking for a generic summary, I drop rough thoughts into the chat as they occur. The LLM helps me preserve observations, clarify unfamiliar ideas, flag ambiguities, verify what was said, find related references, and organize everything into linked, timestamped notes.

So far, I’ve found this surprisingly useful. The LLM isn’t watching or thinking for me. It acts more like shared working memory: I decide what feels important, while it helps me preserve, sharpen, and connect those thoughts.

In a way, this feels like another iteration of an LLM wiki: a page assembled with an LLM, but guided by the moments I chose to notice, question, and connect.

I’m sharing my notes in case others find the ideas—or the workflow—useful. I might do this more often.

• Watch directly on YouTube

Automated experimentation by 2027

At 1:47, Diana asks Dean for a prediction about AI in 2027.

He expects much more automation of ML systems themselves: decomposing problems into subproblems, running the subproblems through tight automatic experimentation loops, putting the results back together, and producing an improved overall system. He thinks the same approach will apply beyond ML to other areas of science and engineering where progress can be measured.

TPUs: specialize around the important computation

At 7:47, Dean explains that TPUs specialize in low-precision dense linear algebra, which sits at the heart of modern machine-learning algorithms.

That means giving up general-purpose capability in exchange for much better performance on the operations ML actually needs. A TPU cannot run Chrome or Word, but the first version was dramatically more energy-efficient and lower-latency than the CPUs and GPUs of its time for ML inference.

“Low-precision dense linear algebra” was unfamiliar language to me. My working understanding is that it means doing enormous numbers of matrix operations using compact numeric formats, where most entries in the matrices contain useful values. Neural networks rely heavily on those operations, so specialized hardware can execute them very efficiently.

From latency numbers to energy efficiency

I’ve long associated Dean with Latency Numbers Every Programmer Should Know—the rough hierarchy of costs programmers use to reason about systems before building them. In this conversation, he pushes that same instinct toward energy.

Starting around 11:28, Dean talks about knowing how much energy individual operations, memory access, and communication between chips consume. At 12:08, the important imbalance becomes clear: doing the arithmetic can be extremely cheap, while moving the data into the processor can cost roughly a thousand times more energy.

That creates a push and pull. At 13:14, Dean explains that batching many examples or tokens together spreads the cost of moving data across more useful work. It saves energy and makes the hardware more efficient.

But batching works against latency: the system has to collect more work before processing it. Batch size one is better for responsiveness, but currently worse for efficiency. At 14:15, they discuss efficient batch-size-one training as an attractive goal. Dean then turns to inference, where low latency matters more, and says there is still a lot of room for specialized hardware that can loosen this trade-off.

AI as a compression problem

At 15:44, Diana describes the whole process of AI as a large compression problem. Her point is that to compress data well and then reconstruct it, you effectively need to understand it. Dean agrees: if you truly understand the data, you should be able to compress it very well.

I initially wasn’t sure whether Diana said lossy or lossless. The captions say “lossy,” and that also matches what I heard. After thinking it through, lossy seems to fit the idea better.

They aren’t describing a conventional compression-and-decompression task where success means reproducing the original bytes exactly. The interesting part is preserving patterns and semantic structure while discarding incidental detail, then using that understanding to reconstruct something useful or plausible.

Context engineering makes knowledge clearer

At 16:34, Dean describes the model as only one part of a system for solving interesting problems. The rest includes tools, retrieval, memory, previous results, and the ability to put relevant information into the model’s context.

Retrieved information is explicit and clear to the model. By contrast, knowledge learned during training comes from trillions of tokens blended together into what Dean calls a “soup.” That closely matches what I recently wrote on agentic context construction: retrieval is only one step in assembling the working context an agent needs to solve a problem.

It also reinforces the point I took from Your Agent Is Only as Smart as Its Harness. The model is only one component; the system around it changes the capability we actually experience.

This note-taking experiment feels like a small example of context engineering. Each observation makes the temporary working context clearer. I’m not training the model; I’m building a sharper shared representation of the ideas I care about in this session.

Agent skills and benchmark loops

At 19:14, Dean discusses improving an agent without changing its model weights. You can instead give the model better guidelines and reusable skills that teach it how to approach a class of problems or use particular tools.

His performance-engineering example at 20:31 follows a loop: measure the current performance, modify the code, rerun benchmarks, evaluate the change, and iterate.

Dean and Sanjay Ghemawat encoded much of the process they would use themselves into a skill so an agent could perform parts of this loop. At 21:43, Dean also mentions their freely available Performance Hints, which people have summarized and given to models to improve their performance-engineering reasoning. This remains on my to-read list.

Where can a two- or three-person team still win?

At 25:23, Diana asks where a two- or three-person team can still win when a company like Google builds across the entire stack.

Dean’s answer is specialization. General models try to do almost everything, which means their builders cannot devote equal attention to every domain. A small team can build something more accurate, delightful, and deeply adapted to a field it cares about.

My extension to the question: could the relevant unit become even smaller? Could one or two people assisted by agents now build what previously required a much larger startup? Agents might not merely reduce headcount; they might make some previously impractical companies viable.

Specification matters more with agents

At 31:55, Dean emphasizes the importance of clearly specifying what you want.

Agents can follow what you specify. Anything left unstated must be inferred, and the agent may infer something different from what you intended. Now that agents can do more of the implementation, the importance of defining the intended result has gone up.

Tests are useful because they provide clear, verifiable specifications. Dean uses language-to-language code translation as an example at 33:34: the original implementation and its tests provide a detailed behavioral specification that the agent can repeatedly check.

The limitation is that tests specify only what they measure. Incomplete tests can still reward the wrong implementation.

Taste: what problem deserves your time?

At 34:03, Diana asks what becomes scarce when founders can run hundreds of agents and have them write the code.

Dean’s answer is having excellent taste about what to ask the agents to work on. A researcher can have all the tools and techniques, but most of the battle is deciding which problem deserves their time.

Successfully solving an important problem is much more valuable than delightfully executing an excellent research investigation into a boring one. That high-level wisdom about what to work on becomes even more important as agents accelerate the execution.

I saw a concrete version of this in Anthropic’s AES experiment. The agent had the capability to do the research, but a human still had to keep it focused on the hard problem instead of accepting an easier target.

People will increasingly steer large amounts of AI-assisted computation. The essential human contribution is deciding what should be accomplished and communicating the essence of that intention clearly.

Make the experimental loop fast

At 42:39, Dean returns to the scientific loop: propose an experiment, implement what is needed to run it, evaluate the results, and iterate.

More problems are becoming suitable for automating this whole loop. The important advantage is not automation alone, but reducing the latency between an idea and evidence so that many more experiments can run. That is close to the learning loop Satya Nadella described: models helping people and institutions notice patterns, test judgments, encode knowledge, and compound what they learn.

An orchestration system could take a high-level objective, decompose it into subproblems, run an automated experimentation loop for each one, and combine the partial solutions into an answer to the larger problem. Dean sees this accelerating ML research, chip design, science, and engineering.

Knowledge distillation: rejected, then widely adopted

At 48:03, the conversation turns to knowledge distillation. I didn’t know Jeff Dean was one of the authors of the landmark 2015 paper, alongside Geoffrey Hinton and Oriol Vinyals:

• Distilling the Knowledge in a Neural Network

Distillation uses a large teacher model to train a smaller, more efficient student model. Dean recalls that the paper was rejected, with one reviewer calling it unlikely to have significant impact. The technique later became widely used across the industry.

They already believed it was important because increasingly capable large models would be expensive to serve. Distillation could transfer much of that capability into cheaper models suitable for more people and more domains, including speech and vision.

That feels even more relevant now. In The Great Token Arbitrage, I argued that the long-term advantage may come from lowering the cost per useful answer. With growing demand for GPUs, memory, and energy, distillation is one way to push that cost down, alongside quantization, sparsity, caching, batching, and specialized hardware.