David Abitbol, who leads AI operations at Revolut, shares data from 2,000 engineers on what AI tools did to coding time and lead time, and why self-hosting open-source GLM models made the biggest difference: about 60-70% faster delivery at roughly a tenth of what they were paying frontier labs. He covers the GPUs needed, the latency targets that keep engineers happy, the tuning that made the same hardware 88% faster, and how an EU-regulated bank keeps it compliant.
Chapters:
0:00 Intro
0:36 Who Revolut is
1:18 What AI did to 2,000 engineers' lead time
2:45 Why they started self-hosting
3:33 Coding isn't the bottleneck
4:24 What it takes to self-host GLM
5:46 Tuning: same GPUs, 88% faster
7:06 The parameters that matter
8:14 Production setup
9:23 For the CIO: compliance and safety
David Abitbol
https://www.revolut.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
11:47
David Abitbol, who leads AI operations at Revolut, shares data from 2,000 engineers on what AI tools did to coding time and lead time, and why self-hosting open-source GLM models made the biggest difference: about 60-70% faster delivery at roughly a tenth of what they were paying frontier labs. He covers the GPUs needed, the latency targets that keep engineers happy, the tuning that made the same hardware 88% faster, and how an EU-regulated bank keeps it compliant.
Chapters:
0:00 Intro
0:36 Who Revolut is
1:18 What AI did to 2,000 engineers' lead time
2:45 Why they started self-hosting
3:33 Coding isn't the bottleneck
4:24 What it takes to self-host GLM
5:46 Tuning: same GPUs, 88% faster
7:06 The parameters that matter
8:14 Production setup
9:23 For the CIO: compliance and safety
David Abitbol
https://www.revolut.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Temporal's State of Development survey found that 80.8% of engineers use agents daily, but only 49.1% run them in production. Developer advocate Shy Ruparel walks through the four gaps behind that number (usage vs. production, trust vs. reality, prototype vs. enterprise, seeing vs. recovering), then demos a $2,500 payout workflow with a local model that survives having its workers killed mid-flight, without losing state, repeating tool calls or re-spending tokens.
Chapters:
0:00 Intro
0:34 Finance runs on long-running workflows
2:05 The State of Development survey
3:16 Gap 1: Usage vs. production
4:33 Gap 2: Trust vs. reality
7:18 Gap 3: Prototype vs. enterprise
8:55 Gap 4: Seeing vs. recovering
10:53 Demo: a payout that survives a crash
13:06 Durable execution explained
18:12 What the demo solved
20:10 Who runs this in production
20:50 Wrap-up
Shy Ruparel
https://temporal.io
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
21:32
Temporal's State of Development survey found that 80.8% of engineers use agents daily, but only 49.1% run them in production. Developer advocate Shy Ruparel walks through the four gaps behind that number (usage vs. production, trust vs. reality, prototype vs. enterprise, seeing vs. recovering), then demos a $2,500 payout workflow with a local model that survives having its workers killed mid-flight, without losing state, repeating tool calls or re-spending tokens.
Chapters:
0:00 Intro
0:34 Finance runs on long-running workflows
2:05 The State of Development survey
3:16 Gap 1: Usage vs. production
4:33 Gap 2: Trust vs. reality
7:18 Gap 3: Prototype vs. enterprise
8:55 Gap 4: Seeing vs. recovering
10:53 Demo: a payout that survives a crash
13:06 Durable execution explained
18:12 What the demo solved
20:10 Who runs this in production
20:50 Wrap-up
Shy Ruparel
https://temporal.io
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Code generation is moving faster than ever, and the bottleneck has moved to review. Cole Gannaway, a solutions engineer at Qodo, shows how a one-line data model change that passes every test can quietly break web, mobile, CLI and MCP clients, and how a software graph of your repos reveals that blast radius. He covers turning tribal knowledge and PR comments into custom rules, bringing review context in before code is written, why the agent that wrote the code shouldn't grade it, and checking PRs against the original ticket.
Chapters:
0:00 Intro
0:59 Review is the new bottleneck
2:03 A one-line change that passes every check
3:35 The hidden blast radius
4:15 Building a software graph
4:56 Custom rules and tribal knowledge
5:47 Context before code is written
6:29 Don't let the writer grade its own code
8:20 Turning PR comments into rules
10:06 Q&A: catching the wrong intent
13:55 Wrap-up
Cole Gannaway
https://www.qodo.ai
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
14:51
Code generation is moving faster than ever, and the bottleneck has moved to review. Cole Gannaway, a solutions engineer at Qodo, shows how a one-line data model change that passes every test can quietly break web, mobile, CLI and MCP clients, and how a software graph of your repos reveals that blast radius. He covers turning tribal knowledge and PR comments into custom rules, bringing review context in before code is written, why the agent that wrote the code shouldn't grade it, and checking PRs against the original ticket.
Chapters:
0:00 Intro
0:59 Review is the new bottleneck
2:03 A one-line change that passes every check
3:35 The hidden blast radius
4:15 Building a software graph
4:56 Custom rules and tribal knowledge
5:47 Context before code is written
6:29 Don't let the writer grade its own code
8:20 Turning PR comments into rules
10:06 Q&A: catching the wrong intent
13:55 Wrap-up
Cole Gannaway
https://www.qodo.ai
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Coding agents are opening PRs faster than anyone can review them. Manish Kapur (Sonar) and Addy Osmani (Anthropic) discuss "light" vs. "dark" software factories, how Anthropic scaled from about 2,500 to 30,000 merged changes a day with Claude doing first-pass review backed by real verification evidence, new data from Sonar's State of Code survey, and why every loop in the factory needs independent, multi-layered verification. They close with a lightning round on labeling agent-written PRs and when it's safe to merge.
Chapters:
0:00 Intro
1:57 Throughput is up, merges are failing
2:49 Light vs. dark software factories
3:51 Coding is solved, confidence isn't
5:38 How Anthropic reviews 30,000 changes a day
8:29 State of Code survey findings
9:20 Where the next gains come from
11:38 Multi-layered verification
13:19 Controls, permissions and sandboxes
14:41 Lightning round
17:40 Takeaways
Manish Kapur
https://www.sonarsource.com
Addy Osmani
https://x.com/addyosmani
https://www.anthropic.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
18:53
Coding agents are opening PRs faster than anyone can review them. Manish Kapur (Sonar) and Addy Osmani (Anthropic) discuss "light" vs. "dark" software factories, how Anthropic scaled from about 2,500 to 30,000 merged changes a day with Claude doing first-pass review backed by real verification evidence, new data from Sonar's State of Code survey, and why every loop in the factory needs independent, multi-layered verification. They close with a lightning round on labeling agent-written PRs and when it's safe to merge.
Chapters:
0:00 Intro
1:57 Throughput is up, merges are failing
2:49 Light vs. dark software factories
3:51 Coding is solved, confidence isn't
5:38 How Anthropic reviews 30,000 changes a day
8:29 State of Code survey findings
9:20 Where the next gains come from
11:38 Multi-layered verification
13:19 Controls, permissions and sandboxes
14:41 Lightning round
17:40 Takeaways
Manish Kapur
https://www.sonarsource.com
Addy Osmani
https://x.com/addyosmani
https://www.anthropic.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Pattern-based SAST catches bad code, but not valid code with flawed intent: broken access control, business logic gaps and authentication holes. Taylor Luttrell-Williams of Sonar introduces the SonarQube Hunter Agent, which maps a repo, runs three specialist agents and adversarially validates every finding, and announces Sonar joining the OpenAI Daybreak cyber partner program. The live demo turns up 46 issues in a Python app, including a user who can make themselves admin and an MFA bypass.
Chapters:
0:00 Intro
0:46 Why SAST misses intent bugs
2:37 How Hunter Agent works
4:12 Sonar joins OpenAI Daybreak
5:44 Layering SAST and Hunter Agent
6:27 Demo: running a scan
8:29 46 findings across three classes
10:01 Inside a broken access control bug
12:26 Scheduled scans
13:12 Automatic remediation
Taylor Luttrell-Williams
https://www.sonarsource.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
14:10
Pattern-based SAST catches bad code, but not valid code with flawed intent: broken access control, business logic gaps and authentication holes. Taylor Luttrell-Williams of Sonar introduces the SonarQube Hunter Agent, which maps a repo, runs three specialist agents and adversarially validates every finding, and announces Sonar joining the OpenAI Daybreak cyber partner program. The live demo turns up 46 issues in a Python app, including a user who can make themselves admin and an MFA bypass.
Chapters:
0:00 Intro
0:46 Why SAST misses intent bugs
2:37 How Hunter Agent works
4:12 Sonar joins OpenAI Daybreak
5:44 Layering SAST and Hunter Agent
6:27 Demo: running a scan
8:29 46 findings across three classes
10:01 Inside a broken access control bug
12:26 Scheduled scans
13:12 Automatic remediation
Taylor Luttrell-Williams
https://www.sonarsource.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Most teams spin up a new open model with the default vLLM recipe and hope for the best. Runpod forward-deployed engineer Chris Vela shows how to choose configs from data instead, starting with the use case and concurrency, and demos BenchPod, a benchmarking dashboard built on vLLM that tests configs live without restarts. He runs Kimi K3 on a B300, shows how tuning max batch tokens cuts inter-token latency, and asks whether you need a frontier model at all.
Chapters:
0:00 Intro
0:29 What Runpod does
1:32 Everyone's guessing at vLLM configs
2:53 Version and CUDA gotchas
3:48 BenchPod: benchmarking on top of vLLM
6:07 Running Kimi K3 on a B300
7:40 Demo
10:01 Tuning max batch tokens
13:34 From R&D to production
14:21 Do you need the frontier model?
Chris Vela
https://www.runpod.io
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
16:29
Most teams spin up a new open model with the default vLLM recipe and hope for the best. Runpod forward-deployed engineer Chris Vela shows how to choose configs from data instead, starting with the use case and concurrency, and demos BenchPod, a benchmarking dashboard built on vLLM that tests configs live without restarts. He runs Kimi K3 on a B300, shows how tuning max batch tokens cuts inter-token latency, and asks whether you need a frontier model at all.
Chapters:
0:00 Intro
0:29 What Runpod does
1:32 Everyone's guessing at vLLM configs
2:53 Version and CUDA gotchas
3:48 BenchPod: benchmarking on top of vLLM
6:07 Running Kimi K3 on a B300
7:40 Demo
10:01 Tuning max batch tokens
13:34 From R&D to production
14:21 Do you need the frontier model?
Chris Vela
https://www.runpod.io
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
When an agent sends an invoice using your token, who actually made the request? Kamil Potrec of Keycard breaks down how agents get credentials today (personal tokens, static keys, over-privileged service accounts) and what proper agent auth requires: per-agent identity, delegation, short-lived scoped credentials, federation with legacy systems, policy and audit. He compares SDKs and gateways for keeping credentials out of an agent's reach, including the gateway tradeoffs.
Chapters:
0:00 Intro
0:43 How agents get credentials today
3:46 What's wrong with this picture
5:15 Using OAuth for agents
7:10 Why it gets hard at scale
8:21 What proper agent auth needs
9:52 One request, many steps
11:16 Why SDKs aren't enough
12:37 Gateways: keep credentials away from agents
14:55 Gateway tradeoffs
16:24 How Keycard approaches it
17:26 Takeaways
Kamil Potrec
https://keycard.ai
Agent Baseline: https://agentbaseline.org
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
18:35
When an agent sends an invoice using your token, who actually made the request? Kamil Potrec of Keycard breaks down how agents get credentials today (personal tokens, static keys, over-privileged service accounts) and what proper agent auth requires: per-agent identity, delegation, short-lived scoped credentials, federation with legacy systems, policy and audit. He compares SDKs and gateways for keeping credentials out of an agent's reach, including the gateway tradeoffs.
Chapters:
0:00 Intro
0:43 How agents get credentials today
3:46 What's wrong with this picture
5:15 Using OAuth for agents
7:10 Why it gets hard at scale
8:21 What proper agent auth needs
9:52 One request, many steps
11:16 Why SDKs aren't enough
12:37 Gateways: keep credentials away from agents
14:55 Gateway tradeoffs
16:24 How Keycard approaches it
17:26 Takeaways
Kamil Potrec
https://keycard.ai
Agent Baseline: https://agentbaseline.org
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Frontier AI has made every attacker a level more dangerous, and on average, exploits now land before the patch. Patrick Smyth of Chainguard covers this year's AI-assisted attacks and supply chain malware waves, why 30-day remediation windows no longer hold up, the four attack surfaces to cover (malware, known CVEs, zero days and "side doors" like CI/CD, developer laptops and agent skills), and what to do now: cooldown periods, faster CVE fixes, and signing everything with SLSA and Sigstore.
Chapters:
0:00 Intro
0:58 AI-powered attacks this year
2:33 Exploits now arrive before patches
3:45 Why 30-day remediation fails
4:59 A wave of supply chain malware
6:44 The eternal September
8:11 The core supply chain problem
9:12 What to do this year
11:19 Four attack surfaces
13:56 How Chainguard tackles each one
17:01 Sign everything: SLSA and Sigstore
19:16 Hardening agent skills
20:22 Wrap-up
Patrick Smyth
https://www.chainguard.dev
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
21:47
Frontier AI has made every attacker a level more dangerous, and on average, exploits now land before the patch. Patrick Smyth of Chainguard covers this year's AI-assisted attacks and supply chain malware waves, why 30-day remediation windows no longer hold up, the four attack surfaces to cover (malware, known CVEs, zero days and "side doors" like CI/CD, developer laptops and agent skills), and what to do now: cooldown periods, faster CVE fixes, and signing everything with SLSA and Sigstore.
Chapters:
0:00 Intro
0:58 AI-powered attacks this year
2:33 Exploits now arrive before patches
3:45 Why 30-day remediation fails
4:59 A wave of supply chain malware
6:44 The eternal September
8:11 The core supply chain problem
9:12 What to do this year
11:19 Four attack surfaces
13:56 How Chainguard tackles each one
17:01 Sign everything: SLSA and Sigstore
19:16 Hardening agent skills
20:22 Wrap-up
Patrick Smyth
https://www.chainguard.dev
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Arize's in-product agent Alex went from a 2023 prototype that was "50% hallucination" to a single agent with planning, memory and on-demand skills, all built with custom harness code instead of a framework. AI PM Fuad Ali covers each decision: moving the plan into the harness, making "finish" an explicit gate, treating large tool results like files, why one 8,000-character memory file per user beat vector stores and knowledge graphs, merging 10 specialists into one agent, and using production sessions as tests.
Chapters:
0:00 Intro
0:53 Agents all the way down
2:56 What Alex does
4:15 Timeline: 2023 to 2026
6:19 Who owns the plan?
8:33 What ends a turn?
9:44 Memory: treat big results like files
12:10 One small file beat a knowledge graph
13:33 Specialists vs. one agent
14:13 The newest model isn't always best
15:28 Production sessions as tests
17:46 What still needs a human
18:31 What survived every rebuild
Fuad Ali
https://arize.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
20:06
Arize's in-product agent Alex went from a 2023 prototype that was "50% hallucination" to a single agent with planning, memory and on-demand skills, all built with custom harness code instead of a framework. AI PM Fuad Ali covers each decision: moving the plan into the harness, making "finish" an explicit gate, treating large tool results like files, why one 8,000-character memory file per user beat vector stores and knowledge graphs, merging 10 specialists into one agent, and using production sessions as tests.
Chapters:
0:00 Intro
0:53 Agents all the way down
2:56 What Alex does
4:15 Timeline: 2023 to 2026
6:19 Who owns the plan?
8:33 What ends a turn?
9:44 Memory: treat big results like files
12:10 One small file beat a knowledge graph
13:33 Specialists vs. one agent
14:13 The newest model isn't always best
15:28 Production sessions as tests
17:46 What still needs a human
18:31 What survived every rebuild
Fuad Ali
https://arize.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Classic tool-calling agents fill their context with tool catalogs and raw outputs. Code mode gives the agent a sandbox and lets it search for and compose tools in code, returning only what it needs. Mikyo King, head of open source at Arize AI, explains why code mode works, how Arize built it into Phoenix (Bash, a Python sandbox, SQL and even UI control), its hidden costs (secrets, permissions, durability, double sandboxing), and how Harbor evals showed 43% fewer tokens and 31% lower cost.
Chapters:
0:00 Intro
0:36 The line between engineers and everyone else is dissolving
1:52 The classic tool-calling agent
3:03 What is code mode?
4:37 Why code mode works
6:11 How Phoenix uses code mode
8:19 The hidden costs of code mode
10:42 Open problems
11:44 Evaluating code mode with Harbor
12:55 Results: 43% fewer tokens, 31% lower cost
13:59 Takeaways
Mikyo King
https://arize.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
14:38
Classic tool-calling agents fill their context with tool catalogs and raw outputs. Code mode gives the agent a sandbox and lets it search for and compose tools in code, returning only what it needs. Mikyo King, head of open source at Arize AI, explains why code mode works, how Arize built it into Phoenix (Bash, a Python sandbox, SQL and even UI control), its hidden costs (secrets, permissions, durability, double sandboxing), and how Harbor evals showed 43% fewer tokens and 31% lower cost.
Chapters:
0:00 Intro
0:36 The line between engineers and everyone else is dissolving
1:52 The classic tool-calling agent
3:03 What is code mode?
4:37 Why code mode works
6:11 How Phoenix uses code mode
8:19 The hidden costs of code mode
10:42 Open problems
11:44 Evaluating code mode with Harbor
12:55 Results: 43% fewer tokens, 31% lower cost
13:59 Takeaways
Mikyo King
https://arize.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
In 2020, Citibank accidentally sent about $900 million to Revlon’s lenders instead of an $8 million interest payment, even though the procedure, the warning pop-up and three approvers all did what they were supposed to. John Feminella, who works on AI architecture and engineering at Two Sigma, uses that story to show how to give AI agents real autonomy without locking everything down. He walks through five composable controls: an agent registry, risk-based review, tool access that follows an agent’s declared purpose, shared firm-wide evals, and classifiers that send humans only the approvals that matter. Views are his own, not Two Sigma’s.
Chapters:
0:00 Intro
0:43 When controls fail
1:03 The $900M Revlon payment mistake
4:58 What this has to do with AI agents
5:22 Humans aren’t deterministic either
6:07 Approaches that don’t work
7:32 Control 1: An agent registry
9:47 Control 2: Risk-based review
11:47 Control 3: Tools follow purpose
14:42 Control 4: Shared evals
16:17 Control 5: Ending approval fatigue
18:07 Putting it all together
18:32 Why cars have brakes
John Feminella
https://www.twosigma.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
18:56
In 2020, Citibank accidentally sent about $900 million to Revlon’s lenders instead of an $8 million interest payment, even though the procedure, the warning pop-up and three approvers all did what they were supposed to. John Feminella, who works on AI architecture and engineering at Two Sigma, uses that story to show how to give AI agents real autonomy without locking everything down. He walks through five composable controls: an agent registry, risk-based review, tool access that follows an agent’s declared purpose, shared firm-wide evals, and classifiers that send humans only the approvals that matter. Views are his own, not Two Sigma’s.
Chapters:
0:00 Intro
0:43 When controls fail
1:03 The $900M Revlon payment mistake
4:58 What this has to do with AI agents
5:22 Humans aren’t deterministic either
6:07 Approaches that don’t work
7:32 Control 1: An agent registry
9:47 Control 2: Risk-based review
11:47 Control 3: Tools follow purpose
14:42 Control 4: Shared evals
16:17 Control 5: Ending approval fatigue
18:07 Putting it all together
18:32 Why cars have brakes
John Feminella
https://www.twosigma.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Nobody would hire a worker with no ID, no manager and shared passwords, yet that's how most companies deploy AI agents. Tim Tyler of Gravitee presents the Agent Accountability framework, which governs agents like a digital workforce in three stages: Assignment (identity, team, a named human sponsor), Alignment (role, mission and least-privilege authority) and Administration (oversight scorecards, guardian agents and step-by-step recourse when agents go off course).
Chapters:
0:00 The job posting nobody would accept
1:15 Agents as a digital workforce
2:27 Assignment: identity, team and sponsor
6:00 Alignment: role and mission
7:37 Authority and lowest scope
9:34 Never assume alignment
11:17 Administration: oversight scorecards
13:35 Agents need micromanagers
14:21 Recourse when agents go wrong
17:05 Recap
Tim Tyler
https://www.gravitee.io
Agent Accountability Framework: https://agentaccountabilityframework.org
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
18:13
Nobody would hire a worker with no ID, no manager and shared passwords, yet that's how most companies deploy AI agents. Tim Tyler of Gravitee presents the Agent Accountability framework, which governs agents like a digital workforce in three stages: Assignment (identity, team, a named human sponsor), Alignment (role, mission and least-privilege authority) and Administration (oversight scorecards, guardian agents and step-by-step recourse when agents go off course).
Chapters:
0:00 The job posting nobody would accept
1:15 Agents as a digital workforce
2:27 Assignment: identity, team and sponsor
6:00 Alignment: role and mission
7:37 Authority and lowest scope
9:34 Never assume alignment
11:17 Administration: oversight scorecards
13:35 Agents need micromanagers
14:21 Recourse when agents go wrong
17:05 Recap
Tim Tyler
https://www.gravitee.io
Agent Accountability Framework: https://agentaccountabilityframework.org
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Runlayer is a company of about 60 people running 300+ agents every day. Founding engineer Aiden Sochowski shares what they found once they put a gateway in front of everything and logged every session and tool call: prompt injection through email, PR-review agents getting talked into approvals, over-permissioned bots, one poisoned agent contaminating others, buggy vibe-coded MCP servers burning tokens, and a model switch that raised costs over 100x in a week.
Chapters:
0:00 Intro
0:25 Zero visibility into company agents
1:22 300 agents for 60 people
3:38 Example: data exfiltration through email
6:16 Attacks that unfold over a whole session
8:50 Using session data to scope agents down
10:56 One poisoned agent affects them all
12:47 Beyond security: broken MCP servers
14:08 Cost blowups from model switches
15:36 Wrap-up
Aiden Sochowski
https://runlayer.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
16:38
Runlayer is a company of about 60 people running 300+ agents every day. Founding engineer Aiden Sochowski shares what they found once they put a gateway in front of everything and logged every session and tool call: prompt injection through email, PR-review agents getting talked into approvals, over-permissioned bots, one poisoned agent contaminating others, buggy vibe-coded MCP servers burning tokens, and a model switch that raised costs over 100x in a week.
Chapters:
0:00 Intro
0:25 Zero visibility into company agents
1:22 300 agents for 60 people
3:38 Example: data exfiltration through email
6:16 Attacks that unfold over a whole session
8:50 Using session data to scope agents down
10:56 One poisoned agent affects them all
12:47 Beyond security: broken MCP servers
14:08 Cost blowups from model switches
15:36 Wrap-up
Aiden Sochowski
https://runlayer.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
At Optiver, a change rolled out to traders shows up in P&L within hours, so AI has to move fast and still be measurable. Noyan Tokgozoglu, head of AI Engineering, walks through Optiver's on-prem AI platform: an inference gateway for routing and budgets, same-day adoption of new frontier models benchmarked on internal golden sets, a model- and harness-agnostic "meta harness", and how they got AI spend back under control after the summer's big model releases.
Chapters:
0:00 Intro
0:22 What a market maker does
1:00 Feedback in hours, not quarters
2:53 Platform principles
5:11 The AI platform stack
7:06 Shipping frontier models the same day
8:31 Staying model and harness agnostic
9:53 Getting AI spend under control
13:54 Meeting users where they are
16:11 Q&A
Noyan Tokgozoglu
https://optiver.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
23:23
At Optiver, a change rolled out to traders shows up in P&L within hours, so AI has to move fast and still be measurable. Noyan Tokgozoglu, head of AI Engineering, walks through Optiver's on-prem AI platform: an inference gateway for routing and budgets, same-day adoption of new frontier models benchmarked on internal golden sets, a model- and harness-agnostic "meta harness", and how they got AI spend back under control after the summer's big model releases.
Chapters:
0:00 Intro
0:22 What a market maker does
1:00 Feedback in hours, not quarters
2:53 Platform principles
5:11 The AI platform stack
7:06 Shipping frontier models the same day
8:31 Staying model and harness agnostic
9:53 Getting AI spend under control
13:54 Meeting users where they are
16:11 Q&A
Noyan Tokgozoglu
https://optiver.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
In this workshop, Fuad Ali of Arize AI shows how to build a self-improvement loop for AI agents: trace everything, let Signal (an always-on agent) read your traces on a schedule and file deduplicated issues, turn failures into evals and living regression datasets, run experiments to prove a fix works, and keep humans at the highest-leverage points: eval design, dataset curation and reviewing fixes. He demos the loop on a toy-shopping agent with a planted bug and takes questions on evaluating qualitative output, decision models and cost.
Chapters
00:00 Intro and workshop setup
05:42 Where is your team on the maturity curve?
07:39 The loop: trace, signal, eval, experiment, fix
08:46 Demo agent and tracing
12:15 Signal: an always-on agent reading your traces
16:11 Agent Studio and managed agents
18:31 Turning failures into datasets
20:32 Evals: code, LLM as a judge, decision models
24:48 Creating an eval from a Signal issue
27:01 Experiments: proving the fix works
31:12 Keeping humans in the loop
34:16 Toward a fully automated loop
40:16 Q&A: evaluating qualitative output
48:21 Q&A: decision models, data sources and cost
Fuad Ali, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
53:05
In this workshop, Fuad Ali of Arize AI shows how to build a self-improvement loop for AI agents: trace everything, let Signal (an always-on agent) read your traces on a schedule and file deduplicated issues, turn failures into evals and living regression datasets, run experiments to prove a fix works, and keep humans at the highest-leverage points: eval design, dataset curation and reviewing fixes. He demos the loop on a toy-shopping agent with a planted bug and takes questions on evaluating qualitative output, decision models and cost.
Chapters
00:00 Intro and workshop setup
05:42 Where is your team on the maturity curve?
07:39 The loop: trace, signal, eval, experiment, fix
08:46 Demo agent and tracing
12:15 Signal: an always-on agent reading your traces
16:11 Agent Studio and managed agents
18:31 Turning failures into datasets
20:32 Evals: code, LLM as a judge, decision models
24:48 Creating an eval from a Signal issue
27:01 Experiments: proving the fix works
31:12 Keeping humans in the loop
34:16 Toward a fully automated loop
40:16 Q&A: evaluating qualitative output
48:21 Q&A: decision models, data sources and cost
Fuad Ali, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Xander Song, an engineer at Arize AI who works on Phoenix, introduces Harbor, an open source framework for evaluating agents on realistic tasks with real data in isolated sandboxes. He explains tasks, Docker environments, verifiers and rewards, trials and jobs, oracles that validate your setup before you spend tokens, and ATIF trajectories. Then he shows how his team runs Harbor at scale on Daytona and with Phoenix to test a prompt change that steered agents toward a new SQL tool, cutting tool calls without hurting correctness.
Chapters
00:00 Intro: shipping on evidence, not vibes
01:28 What Harbor is
03:37 Architecture: tasks, environments, verifiers
06:08 Trajectories, jobs and oracles
07:58 Example: fixing a failing migration
10:48 Verifiers as black-box integration tests
14:21 Running at scale in parallel sandboxes
17:04 Harbor with Phoenix
20:37 How the Phoenix team uses Harbor
25:07 Q&A: how Harbor orchestrates runs
29:24 Q&A: complex systems and data size
33:21 Case study: steering agents to a SQL tool
38:34 Demo: datasets and experiments in Phoenix
44:10 Demo: comparing jobs task by task
48:31 Regression tests and skills
Xander Song, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
56:01
Xander Song, an engineer at Arize AI who works on Phoenix, introduces Harbor, an open source framework for evaluating agents on realistic tasks with real data in isolated sandboxes. He explains tasks, Docker environments, verifiers and rewards, trials and jobs, oracles that validate your setup before you spend tokens, and ATIF trajectories. Then he shows how his team runs Harbor at scale on Daytona and with Phoenix to test a prompt change that steered agents toward a new SQL tool, cutting tool calls without hurting correctness.
Chapters
00:00 Intro: shipping on evidence, not vibes
01:28 What Harbor is
03:37 Architecture: tasks, environments, verifiers
06:08 Trajectories, jobs and oracles
07:58 Example: fixing a failing migration
10:48 Verifiers as black-box integration tests
14:21 Running at scale in parallel sandboxes
17:04 Harbor with Phoenix
20:37 How the Phoenix team uses Harbor
25:07 Q&A: how Harbor orchestrates runs
29:24 Q&A: complex systems and data size
33:21 Case study: steering agents to a SQL tool
38:34 Demo: datasets and experiments in Phoenix
44:10 Demo: comparing jobs task by task
48:31 Regression tests and skills
Xander Song, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Daksh Gupta, co-founder of Greptile, which reviews about 1% of all pull requests on GitHub, shares what Greptile tells companies that want to build their own review agent. Starting from a simple sandbox-plus-webhook setup, he explains why it misses bugs, costs $3-25 per PR and floods you with nitpicks, then walks through the fixes: targeted prompts, a dataset of past incident PRs, model inversion (reviewing Claude-written code with GPT and vice versa catches about 10% more bugs), codebase context and a living wiki, autonomous testing in sandboxes with computer use, and routing cheaper models to most subtasks.
Chapters
00:00 Intro: Greptile and why companies build their own
01:22 What makes a great review agent
02:38 The simple version: sandbox, agent, webhook
03:55 What goes wrong
06:02 Fix 1: Better prompts and fewer nitpicks
07:44 Fix 2: A dataset of historical PRs
09:02 Fix 3: Model inversion
11:04 Fix 4: Codebase context
12:36 Docs, tickets and a living wiki
14:26 Fix 5: Autonomous testing with computer use
16:05 Fix 6: Cheaper models for most subtasks
Daksh Gupta, Greptile
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
18:27
Daksh Gupta, co-founder of Greptile, which reviews about 1% of all pull requests on GitHub, shares what Greptile tells companies that want to build their own review agent. Starting from a simple sandbox-plus-webhook setup, he explains why it misses bugs, costs $3-25 per PR and floods you with nitpicks, then walks through the fixes: targeted prompts, a dataset of past incident PRs, model inversion (reviewing Claude-written code with GPT and vice versa catches about 10% more bugs), codebase context and a living wiki, autonomous testing in sandboxes with computer use, and routing cheaper models to most subtasks.
Chapters
00:00 Intro: Greptile and why companies build their own
01:22 What makes a great review agent
02:38 The simple version: sandbox, agent, webhook
03:55 What goes wrong
06:02 Fix 1: Better prompts and fewer nitpicks
07:44 Fix 2: A dataset of historical PRs
09:02 Fix 3: Model inversion
11:04 Fix 4: Codebase context
12:36 Docs, tickets and a living wiki
14:26 Fix 5: Autonomous testing with computer use
16:05 Fix 6: Cheaper models for most subtasks
Daksh Gupta, Greptile
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Jack Wotherspoon, developer experience engineer at Google DeepMind, introduces Gemini Managed Agents: coding agents that run in cloud sandboxes from a single API call instead of on your laptop. He covers the Interactions API, the Antigravity harness, environments that several agents can share, uploaded skills and AGENTS.md files, a proxy that keeps secrets out of the sandbox, scheduled triggers and multi-agent workflows. Demos include an SEC filing researcher handing its work to a quant agent, and a stock research app built in AI Studio.
Chapters
00:00 Intro: what are managed agents?
00:54 Why run agents in the cloud
02:13 Doing it yourself vs one API call
03:26 The Interactions API
04:32 How it works under the hood
05:22 Building blocks: harness, environment, skills, credentials, triggers
07:02 Choosing models, including Gemini 4 Argon
08:51 Interactions and environments
10:46 Tools, MCP servers and file mounts
12:21 Keeping secrets out of the sandbox
13:12 Triggers and scheduled agents
13:37 Multi-agent workflows
14:32 Demo: SEC research to quant analysis to report
16:34 Demo: managed agents inside an app
17:44 Getting started in AI Studio
Jack Wotherspoon, Google DeepMind
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
19:44
Jack Wotherspoon, developer experience engineer at Google DeepMind, introduces Gemini Managed Agents: coding agents that run in cloud sandboxes from a single API call instead of on your laptop. He covers the Interactions API, the Antigravity harness, environments that several agents can share, uploaded skills and AGENTS.md files, a proxy that keeps secrets out of the sandbox, scheduled triggers and multi-agent workflows. Demos include an SEC filing researcher handing its work to a quant agent, and a stock research app built in AI Studio.
Chapters
00:00 Intro: what are managed agents?
00:54 Why run agents in the cloud
02:13 Doing it yourself vs one API call
03:26 The Interactions API
04:32 How it works under the hood
05:22 Building blocks: harness, environment, skills, credentials, triggers
07:02 Choosing models, including Gemini 4 Argon
08:51 Interactions and environments
10:46 Tools, MCP servers and file mounts
12:21 Keeping secrets out of the sandbox
13:12 Triggers and scheduled agents
13:37 Multi-agent workflows
14:32 Demo: SEC research to quant analysis to report
16:34 Demo: managed agents inside an app
17:44 Getting started in AI Studio
Jack Wotherspoon, Google DeepMind
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Jesse Hall of LiveKit shows how to build voice agents for financial services that feel human, not robotic. He explains why model leaderboards mislead (they score components, but you ship a conversation), why you should measure time to first audio and time to first sentence instead of first byte or first token, and how async tool calls keep a conversation flowing while slow back-end work runs. He also covers audio-aware turn detection, the roughly 1.5-second latency budget (500-600ms feels truly human), and benchmarking your own stack with simulated callers.
Chapters
00:00 Intro: real-time agents vs background agents
01:47 Humanlike voice agents for financial services
03:46 LiveKit and the voice pipeline
04:21 Cascaded pipelines vs speech-to-speech
05:05 Why leaderboards don't pick your stack
06:46 Time to first audio, not first byte
07:52 Time to first sentence, not first token
08:38 Hiding slow tool calls like a human would
10:14 Demo: hotel booking agent
12:33 Turn detection: knowing when to listen
14:02 The latency budget
15:44 Benchmarking your stack with simulated callers
18:35 Results and a pre-launch checklist
Jesse Hall, LiveKit
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
20:55
Jesse Hall of LiveKit shows how to build voice agents for financial services that feel human, not robotic. He explains why model leaderboards mislead (they score components, but you ship a conversation), why you should measure time to first audio and time to first sentence instead of first byte or first token, and how async tool calls keep a conversation flowing while slow back-end work runs. He also covers audio-aware turn detection, the roughly 1.5-second latency budget (500-600ms feels truly human), and benchmarking your own stack with simulated callers.
Chapters
00:00 Intro: real-time agents vs background agents
01:47 Humanlike voice agents for financial services
03:46 LiveKit and the voice pipeline
04:21 Cascaded pipelines vs speech-to-speech
05:05 Why leaderboards don't pick your stack
06:46 Time to first audio, not first byte
07:52 Time to first sentence, not first token
08:38 Hiding slow tool calls like a human would
10:14 Demo: hotel booking agent
12:33 Turn detection: knowing when to listen
14:02 The latency budget
15:44 Benchmarking your stack with simulated callers
18:35 Results and a pre-launch checklist
Jesse Hall, LiveKit
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
A hands-on workshop from SallyAnn DeLucia, head of product at Arize AI, on moving from vibe checks to real evals. She covers what an eval is and how it differs from a unit test, the main types (LLM as a judge, decision models, code evals, annotations, agent as a judge), how to read a trace, how to write concrete pass/fail criteria, and how to build and iterate on an LLM judge in Arize's prompt playground.
Chapters
00:00 Intro and agenda
02:06 Why vibe checks don't scale
03:08 Is an eval just a unit test?
04:13 Why agents make evals harder
05:15 Types of evals
07:29 Workshop setup
10:16 Exploring the travel agent's traces
14:20 How to read a trace
19:03 Defining what good means
22:02 Writing pass/fail criteria
25:19 Building a dataset from traces
26:53 Building and testing an LLM judge
32:08 Next steps: alignment, online evals, Signal
SallyAnn DeLucia, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
35:20
A hands-on workshop from SallyAnn DeLucia, head of product at Arize AI, on moving from vibe checks to real evals. She covers what an eval is and how it differs from a unit test, the main types (LLM as a judge, decision models, code evals, annotations, agent as a judge), how to read a trace, how to write concrete pass/fail criteria, and how to build and iterate on an LLM judge in Arize's prompt playground.
Chapters
00:00 Intro and agenda
02:06 Why vibe checks don't scale
03:08 Is an eval just a unit test?
04:13 Why agents make evals harder
05:15 Types of evals
07:29 Workshop setup
10:16 Exploring the travel agent's traces
14:20 How to read a trace
19:03 Defining what good means
22:02 Writing pass/fail criteria
25:19 Building a dataset from traces
26:53 Building and testing an LLM judge
32:08 Next steps: alignment, online evals, Signal
SallyAnn DeLucia, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Aparna Dhinakaran of Arize AI revisits what an AI engineer actually does as the field moves from writing features to running factories of agents. The job now is building the loop: evals that review non-deterministic work, telemetry that agents consume, and agents that act on it. She walks through how Arize built Signal, an always-on agent that reads traces, remembers past issues and opens PRs, covering triggers, sandboxes, durable state, and evaluating the evaluators.
Chapters
00:00 Intro: what does an AI engineer do?
00:38 Karpathy's funnel: who's running thousands of agents
01:45 Rising abstraction, from code to agent factories
02:23 Then vs now: build the loop, not the feature
03:31 Signal: an always-on agent for your traces
04:45 Agents are now the main consumers of telemetry
05:19 What triggers an always-on agent
06:01 Inside a run: context, sandbox, proxy
07:11 What persists and what gets torn down
08:13 Designing durable agents
09:14 Model and harness agnostic
10:21 Writing evals, and evaluating the evaluators
12:48 Trace everything, including your coding agents
14:17 Closing the gap between labs and businesses
Aparna Dhinakaran, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
15:28
Aparna Dhinakaran of Arize AI revisits what an AI engineer actually does as the field moves from writing features to running factories of agents. The job now is building the loop: evals that review non-deterministic work, telemetry that agents consume, and agents that act on it. She walks through how Arize built Signal, an always-on agent that reads traces, remembers past issues and opens PRs, covering triggers, sandboxes, durable state, and evaluating the evaluators.
Chapters
00:00 Intro: what does an AI engineer do?
00:38 Karpathy's funnel: who's running thousands of agents
01:45 Rising abstraction, from code to agent factories
02:23 Then vs now: build the loop, not the feature
03:31 Signal: an always-on agent for your traces
04:45 Agents are now the main consumers of telemetry
05:19 What triggers an always-on agent
06:01 Inside a run: context, sandbox, proxy
07:11 What persists and what gets torn down
08:13 Designing durable agents
09:14 Model and harness agnostic
10:21 Writing evals, and evaluating the evaluators
12:48 Trace everything, including your coding agents
14:17 Closing the gap between labs and businesses
Aparna Dhinakaran, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
An AI analyst at a firm like Bridgewater needs private documents, but every email, memo and research note carries its own permissions. Aaron Linsky (head of product engineering, Bridgewater Associates) explains why the Vault's 100 million permissioned documents broke their first search setup, and Nikhil Benesch (CTO, turbopuffer) breaks down the fix: post-filtering loses recall, pre-filtering is too slow, and "native filtering" over a clustered vector index gets both, delivering about 120ms P90 searches across 400 million vectors.
Chapters
00:00 Intro: why permissions matter for AI analysts
02:07 Bridgewater's research process
03:43 The Vault: 100M documents, many permissions
05:14 Modeling permissions, and where it broke
06:25 What turbopuffer is
08:01 Napkin math: why brute force fails at scale
09:43 How vector indexes work
11:50 Permissions break the index
12:19 Post-filtering vs pre-filtering
14:09 Native filtering: the best of both
15:23 Results: 120ms across 100M documents
15:47 Writing permission filters in practice
17:00 Beyond permissions: isolation-native search
Aaron Linsky, Bridgewater Associates
Nikhil Benesch, turbopuffer
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
17:57
An AI analyst at a firm like Bridgewater needs private documents, but every email, memo and research note carries its own permissions. Aaron Linsky (head of product engineering, Bridgewater Associates) explains why the Vault's 100 million permissioned documents broke their first search setup, and Nikhil Benesch (CTO, turbopuffer) breaks down the fix: post-filtering loses recall, pre-filtering is too slow, and "native filtering" over a clustered vector index gets both, delivering about 120ms P90 searches across 400 million vectors.
Chapters
00:00 Intro: why permissions matter for AI analysts
02:07 Bridgewater's research process
03:43 The Vault: 100M documents, many permissions
05:14 Modeling permissions, and where it broke
06:25 What turbopuffer is
08:01 Napkin math: why brute force fails at scale
09:43 How vector indexes work
11:50 Permissions break the index
12:19 Post-filtering vs pre-filtering
14:09 Native filtering: the best of both
15:23 Results: 120ms across 100M documents
15:47 Writing permission filters in practice
17:00 Beyond permissions: isolation-native search
Aaron Linsky, Bridgewater Associates
Nikhil Benesch, turbopuffer
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Some of the most important financial data sits behind logins and anti-bot walls, with no API. Jay Sahnan and Arik Bird of Browserbase show how browser agents solve this through three case studies: Ramp fetching receipts at scale (over 30,000 manual hours saved), Bill paying across thousands of different vendor portals with one agent, and Stripe automating KYC checks. They also cover how Browserbase forked Chromium to run reliably in the cloud, and how Stagehand's act, extract and observe primitives keep automations from breaking when sites change.
Chapters
00:00 Intro
00:27 What is a browser agent?
01:37 Ramp, Bill and Stripe: data on the hardest-to-reach web
02:13 Ramp: fetching receipts at scale
03:34 Why browsers get blocked in the data center
06:03 Forking Chromium for the cloud
07:49 Demo: 100 browsers at once
10:11 Bill: one agent, thousands of portals
11:08 Stagehand vs Playwright
14:52 Stripe: automating KYC
15:52 Act, extract and observe
16:35 Demo: researching 20 companies in parallel
Jay Sahnan, Browserbase
Arik Bird, Browserbase
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
19:38
Some of the most important financial data sits behind logins and anti-bot walls, with no API. Jay Sahnan and Arik Bird of Browserbase show how browser agents solve this through three case studies: Ramp fetching receipts at scale (over 30,000 manual hours saved), Bill paying across thousands of different vendor portals with one agent, and Stripe automating KYC checks. They also cover how Browserbase forked Chromium to run reliably in the cloud, and how Stagehand's act, extract and observe primitives keep automations from breaking when sites change.
Chapters
00:00 Intro
00:27 What is a browser agent?
01:37 Ramp, Bill and Stripe: data on the hardest-to-reach web
02:13 Ramp: fetching receipts at scale
03:34 Why browsers get blocked in the data center
06:03 Forking Chromium for the cloud
07:49 Demo: 100 browsers at once
10:11 Bill: one agent, thousands of portals
11:08 Stagehand vs Playwright
14:52 Stripe: automating KYC
15:52 Act, extract and observe
16:35 Demo: researching 20 companies in parallel
Jay Sahnan, Browserbase
Arik Bird, Browserbase
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
In finance, you never show a figure you can't trace. Vinoo Ganesh, CEO of Kepler (previously Palantir and Citadel), explains Kepler's architecture where models cite numbers but never write them: retrieval finds the source, deterministic code does the math, and a post-processing filter strips any figure that can't be verified. He covers the citation registry built from more than 30 million SEC documents, hybrid search and re-ranking on turbopuffer, tenant isolation, and why plain RAG produced thousands of wrong numbers in their evals.
Chapters
00:00 Intro
01:47 Why LLMs and financial numbers don't mix
04:02 Models cite numbers, code does the math
04:30 Building a citation registry from SEC filings
06:03 How Kepler answers a question end to end
08:27 Structuring facts: value, unit, scale, period
09:48 Hybrid search and re-ranking on turbopuffer
11:30 Tenant isolation and security
12:37 Making every number traceable
14:42 Stripping what can't be verified
16:41 What plain RAG gets wrong
18:32 Kepler in action
19:46 Where the guarantees end
Vinoo Ganesh, Kepler
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
20:58
In finance, you never show a figure you can't trace. Vinoo Ganesh, CEO of Kepler (previously Palantir and Citadel), explains Kepler's architecture where models cite numbers but never write them: retrieval finds the source, deterministic code does the math, and a post-processing filter strips any figure that can't be verified. He covers the citation registry built from more than 30 million SEC documents, hybrid search and re-ranking on turbopuffer, tenant isolation, and why plain RAG produced thousands of wrong numbers in their evals.
Chapters
00:00 Intro
01:47 Why LLMs and financial numbers don't mix
04:02 Models cite numbers, code does the math
04:30 Building a citation registry from SEC filings
06:03 How Kepler answers a question end to end
08:27 Structuring facts: value, unit, scale, period
09:48 Hybrid search and re-ranking on turbopuffer
11:30 Tenant isolation and security
12:37 Making every number traceable
14:42 Stripping what can't be verified
16:41 What plain RAG gets wrong
18:32 Kepler in action
19:46 Where the guarantees end
Vinoo Ganesh, Kepler
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Janie Gu, product manager at Google DeepMind leading Kaggle Benchmarks, explains why most AI benchmarks are broken: they don't reflect real work, they're expensive to keep current, and their scores can't be verified. After reproducing 65 industry benchmarks, Kaggle found models reading other models' API keys, rewriting grading scripts, and scores swinging up to 20% between identical runs. Suyeol Yun, head of AI at LinqAlpha, then presents MandateBench, built with Google DeepMind and Kaggle, which grades whether a model can work like a buy-side analyst, from a fund's mandate to a full investment memo.
Chapters
00:00 Intro
01:10 Kaggle: from grading humans to grading models
02:52 Why benchmarks are broken
05:25 What reproducing 65 benchmarks revealed
08:20 How Google DeepMind and Kaggle are fixing it
10:20 Why your company should build a benchmark
11:44 LinqAlpha and MandateBench
12:32 Where existing finance benchmarks fall short
13:31 A real analyst task, start to finish
15:33 Grading the reasoning, not just the answer
16:41 Early results
17:18 What's next and how to get involved
Janie Gu, Google DeepMind
Suyeol Yun, LinqAlpha
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
18:32
Janie Gu, product manager at Google DeepMind leading Kaggle Benchmarks, explains why most AI benchmarks are broken: they don't reflect real work, they're expensive to keep current, and their scores can't be verified. After reproducing 65 industry benchmarks, Kaggle found models reading other models' API keys, rewriting grading scripts, and scores swinging up to 20% between identical runs. Suyeol Yun, head of AI at LinqAlpha, then presents MandateBench, built with Google DeepMind and Kaggle, which grades whether a model can work like a buy-side analyst, from a fund's mandate to a full investment memo.
Chapters
00:00 Intro
01:10 Kaggle: from grading humans to grading models
02:52 Why benchmarks are broken
05:25 What reproducing 65 benchmarks revealed
08:20 How Google DeepMind and Kaggle are fixing it
10:20 Why your company should build a benchmark
11:44 LinqAlpha and MandateBench
12:32 Where existing finance benchmarks fall short
13:31 A real analyst task, start to finish
15:33 Grading the reasoning, not just the answer
16:41 Early results
17:18 What's next and how to get involved
Janie Gu, Google DeepMind
Suyeol Yun, LinqAlpha
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Laurie Voss, head of developer relations at Arize AI and co-founder of npm, argues that coding agents made writing code nearly free, so the bottleneck keeps moving: first to code review, then to operations and security. What agents can't take over is finding out what people actually need and defining what "good" means. He calls the job that remains the product engineer, looks at the data on collapsing junior hiring, and lays out what engineering leaders should do now.
Chapters
00:00 Intro: beyond the AI engineer
01:19 Code got free, so the bottleneck moved
03:04 8x more code, only 30% faster shipping
04:39 Can machines do code review?
08:02 The theory of constraints
09:00 The next bottlenecks: ops and security
11:21 What agents won't eat: deciding what good means
13:45 Essential complexity and the spec
16:34 The product engineer
19:45 The junior developer problem
22:57 For people who love the craft
25:11 What to do now
Laurie Voss, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
27:09
Laurie Voss, head of developer relations at Arize AI and co-founder of npm, argues that coding agents made writing code nearly free, so the bottleneck keeps moving: first to code review, then to operations and security. What agents can't take over is finding out what people actually need and defining what "good" means. He calls the job that remains the product engineer, looks at the data on collapsing junior hiring, and lays out what engineering leaders should do now.
Chapters
00:00 Intro: beyond the AI engineer
01:19 Code got free, so the bottleneck moved
03:04 8x more code, only 30% faster shipping
04:39 Can machines do code review?
08:02 The theory of constraints
09:00 The next bottlenecks: ops and security
11:21 What agents won't eat: deciding what good means
13:45 Essential complexity and the spec
16:34 The product engineer
19:45 The junior developer problem
22:57 For people who love the craft
25:11 What to do now
Laurie Voss, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
A one-line prompt change made a support agent 17 points safer and 17 points worse at finishing tasks. Sofia Jakovcevic, forward deployed engineer at Arize AI, explains why agents fail at runtime rather than in code, and walks through a five-step regression loop built from production traces: curate datasets, score with evaluators, run baseline experiments, compare changes, and gate merges in CI. Then she shows how to close the loop at agent speed, with coding agents that read traces and fix themselves.
Chapters
00:00 Intro: observability for agents
01:08 The real problem: fixes that break other things
02:26 A one-line prompt change: +17 and -17
04:48 Agents fail at runtime, not in code
06:25 The five-step regression loop
06:48 Step 1: Build datasets from production traces
07:50 Step 2: Score with evaluators
09:48 Steps 3-4: Baseline experiments and comparisons
11:14 Step 5: Gate merges in CI
11:48 Demo: experiments in Arise
15:38 Using coding agents and skills
16:12 Closing the loop at agent speed
Sofia Jakovcevic, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
19:25
A one-line prompt change made a support agent 17 points safer and 17 points worse at finishing tasks. Sofia Jakovcevic, forward deployed engineer at Arize AI, explains why agents fail at runtime rather than in code, and walks through a five-step regression loop built from production traces: curate datasets, score with evaluators, run baseline experiments, compare changes, and gate merges in CI. Then she shows how to close the loop at agent speed, with coding agents that read traces and fix themselves.
Chapters
00:00 Intro: observability for agents
01:08 The real problem: fixes that break other things
02:26 A one-line prompt change: +17 and -17
04:48 Agents fail at runtime, not in code
06:25 The five-step regression loop
06:48 Step 1: Build datasets from production traces
07:50 Step 2: Score with evaluators
09:48 Steps 3-4: Baseline experiments and comparisons
11:14 Step 5: Gate merges in CI
11:48 Demo: experiments in Arise
15:38 Using coding agents and skills
16:12 Closing the loop at agent speed
Sofia Jakovcevic, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Bridgewater Associates built the Vault: more than 100 million documents from 135 external sources plus the firm's own memos and meetings, cleaned, permissioned and tagged to its internal ontology. Sam Green (team lead, Applied AI) and Aaron Linsky (head of product engineering) share three lessons: humans and agents need different things from search, the biggest wins come from one tool that serves both, and the real prize is agents that proactively bring investors what deserves their attention. Includes how a hand-built benchmark took precision at 10 from about 60% to over 90%.
Chapters
00:00 Intro: three lessons from building the Vault
01:11 Bridgewater's 50-year research process
02:35 Why unstructured data needed the same rigor
03:25 What's in the Vault
04:14 Open web vs the Vault: a side-by-side
06:53 Search for humans vs search for agents
08:19 Benchmarking precision: "Vault search is bad"
09:32 Fast filtering at scale
11:35 Getting investors to change their habits
13:02 What the Vault enables: analysts, forecasts, signals
14:05 Proactive agents: the investor workbench
15:44 How the Vault compounds what the firm learns
Sam Green, Bridgewater Associates
Aaron Linsky, Bridgewater Associates
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
16:45
Bridgewater Associates built the Vault: more than 100 million documents from 135 external sources plus the firm's own memos and meetings, cleaned, permissioned and tagged to its internal ontology. Sam Green (team lead, Applied AI) and Aaron Linsky (head of product engineering) share three lessons: humans and agents need different things from search, the biggest wins come from one tool that serves both, and the real prize is agents that proactively bring investors what deserves their attention. Includes how a hand-built benchmark took precision at 10 from about 60% to over 90%.
Chapters
00:00 Intro: three lessons from building the Vault
01:11 Bridgewater's 50-year research process
02:35 Why unstructured data needed the same rigor
03:25 What's in the Vault
04:14 Open web vs the Vault: a side-by-side
06:53 Search for humans vs search for agents
08:19 Benchmarking precision: "Vault search is bad"
09:32 Fast filtering at scale
11:35 Getting investors to change their habits
13:02 What the Vault enables: analysts, forecasts, signals
14:05 Proactive agents: the investor workbench
15:44 How the Vault compounds what the firm learns
Sam Green, Bridgewater Associates
Aaron Linsky, Bridgewater Associates
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Harneet Kaur, founder and CEO of Axim Intelligence and a former distressed credit investor, tests whether frontier models can reason about a live distressed situation: iHeartMedia. Given the same context, two top models disagreed sharply on the odds of bankruptcy versus a negotiated restructuring. Breaking the analysis into checked layers (extraction, derivation, judgment, prediction) brought them to the same answer, which raises a new question: is agreement the same as being right?
Chapters
00:00 Intro
00:42 What distressed credit actually means
02:34 Why old scoring models fall short
04:30 What it means to understand a credit
05:46 Can AI predict what happens next?
06:13 Case study: iHeartMedia, and two models disagree
08:01 Benchmarking each layer of reasoning
11:07 Structured reasoning makes the models converge
12:03 Takeaways: convergence vs correctness
Harneet Kaur, Axim Intelligence
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
14:15
Harneet Kaur, founder and CEO of Axim Intelligence and a former distressed credit investor, tests whether frontier models can reason about a live distressed situation: iHeartMedia. Given the same context, two top models disagreed sharply on the odds of bankruptcy versus a negotiated restructuring. Breaking the analysis into checked layers (extraction, derivation, judgment, prediction) brought them to the same answer, which raises a new question: is agreement the same as being right?
Chapters
00:00 Intro
00:42 What distressed credit actually means
02:34 Why old scoring models fall short
04:30 What it means to understand a credit
05:46 Can AI predict what happens next?
06:13 Case study: iHeartMedia, and two models disagree
08:01 Benchmarking each layer of reasoning
11:07 Structured reasoning makes the models converge
12:03 Takeaways: convergence vs correctness
Harneet Kaur, Axim Intelligence
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
