Agentic + AI Observability Meetup Seattle

Join us for the Agentic + AI Observability Meetup: AI Governance Edition on Thursday, October 22 from 5:00 PM – 8:00 PM PST at the Databricks Bellevue office! This evening is dedicated to engineering leaders, ML practitioners, and AI founders focused on building production-grade agents with robust governance, cost controls, and quality guarantees.

Deploying AI agents to production demands strong governance, intelligent task routing, and continuous evaluation. We'll explore how to get maximum performance out of every dollar, avoid model mismatch, and prevent cost spirals without slowing down developer velocity.

Whether you’re scaling an early-stage startup or managing enterprise AI workloads, join us to learn practical strategies for governing, observing, and optimizing agentic systems.

Why you should attend:

• Control AI Costs & Token Overhead: Learn how to spot high-value models, route tasks dynamically based on complexity, and manage developer usage through unified gateways.

• Master Per-Task Evals: Discover why single-model setups fail in production and how to build task-specific evaluation harnesses that match real-world traces.

• Observe & Govern Production Agents: Go beyond simple logging with structured tracing, production evaluation metrics, and governance frameworks that keep agents reliable.

• Connect with AI Builders: Network with engineers, founders, and practitioners building scalable, well-governed AI architectures.

Previous series speakers include Databricks, Anthropic, OpenAI, SpaceXAI and more.

Agenda

• 5:00pm: Registration/Mingling

• 6:00pm: Welcome Remarks

• 6:15pm: Talk #1: More Productivity, Less Cost: Chasing the AI Efficiency Frontier (Ahmed Bilal, Product Lead, Coding Agent Inference & Governance, Databricks)

• 6:45pm: Talk #2: The customer is probably wrong: How to find the right model to use for your Agents using Per-Task Evals (Dheeraj Bandaru, Product Manager, Arize AI)

• 7:15pm: Mingling with bites + dessert

• 8:00pm: Night Ends

Session Descriptions:

• More Productivity, Less Cost: Chasing the AI Efficiency Frontier

  • AI coding is taking off, and so is the bill. How do you give developers powerful tools without letting costs spiral? The opportunity lies in the efficiency frontier: getting more useful work from every dollar as models improve. Drawing on Databricks’ experience, this talk explores how to spot better-value models, route tasks to the most cost-effective model that can handle the job, and cut hidden token overhead. See how Unity Gateway brings these techniques together, and leave with a practical playbook for keeping developers moving fast and AI costs under control.

• The customer is probably wrong: How to find the right model to use for your Agents using Per-Task Evals

  • Most teams pick one model for their agent and keep it. On our production agent, 72% of scored tasks ran on a model that wasn’t the best performer for that task, with gaps of up to 41 points in pass rate. No model led on every task, and our own benchmark’s picks disagreed with production in 11 of 12 task families. This talk covers how we found the right model for each task: grouping production traces into tasks, writing an eval for each, benchmarking models through the agent’s real harness, and checking the results against live traffic.