Shape Happens: More LLM Geometry w/ Frank Niu

Do language models really encode everything as tidy linear directions? Or is something curvier going on?

Previously in our Mox Summer Season, Thomas Fel of Goodfire made the case that concepts in neural nets live on curved, low-dimensional manifolds, and that SAEs, our standard tool for finding features, tend to represent these shapes in fragments: many local detectors, but no coherent global object.

This talk picks up where that one left off: If manifolds are the better representation, how do you find out what shape corresponds to a concept?

Frank Niu, co-author of "Hypothesis-Driven Feature Manifold Analysis in LLMs via SMDS" (TMLR 2026), will present a method built for a manifold-first world: propose a geometric hypothesis (ie circle, line, cluster) and SMDS delivers a quantitative verdict on which shape best fits. These manifolds reshape dynamically depending on the task the prompt poses, and models appear to genuinely use them to reason. Perturb the manifold and reasoning degrades; manifold quality predicts task performance.

The upshot is a picture of LLM reasoning as operating over structured manifolds rather than isolated features — models encoding concepts as shape, transforming the shape to suit the question, and using it to derive an answer.

📄 Paper: https://openreview.net/pdf?id=vCKZ40YYPr
💻 Code: https://github.com/UKPLab/tmlr2026-manifold-analysis

AGENDA

7:00 - doors open, light refreshments served
7:30 - talk begins, followed by Q&A
8:30 - continued hangout