Intro

They are testing weather manifold steering over the curved surface help in better steering , i didnt understand when they said if it will yeild behavios close to what models exhibit naturally Analysing encoded concepts in transformer language models

Papers which identified manifold structures[[Manifold Hypothesis]]

So what i understood is that there are these geometric structures inside the models, if we can identify these structures, then we can probably, help in steerting over the model, there by improving certain capabilities of models. My hypothesis is that if we are able to get a hold of these concepts, then can we combine multiple sources and may be improve the model’s performance. That is also one good direction. If we can fuse knowledge from different domains.

So lets imagine , if we have these many manifolds, there are 2 ways to think about this, one is the concepts are seperated, and one is that the concepts are intertwned.

If they are seperated, then how does model effectively route data from different sources. i dont know if thats any helpful.so what can we do about it ?

so the paper is talking about how

Limitations

They have studied establied strcutures like weekdays and they geometric representations. They haven’t done a detailed study on abstract concepts.

TODO Tomorrow Morning 8AM - 9.30AM

  • Write the Proposal
  • run the experiment, i see a gap in their work, add that to the works and create the solid proposal

A different paper direction

  • Is there research on how steering affects model’s capabilities, with the help of benchmarks and observe the benchmarks stats
  • Can we detect hallucinatins ? Or rather we determine when hallucination happens in small languaage models by gathering a corpus of text which would differentiate between hallucinated text and non hallucinated text, look at the activations of model and observe what can we do to mitigate it ?

What should i do next ?

What are my goals for today ?

  • Find the limitation, the limitation should add new information to the field. We have only studied simple concepts. And i have seen them doiung PCA and finding the corleations, So may be i should do the corelation study with help of UMAP. if that also prooves the same theory ?
  • Write the proposal to work on them

Cluade internal Thinking

Claude is responding

In the paper Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

I want to peel the oniopn. i havent understood when they said they have 2 representations, one is some thing and other is behavior representations. How did they create the mapping

I want to understand their experiment. Tell me the experiment methodloloy.

Retrieving knowledge of recent neural network representation paper

Retrieving knowledge of recent neural network representation paper

I’m trying to recall whether I have reliable knowledge of this specific paper about manifold steering and neural network representations. The title sounds like it could be about mapping between different representation spaces in neural networks, but I’m hitting the limits of my training data and I’m not confident enough in the details to give an accurate explanation of their methodology. I should be honest about my uncertainty rather than risk giving inaccurate details about a paper I’m not confident I know. The best approach is to check if the user uploaded the paper itself — if they did, I can give them a precise answer based on the actual text instead of relying on potentially faulty memory.

Checking whether the paper was uploaded

Script

The uploads directory is empty, so the paper wasn’t provided. I’m trying to recall if I have solid knowledge of “Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior” — it could be a recent paper from 2024-2025 that I have limited exposure to, and I don’t feel confident about the specifics. The right move here is to be honest about that uncertainty.

Claude’s response was interrupted.

2605_05115v1.pdf

pdf

here is the paper

Inventoried paper structure and extracted textual content

Inventoried paper structure and extracted textual content

Let me inventory the PDF, then pull out the text.

Excavated novel academic research and prepared methodological analysis

Excavated novel academic research and prepared methodological analysis

Claude’s response was interrupted.

2605.05115v1.pdf

pdf

here is the paper

Navigated corrupted sections to extract methodology and headers

Navigated corrupted sections to extract methodology and headers

I need to extract the full text from this 36-page paper to understand the core methodology—specifically how they define the two representations and establish the mapping between them, plus the experimental setup. Given the length, I’m looking at roughly 7,000 to 14,000 tokens of extracted text.

Extracting full text and locating where the main body ends

Script

Reading abstract and introduction

Script

I’m moving through the methodology sections now to understand how they fit the manifolds and establish the isometry, then I’ll look at the steering experiments in Section 3.

Reading section 2.2 onwards - manifold fitting methodology

Script

I’m skipping past the corrupted figure sections to locate the actual methodology content and key section headers.

Done

The figure areas are garbled. Let me navigate by section headers to get the clean methodology text.

Architecting response structure and methodology breakdown

Architecting response structure and methodology breakdown

Locating all section headers in the paper

Script

Reading section 2.2 Fitting the Manifolds

Script

Reading sections 2.3 and 3.1-3.2 on isometry and steering interventions

Script

Now I’m moving through the key sections on pullback optimization and the geometric framework that unifies different steering strategies, then diving into the experimental details across the appendix and the specific tasks like the ICL graph problem and mountain car environment.

Reading naturalness metric and section 3.3 pullback optimization

Script

Reading naturalness results, section 3.3 pullback, and 3.4 geometry unification

Script

Reading section 4 - ICL grid task methodology

Script

Now I’m moving through the mountain car section and then diving into the appendix to review the concrete methodology details, particularly the spline fitting approaches and the optimization techniques for steering interventions and pullback.

Reading section 5 - mountain car world model experiment

Script

Reading appendix A - full experimental methodology details

Script

I’m reviewing the visual components in Figures 2 and 4 to solidify my understanding of the pipeline, and checking the figure captions and relevant discussion sections to ensure I’m describing everything accurately.

The user is asking for three specific things: the two representation spaces (activation versus behavior), how the mapping between them was created through manifold fitting and isometry testing, and the full experimental methodology. They want a layered explanation that progressively deepens the understanding—starting with what the two spaces are, then how manifolds are fit within each, followed by the correlational mapping test, and finally the causal steering experiments in both directions.

I should be careful here: there’s no learned mapping function between the spaces. Instead, the correspondence is anchored through shared indexing—both manifolds are parameterized by the same conceptual domain, so each concept value (like “Wednesday”) maps to a centroid in activation space and a corresponding centroid in behavior space, establishing the alignment.

From there, they compare pairwise geodesic distances to check for scaled isometry, then test causality by intervening along the activation manifold and observing whether the induced trajectories match the behavior manifold’s geodesics. The forward pass F itself—from activations to output distribution—is the causal map connecting them, with the pullback metric using the Jacobian of F to relate the two spaces.

Now I’m laying out the experimental setup: Llama 3.1 8B with layer 28 residual stream activations, tested on cyclic tasks like weekdays and months plus sequential tasks like letter sequences and ages, along with some grid and world model benchmarks. The activation space is the residual stream in R^n, while the behavior space is a probability distribution over concept tokens, where variants like “Monday” and “monday” get aggregated into a single concept with remaining probability mass going to an “other” bin.

For manifold fitting, I’m reducing activations to 64 dimensions via PCA, computing concept centroids as the average activations for each answer, then fitting a cubic spline through those centroids—using periodic splines for cyclic tasks and natural splines for sequential ones, with the intrinsic coordinate being either an ordinal index or an angular parameter depending on the task structure.

On the behavior side, I’m averaging output distributions per answer, mapping them to Hellinger coordinates on a unit sphere via square root, fitting a spline in the tangent plane, and then decoding back to valid probability distributions. The correspondence between the two manifolds comes directly from matching concept labels—the “Wednesday” centroid in activation space maps to the “Wednesday” centroid in behavior space—and I’m computing geodesic distances on each manifold by discretizing paths and summing consecutive distances.

The correlation results are striking: geodesic distances on the manifolds correlate at 0.99 for weekdays, 0.89 for months, and 0.999 for both letters and ages, whereas linear Euclidean distances perform much worse. Now I’m testing causal steering by interpolating between concept centroids—either directly in the residual stream or through the manifold’s intrinsic coordinates—using up to 50 waypoints and averaging across 16 base prompts to measure how well I can steer behavior by manipulating the activation manifold.

For the reverse direction, I’m sampling geodesics between behavior centroids on the output manifold and optimizing activation paths using natural cubic splines constrained to the first 32 PCA dimensions, minimizing Hellinger distance to match target outputs.

The pullback paths achieve much better alignment with the manifold geometry than linear interpolation — R² scores of 0.47–0.78 compared to 0.23–0.42 for linear — even though the optimization never directly observed the hidden manifold structure.

I’m also considering three different geometric frameworks: the flat Euclidean space, a density-weighted metric derived from the energy landscape, and a pullback geometry induced by the Jacobian of the mapping from activations to outputs. For extensions, I’m exploring 2D manifolds like grids and cylinders using thin-plate splines with known ground-truth coordinates.

For the mountain car task, I’m using a recurrent world model with a CNN encoder feeding into a GRU, then decoding back to observations. The position acts as a conceptual domain, and I’m extracting encoder activations from rollouts to define a behavior distribution based on distances to position-bin centroids. The resulting manifolds form closed 1D curves.

A visualization diagram would really help clarify this pipeline — showing how prompts flow through activations and outputs to centroids, then to manifold discovery, isometry testing, and finally steering experiments. I’ll go ahead with the visualization — a clean diagram of the experimental pipeline will really help clarify the methodology. Now I’m structuring the response to address their core confusion first: what’s the actual difference between activation space and behavior space, and how does the “mapping” work. The key insight is that there’s no learned mapping function; instead, the correspondence emerges through shared concept labels and the model’s forward pass itself becomes the mapping. Then I’ll walk through the methodology step by step — starting with the tasks and data collection, moving into how centroids are computed, and then explaining the manifold fitting process.

Show less

I’ve extracted the full methodology from the paper. Now I’ll create a visual diagram of the pipeline.

Now I’m planning out the response structure: I’ll start with a direct explanation of the two spaces and their confusion, then walk through how the mapping gets created using shared concept labels and the forward pass, followed by the methodology broken down step by step. I’m thinking a clean flowchart of the experimental pipeline would help visualize this—showing both how the correspondence is anchored between the spaces and how the causal tests work.

For the diagram itself, I’m laying out a vertical pipeline with the task prompts at the top, then branching into two parallel columns for the activation space and behavior space, each flowing through concept centroids and manifolds, and finally converging with the isometry test and causal steering arrows. Laying out the test boxes in a single row below — three boxes of 180 pixels wide with 30-pixel gaps between them, fitting nicely within the 600-pixel available width. I’ll draw arrows from each manifold down to the test section, with the left arrow from Mh converging toward the first test box and the right arrow from My toward the third. Laying out the diagram with specific positioning: the prompts box goes in row 1, then I’m forking arrows down to row 2 which splits into two boxes for activations and outputs, calculating text widths to ensure everything fits within the canvas constraints. Finishing the right box label, then positioning a double-headed arrow between the two manifold boxes with an “isometry r ≈ 0.99” label centered above it — checking that the label width fits in the 100px gap and doesn’t overlap the boxes below.

Show less

Done

Claude is AI and can make mistakes. Please double-check responses.