Prior work proposed that llm interact in one dimenstional activation sapce. This paper explore if models represent concepts in multi dimentional space. They built techniques over [[Sparse Autoencoder]] and found that models did form cirular structres for days related tasks, they even observed that model actively used these circular representations in its computations.

In conclusion they mentioned that previous work showed single dimentional represntations, this paper work showed that features exist in multi dimentional representation. They said they were fortunate enough to have sparse auto encoders to study the multi dimentionality. They mention that its important to understand the internal representation of what algorithms does the model internally uses.

Limitations : They were not able to find more number of such representations. They only looked at 2 dimentions and question if there exists such representaions in higher dimentions. Another question if their clustering technique is good enough to group them together. Or if there arent many such representations.

Future work might make progress on this

question by investigating new techniques for decomposing model representations, exploring higher

dimensional representations, or determining conclusively whether models use representations in ways

that necessitate the representations are non-linear.