Autoencoders are used for [[dimensionality reduction]]. They take an input with many dimensions, force it through a bottleneck layer([[encoder]]), and then use a decoder to reconstruct the original input. The neural network is trained to minimize the reconstruction error, which forces the compressed/bottleneck layer to learn the most important features. The reduced / bottlenect layer is called the [[Latent Space]]

Bert, roberta are examples of auto encoders. They are used for classification tasks. The attention is applied to entire context. It uses something like [[bidirectional attention]]