In this paper i would understand the 3 prominent LLMs GPT, Llama, Palm, I would understand their characterstics, contributions and limitations. I would also learn about the techniques on building, evalutaion

The LLM evolved from 4 stages Early statistical models ( ngram) -> Neural Language Moedels -> Pretrained language models -> Large language models.

The early transformers models were of 3 types Encoder only, Decoder only and encoder-decoder.

The paper mentioned several smaller models which were trained from opensource llama models making several changes.

Google’s Palm

Paper mentioend several models from google. How they were trained .How a simple QmA / tasks dataset improved the base model with +9% improvement. They also used PEFT for aligning the palm model to Medical domain. It is still inferior to human evaluators. They imroved with MedPalm2 with med domain finetuning and ensamble pompting. +19% on SOTA.

page 1 early Semantic neural networks were used on task specific,

Now i know what are [[Auto-regressive]] and [[autoencoders]] models Autoregressive

QUESTION

  1. How Did transformer help parallelize compared to RNN.
  2. What are contexual representation vectors ? How Bert converts the embeddings to CRV using transformers?
  3. What is layer normalization.
  4. What are auto regressive and auto encoding networks

Research Questions

  1. Since the LLMs are so good at understanding instructions, Why is it still a problem with robotics? I understand that with robotics, there needs to be a contexual understanting layer, and then an actuator layer which converts these instructions to robotic motors, and a loop back layer that tells the current state of the environment. The contexual model would then undertand the next state and find the best outcome. What are the challenges here?
  2. Why is Alpaca being a smaller model perfroms at the same level at GPT3.5 ? . When meta worked on the Llama model it used a few differences than the GPT3, which was usign a swiGLU activation function instead of ReLU. Used Rotary positional embeddings instead of absolute positinal embedding and using root mean squared layer notmalization instead of standard layer normalization. How are these changes impactful ?
  3. In the model Bernie, they fused 2 networks Language generation network and Languate understanding network. to do both tasks, can