New Model: Ava

[Kuduxaaa]

🚀 Add AvaModel: A Scalable & Efficient LLM Architecture

Overview

This PR introduces AvaModel, a new transformer-based language model architecture designed for high performance across different scales (100M to 100B parameters).

Architecture Highlights

AvaForCausalLM(
  (model): AvaModel(
    (embed_tokens): Embedding(32000, 1280)
    (layers): ModuleList(
      (0-11): 12 x AvaDecoderLayer(
        (self_attn): AvaAttention(
          (q_proj): Linear(in_features=1280, out_features=1280, bias=False)
          (k_proj): Linear(in_features=1280, out_features=640, bias=False)
          (v_proj): Linear(in_features=1280, out_features=640, bias=False)
          (o_proj): Linear(in_features=1280, out_features=1280, bias=False)
          (dropout): Dropout(p=0.0, inplace=False)
        )
        (mlp): AvaMLP(
          (gate_proj): Linear(in_features=1280, out_features=5120, bias=False)
          (up_proj): Linear(in_features=1280, out_features=5120, bias=False)
          (down_proj): Linear(in_features=5120, out_features=1280, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): AvaRMSNorm()
        (post_attention_layernorm): AvaRMSNorm()
      )
    )
    (norm): AvaRMSNorm()
    (rotary_emb): AvaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=1280, out_features=32000, bias=False)
)

More info: Kuduxaaa/ava-llm

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[Rocketknight1]

Hi @Kuduxaaa, we generally don't add architectures to transformers until a significant pre-trained model exists using that architecture! The architecture looks good and we appreciate the educational goal, but we probably won't be able to merge this PR. I'm sorry!

[Kuduxaaa]

Thanks for the clarification! Totally understandable. I appreciate the feedback and the kind words about the architecture. Once a pre-trained model is ready and properly evaluated, I’ll revisit this with a new PR 🥳🧠