Updated Albert model Card

[souvikchand]

What does this PR do?

Updates ALBERT model card as per #36979

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? [Community contributions] Model cards #36979
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@stevhliu Please check the PR and see if it's alright 😄

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[stevhliu]

Nice, thanks for your contribution!

[stevhliu]

Suggested change

> ALBERT is not compatible with `AttentionMaskVisualizer` as it uses masked self-attention rather than causal attention. So don't have `_update_causal_mask` method.

[souvikchand]

Since almost every model's model card will have an AttentionMaskVisualizer but ALBERT is not getting it, isn't it good to explain why it don't have any?

[stevhliu]

The AttentionMaskVisualizer has only been implemented for ~50/300 models so far so I think its okay to leave it out at the moment.

[souvikchand]

ok alright. thanks for your response.

[stevhliu]

Suggested change

	- ALBERT supports a maximum sequence length of 512 tokens.
	- Cannot be used for autoregressive generation (unlike GPT)
	- ALBERT requires absolute positional embeddings, and it expects right-padding (i.e., pad tokens should be added at the end, not the beginning).
	- ALBERT uses token_type_ids, just like BERT. So you should indicate which token belongs to which segment (e.g., sentence A vs. sentence B) when doing tasks like question answering or sentence-pair classification.
	- ALBERT uses a different pretraining objective called Sentence Order Prediction (SOP) instead of Next Sentence Prediction (NSP), so fine-tuned models might behave slightly differently from BERT when modeling inter-sentence relationships.

[souvikchand]

could you please briefly explain why it's better not to mention internal differences like SOP or token_type_ids in the model card? I want to make sure I follow the best style when contributing next time.

[stevhliu]

This can also be simplified a bit without the comments and include SDPA:

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2")
model = AutoModelForMaskedLM.from_pretrained(
    "albert/albert-base-v2",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
    device_map="auto"
)

prompt = "Plants create energy through a process known as [MASK]."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model(**inputs)
    mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
    predictions = outputs.logits[0, mask_token_index]

top_k = torch.topk(predictions, k=5).indices.tolist()
for token_id in top_k[0]:
    print(f"Prediction: {tokenizer.decode([token_id])}")

[souvikchand]

it is done

[souvikchand]

instead of .to("cuda") i wrote .to(model.device) . I hope it's okay