diff --git a/docs/source/en/model_doc/cohere.md b/docs/source/en/model_doc/cohere.md index 2ab75e9d1c8b..48b924e1ff13 100644 --- a/docs/source/en/model_doc/cohere.md +++ b/docs/source/en/model_doc/cohere.md @@ -1,124 +1,115 @@ -# Cohere - -
-PyTorch -FlashAttention -SDPA +
+
+ PyTorch + FlashAttention + SDPA +
-## Overview - -The Cohere Command-R model was proposed in the blogpost [Command-R: Retrieval Augmented Generation at Production Scale](https://txt.cohere.com/command-r/) by the Cohere Team. - -The abstract from the paper is the following: -*Command-R is a scalable generative model targeting RAG and Tool Use to enable production-scale AI for enterprise. Today, we are introducing Command-R, a new LLM aimed at large-scale production workloads. Command-R targets the emerging “scalable” category of models that balance high efficiency with strong accuracy, enabling companies to move beyond proof of concept, and into production.* +# Cohere -*Command-R is a generative model optimized for long context tasks such as retrieval augmented generation (RAG) and using external APIs and tools. It is designed to work in concert with our industry-leading Embed and Rerank models to provide best-in-class integration for RAG applications and excel at enterprise use cases. As a model built for companies to implement at scale, Command-R boasts: -- Strong accuracy on RAG and Tool Use -- Low latency, and high throughput -- Longer 128k context and lower pricing -- Strong capabilities across 10 key languages -- Model weights available on HuggingFace for research and evaluation +Cohere Command-R is a 35B parameter multilingual large language model designed for long context tasks like retrieval-augmented generation (RAG) and calling external APIs and tools. The model is specifically trained for grounded generation and supports both single-step and multi-step tool use. It supports a context length of 128K tokens. -Checkout model checkpoints [here](https://huggingface.co/CohereForAI/c4ai-command-r-v01). -This model was contributed by [Saurabh Dash](https://huggingface.co/saurabhdash) and [Ahmet Üstün](https://huggingface.co/ahmetustun). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). +You can find all the original Command-R checkpoints under the [Command Models](https://huggingface.co/collections/CohereForAI/command-models-67652b401665205e17b192ad) collection. -## Usage tips - +> [!TIP] +> Click on the Cohere models in the right sidebar for more examples of how to apply Cohere to different language tasks. -The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be -used by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. +The example below demonstrates how to generate text with [`Pipeline`] or the [`AutoModel`], and from the command line. -The `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype="auto"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. + + -Training the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`. +```python +import torch +from transformers import pipeline + +pipeline = pipeline( + task="text-generation", + model="CohereForAI/c4ai-command-r-v01", + torch_dtype=torch.float16, + device=0 +) +pipeline("Plants create energy through a process known as") +``` - -The model and tokenizer can be loaded via: + + ```python -# pip install transformers +import torch from transformers import AutoTokenizer, AutoModelForCausalLM -model_id = "CohereForAI/c4ai-command-r-v01" -tokenizer = AutoTokenizer.from_pretrained(model_id) -model = AutoModelForCausalLM.from_pretrained(model_id) +tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") +model = AutoModelForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01", torch_dtype=torch.float16, device_map="auto", attn_implementation="sdpa") -# Format message with the command-r chat template -messages = [{"role": "user", "content": "Hello, how are you?"}] -input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt") -## <|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|> - -gen_tokens = model.generate( +# format message with the Command-R chat template +messages = [{"role": "user", "content": "How do plants make energy?"}] +input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda") +output = model.generate( input_ids, max_new_tokens=100, do_sample=True, temperature=0.3, - ) - -gen_text = tokenizer.decode(gen_tokens[0]) -print(gen_text) + cache_implementation="static", +) +print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` -- When using Flash Attention 2 via `attn_implementation="flash_attention_2"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type. - + + -## Resources +```bash +# pip install -U flash-attn --no-build-isolation +transformers-cli chat --model_name_or_path CohereForAI/c4ai-command-r-v01 --torch_dtype auto --attn_implementation flash_attention_2 +``` -A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Command-R. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource. + + +Quantization reduces the memory burden of large models by representing the weights in a lower precision. Refer to the [Quantization](../quantization/overview) overview for more available quantization backends. - +The example below uses [bitsandbytes](../quantization/bitsandbytes) to quantize the weights to 4-bits. -Loading FP16 model ```python -# pip install transformers -from transformers import AutoTokenizer, AutoModelForCausalLM - -model_id = "CohereForAI/c4ai-command-r-v01" -tokenizer = AutoTokenizer.from_pretrained(model_id) -model = AutoModelForCausalLM.from_pretrained(model_id) +import torch +from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM -# Format message with the command-r chat template -messages = [{"role": "user", "content": "Hello, how are you?"}] -input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt") -## <|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|> +bnb_config = BitsAndBytesConfig(load_in_4bit=True) +tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") +model = AutoModelForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01", torch_dtype=torch.float16, device_map="auto", quantization_config=bnb_config, attn_implementation="sdpa") -gen_tokens = model.generate( +# format message with the Command-R chat template +messages = [{"role": "user", "content": "How do plants make energy?"}] +input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda") +output = model.generate( input_ids, max_new_tokens=100, do_sample=True, temperature=0.3, - ) - -gen_text = tokenizer.decode(gen_tokens[0]) -print(gen_text) + cache_implementation="static", +) +print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` -Loading bitsnbytes 4bit quantized model -```python -# pip install transformers bitsandbytes accelerate -from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig +Use the [AttentionMaskVisualizer](https://github.com/huggingface/transformers/blob/beb9b5b02246b9b7ee81ddf938f93f44cfeaad19/src/transformers/utils/attention_visualizer.py#L139) to better understand what tokens the model can and cannot attend to. -bnb_config = BitsAndBytesConfig(load_in_4bit=True) +```py +from transformers.utils.attention_visualizer import AttentionMaskVisualizer -model_id = "CohereForAI/c4ai-command-r-v01" -tokenizer = AutoTokenizer.from_pretrained(model_id) -model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config) +visualizer = AttentionMaskVisualizer("CohereForAI/c4ai-command-r-v01") +visualizer("Plants create energy through a process known as") +``` -gen_tokens = model.generate( - input_ids, - max_new_tokens=100, - do_sample=True, - temperature=0.3, - ) +
+ +
-gen_text = tokenizer.decode(gen_tokens[0]) -print(gen_text) -``` +## Notes +- Don’t use the torch_dtype parameter in [`~AutoModel.from_pretrained`] if you’re using FlashAttention-2 because it only supports fp16 or bf16. You should use [Automatic Mixed Precision](https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html), set fp16 or bf16 to True if using [`Trainer`], or use [torch.autocast](https://pytorch.org/docs/stable/amp.html#torch.autocast). ## CohereConfig @@ -143,5 +134,3 @@ print(gen_text) [[autodoc]] CohereForCausalLM - forward - -