From 8b296c3b37af9e913edfeac382e22c0157f969d8 Mon Sep 17 00:00:00 2001 From: Sanger Steel Date: Thu, 10 Jul 2025 17:22:15 -0400 Subject: [PATCH 1/7] docs: Update docs article with usage patterns Signed-off-by: Sanger Steel --- docs/models/extensions/tensorizer.md | 124 ++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 4 deletions(-) diff --git a/docs/models/extensions/tensorizer.md b/docs/models/extensions/tensorizer.md index 6ea61b080cda..d4f62d7e32e3 100644 --- a/docs/models/extensions/tensorizer.md +++ b/docs/models/extensions/tensorizer.md @@ -1,13 +1,129 @@ -# Loading models with CoreWeave's Tensorizer +--- +title: Loading models with CoreWeave's Tensorizer +--- +[](){ #tensorizer } vLLM supports loading models with [CoreWeave's Tensorizer](https://docs.coreweave.com/coreweave-machine-learning-and-ai/inference/tensorizer). vLLM model tensors that have been serialized to disk, an HTTP/HTTPS endpoint, or S3 endpoint can be deserialized at runtime extremely quickly directly to the GPU, resulting in significantly shorter Pod startup times and CPU memory usage. Tensor encryption is also supported. -For more information on CoreWeave's Tensorizer, please refer to -[CoreWeave's Tensorizer documentation](https://github.com/coreweave/tensorizer). For more information on serializing a vLLM model, as well a general usage guide to using Tensorizer with vLLM, see -the [vLLM example script](../../examples/others/tensorize_vllm_model.md). +vLLM fully integrates Tensorizer in to its model loading machinery. The +following will give a brief overview on how to get started with using +Tensorizer on vLLM. + +## The basics +To load a model using Tensorizer, it first needs to be serialized by Tensorizer. +The example script in [examples/others/tensorize_vllm_model.py] takes care of +this process. +(https://docs.vllm.ai/en/latest/examples/others/tensorize_vllm_model.html) + +Let's walk through a basic example by serializing `facebook/opt-125m` using the +script, and then loading it for inference. + +## Saving a vLLM model with Tensorizer +To save a model with Tensorizer, call the example script with the necessary +CLI arguments. The docstring for the script itself explains the CLI args +and how to use it properly in great detail, and we'll use one of the +examples from the docstring directly, assuming we want to save our model at +our S3 bucket example `s3://my-bucket`: + +```bash +python examples/others/tensorize_vllm_model.py \ + --model facebook/opt-125m \ + serialize \ + --serialized-directory s3://my-bucket \ + --suffix v1 +``` + +This saves the model tensors at `s3://my-bucket/vllm/facebook/opt-125m/v1`. If +you intend on applying a LoRA adapter to your tensorized model, you can pass +the HF id of the LoRA adapter in the above command, and the artifacts will be +saved there too: + +```bash +python examples/others/tensorize_vllm_model.py \ + --model facebook/opt-125m \ + --lora-path \ + serialize \ + --serialized-directory s3://my-bucket \ + --suffix v1 +``` + +## Serving the model using Tensorizer +Once the model is serialized where you want it, you can load the model using +`vllm serve` or the `LLM` entrypoint. The directory where the +model artifacts were saved can be passed to the `model` argument for +`LLM()` and `vllm serve`. For example, to serve the tensorized model +saved previously with the LoRA adapter, you'd do: + +```bash +vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ + --load-format tensorizer \ + --enable-lora +``` + +Or, with `LLM()`: + +```python +from vllm import LLM +llm = LLM( + "s3://my-bucket/vllm/facebook/opt-125m/v1", + load_format="tensorizer", + enable_lora=True +) +``` + +`tensorizer`'s core objects that serialize and deserialize models are +`TensorSerializer` and `TensorDeserializer` respectively. In order to +pass arbitrary kwargs to these, which will configure the serialization +and deserialization processes, you can provide them as keys to +`model_loader_extra_config` with `serialization_kwargs` and +`deserialization_kwargs` respectively. Full docstrings detailing all +parameters for the aforementioned objects can be found in `tensorizer`'s +[serialization.py](https://github. +com/coreweave/tensorizer/blob/main/tensorizer/serialization.py) file. + +As an example, CPU concurrency can be limited when serializing with +`tensorizer` via the `limit_cpu_concurrency` parameter in the +initializer for `TensorSerializer`. To set `limit_cpu_concurrency` to +some arbitrary value, you would do so like this when serializing: + +```bash +python examples/others/tensorize_vllm_model.py \ + --model facebook/opt-125m \ + --lora-path \ + serialize \ + --serialized-directory s3://my-bucket \ + --serialization-kwargs '{"limit_cpu_concurrency": 2}' \ + --suffix v1 +``` + +As an example when customizing the loading process via `TensorDeserializer`, +one could limit the number of concurrency readers during +deserialization with the `num_readers` parameter in the initializer +via `model_loader_extra_config` like so: + +```bash +vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ + --load-format tensorizer \ + --enable-lora \ + --model-loader-extra-config '{"deserialization_kwargs": {"num_readers": 2}}' +``` + +Or with `LLM()`: + +```python +from vllm import LLM +llm = LLM( + "s3://my-bucket/vllm/facebook/opt-125m/v1", + load_format="tensorizer", + enable_lora=True, + model_loader_extra_config={"deserialization_kwargs": {"num_readers": 2}} +) +``` + + !!! note Note that to use this feature you will need to install `tensorizer` by running `pip install vllm[tensorizer]`. From 110a6fdbe8b9ef234ef482e3779bfdf610ff223d Mon Sep 17 00:00:00 2001 From: William Goldby Date: Fri, 11 Jul 2025 10:45:49 -0700 Subject: [PATCH 2/7] fix: Rename headings and move content around Signed-off-by: William Goldby Signed-off-by: Sanger Steel --- docs/models/extensions/tensorizer.md | 66 +++++++++------------------- 1 file changed, 20 insertions(+), 46 deletions(-) diff --git a/docs/models/extensions/tensorizer.md b/docs/models/extensions/tensorizer.md index d4f62d7e32e3..737711c3b926 100644 --- a/docs/models/extensions/tensorizer.md +++ b/docs/models/extensions/tensorizer.md @@ -8,25 +8,24 @@ vLLM model tensors that have been serialized to disk, an HTTP/HTTPS endpoint, or at runtime extremely quickly directly to the GPU, resulting in significantly shorter Pod startup times and CPU memory usage. Tensor encryption is also supported. -vLLM fully integrates Tensorizer in to its model loading machinery. The -following will give a brief overview on how to get started with using -Tensorizer on vLLM. +vLLM fully integrates Tensorizer in to its model loading machinery. The following will give a brief overview on how to get started with using Tensorizer on vLLM. + +## Installing Tensorizer + +To install `tensorizer`, run `pip install vllm[tensorizer]`. ## The basics -To load a model using Tensorizer, it first needs to be serialized by Tensorizer. -The example script in [examples/others/tensorize_vllm_model.py] takes care of -this process. -(https://docs.vllm.ai/en/latest/examples/others/tensorize_vllm_model.html) -Let's walk through a basic example by serializing `facebook/opt-125m` using the -script, and then loading it for inference. +To load a model using Tensorizer, the model first needs to be serialized by +Tensorizer. [The example script](https://docs.vllm.ai/en/latest/examples/others/tensorize_vllm_model.html) takes care of this process. + +Let's walk through a basic example by serializing `facebook/opt-125m` using the script, and then loading it for inference. -## Saving a vLLM model with Tensorizer -To save a model with Tensorizer, call the example script with the necessary +## Serializing a vLLM model with Tensorizer + +To serialize a model with Tensorizer, call the example script with the necessary CLI arguments. The docstring for the script itself explains the CLI args -and how to use it properly in great detail, and we'll use one of the -examples from the docstring directly, assuming we want to save our model at -our S3 bucket example `s3://my-bucket`: +and how to use it properly in great detail, and we'll use one of the examples from the docstring directly, assuming we want to serialize and save our model at our S3 bucket example `s3://my-bucket`: ```bash python examples/others/tensorize_vllm_model.py \ @@ -36,10 +35,7 @@ python examples/others/tensorize_vllm_model.py \ --suffix v1 ``` -This saves the model tensors at `s3://my-bucket/vllm/facebook/opt-125m/v1`. If -you intend on applying a LoRA adapter to your tensorized model, you can pass -the HF id of the LoRA adapter in the above command, and the artifacts will be -saved there too: +This saves the model tensors at `s3://my-bucket/vllm/facebook/opt-125m/v1`. If you intend on applying a LoRA adapter to your tensorized model, you can pass the HF id of the LoRA adapter in the above command, and the artifacts will be saved there too: ```bash python examples/others/tensorize_vllm_model.py \ @@ -51,11 +47,8 @@ python examples/others/tensorize_vllm_model.py \ ``` ## Serving the model using Tensorizer -Once the model is serialized where you want it, you can load the model using -`vllm serve` or the `LLM` entrypoint. The directory where the -model artifacts were saved can be passed to the `model` argument for -`LLM()` and `vllm serve`. For example, to serve the tensorized model -saved previously with the LoRA adapter, you'd do: + +Once the model is serialized where you want it, you can load the model using `vllm serve` or the `LLM` entrypoint. You can pass the directory where you saved the model to the `model` argument for `LLM()` and `vllm serve`. For example, to serve the tensorized model saved previously with the LoRA adapter, you'd do: ```bash vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ @@ -74,20 +67,9 @@ llm = LLM( ) ``` -`tensorizer`'s core objects that serialize and deserialize models are -`TensorSerializer` and `TensorDeserializer` respectively. In order to -pass arbitrary kwargs to these, which will configure the serialization -and deserialization processes, you can provide them as keys to -`model_loader_extra_config` with `serialization_kwargs` and -`deserialization_kwargs` respectively. Full docstrings detailing all -parameters for the aforementioned objects can be found in `tensorizer`'s -[serialization.py](https://github. -com/coreweave/tensorizer/blob/main/tensorizer/serialization.py) file. - -As an example, CPU concurrency can be limited when serializing with -`tensorizer` via the `limit_cpu_concurrency` parameter in the -initializer for `TensorSerializer`. To set `limit_cpu_concurrency` to -some arbitrary value, you would do so like this when serializing: +`tensorizer`'s core objects that serialize and deserialize models are `TensorSerializer` and `TensorDeserializer` respectively. In order to pass arbitrary kwargs to these, which will configure the serialization and deserialization processes, you can provide them as keys to `model_loader_extra_config` with `serialization_kwargs` and `deserialization_kwargs` respectively. Full docstrings detailing all parameters for the aforementioned objects can be found in `tensorizer`'s [serialization.py](https://github.com/coreweave/tensorizer/blob/main/tensorizer/serialization.py) file. + +As an example, CPU concurrency can be limited when serializing with `tensorizer` via the `limit_cpu_concurrency` parameter in the initializer for `TensorSerializer`. To set `limit_cpu_concurrency` to some arbitrary value, you would do so like this when serializing: ```bash python examples/others/tensorize_vllm_model.py \ @@ -99,10 +81,7 @@ python examples/others/tensorize_vllm_model.py \ --suffix v1 ``` -As an example when customizing the loading process via `TensorDeserializer`, -one could limit the number of concurrency readers during -deserialization with the `num_readers` parameter in the initializer -via `model_loader_extra_config` like so: +As an example when customizing the loading process via `TensorDeserializer`, you could limit the number of concurrency readers during deserialization with the `num_readers` parameter in the initializer via `model_loader_extra_config` like so: ```bash vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ @@ -122,8 +101,3 @@ llm = LLM( model_loader_extra_config={"deserialization_kwargs": {"num_readers": 2}} ) ``` - - - -!!! note - Note that to use this feature you will need to install `tensorizer` by running `pip install vllm[tensorizer]`. From 480e84eb8ebca7c72310f299ca7a4ad36cc7f483 Mon Sep 17 00:00:00 2001 From: William Goldby Date: Tue, 15 Jul 2025 15:16:04 -0700 Subject: [PATCH 3/7] fix: Add title for Tensorizer configuration Signed-off-by: William Goldby Signed-off-by: Sanger Steel --- docs/models/extensions/tensorizer.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/models/extensions/tensorizer.md b/docs/models/extensions/tensorizer.md index 737711c3b926..28007037db0e 100644 --- a/docs/models/extensions/tensorizer.md +++ b/docs/models/extensions/tensorizer.md @@ -67,6 +67,8 @@ llm = LLM( ) ``` +## Options for configuring Tensorizer + `tensorizer`'s core objects that serialize and deserialize models are `TensorSerializer` and `TensorDeserializer` respectively. In order to pass arbitrary kwargs to these, which will configure the serialization and deserialization processes, you can provide them as keys to `model_loader_extra_config` with `serialization_kwargs` and `deserialization_kwargs` respectively. Full docstrings detailing all parameters for the aforementioned objects can be found in `tensorizer`'s [serialization.py](https://github.com/coreweave/tensorizer/blob/main/tensorizer/serialization.py) file. As an example, CPU concurrency can be limited when serializing with `tensorizer` via the `limit_cpu_concurrency` parameter in the initializer for `TensorSerializer`. To set `limit_cpu_concurrency` to some arbitrary value, you would do so like this when serializing: From b2efb9ff86119bbbd62a32fc813b87fb28a4e178 Mon Sep 17 00:00:00 2001 From: Sanger Steel Date: Fri, 18 Jul 2025 10:20:44 -0400 Subject: [PATCH 4/7] docs: Update example file docstring Signed-off-by: Sanger Steel --- examples/others/tensorize_vllm_model.py | 31 ++++++++++++++----------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/examples/others/tensorize_vllm_model.py b/examples/others/tensorize_vllm_model.py index 64a6c42ae235..8c33114780e3 100644 --- a/examples/others/tensorize_vllm_model.py +++ b/examples/others/tensorize_vllm_model.py @@ -84,18 +84,24 @@ Once a model is serialized, tensorizer can be invoked with the `LLM` class directly to load models: - llm = LLM(model="facebook/opt-125m", - load_format="tensorizer", - model_loader_extra_config=TensorizerConfig( - tensorizer_uri = path_to_tensors, - num_readers=3, - ) - ) +```python +from vllm import LLM +llm = LLM( + "s3://my-bucket/vllm/facebook/opt-125m/v1", + load_format="tensorizer", + enable_lora=True +) +``` + A serialized model can be used during model loading for the vLLM OpenAI -inference server. `model_loader_extra_config` is exposed as the CLI arg -`--model-loader-extra-config`, and accepts a JSON string literal of the -TensorizerConfig arguments desired. +inference server: + +``` +vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ + --load-format tensorizer \ + --enable-lora +``` In order to see all of the available arguments usable to configure loading with tensorizer that are given to `TensorizerConfig`, run: @@ -116,10 +122,9 @@ `--enable-lora`. For instance: ``` -vllm serve \ +vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ --load-format tensorizer \ - --model-loader-extra-config '{"tensorizer_uri": ".tensors"}' \ - --enable-lora + --enable-lora ``` """ From c1ba86c9e664c5264e25db7332caf11edb168640 Mon Sep 17 00:00:00 2001 From: Sanger Steel Date: Fri, 18 Jul 2025 13:12:21 -0400 Subject: [PATCH 5/7] docs: Revert original markdown title Signed-off-by: Sanger Steel --- docs/models/extensions/tensorizer.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/models/extensions/tensorizer.md b/docs/models/extensions/tensorizer.md index 28007037db0e..3e01be2d8b9a 100644 --- a/docs/models/extensions/tensorizer.md +++ b/docs/models/extensions/tensorizer.md @@ -1,7 +1,4 @@ ---- -title: Loading models with CoreWeave's Tensorizer ---- -[](){ #tensorizer } +# Loading models with CoreWeave's Tensorizer vLLM supports loading models with [CoreWeave's Tensorizer](https://docs.coreweave.com/coreweave-machine-learning-and-ai/inference/tensorizer). vLLM model tensors that have been serialized to disk, an HTTP/HTTPS endpoint, or S3 endpoint can be deserialized From dfe285087eec16ef06452c6152733652f95a1bf9 Mon Sep 17 00:00:00 2001 From: Sanger Steel Date: Mon, 21 Jul 2025 13:41:35 -0400 Subject: [PATCH 6/7] style: Run linter Signed-off-by: Sanger Steel --- docs/models/extensions/tensorizer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/models/extensions/tensorizer.md b/docs/models/extensions/tensorizer.md index 3e01be2d8b9a..ffb3a7d55f85 100644 --- a/docs/models/extensions/tensorizer.md +++ b/docs/models/extensions/tensorizer.md @@ -13,7 +13,7 @@ To install `tensorizer`, run `pip install vllm[tensorizer]`. ## The basics -To load a model using Tensorizer, the model first needs to be serialized by +To load a model using Tensorizer, the model first needs to be serialized by Tensorizer. [The example script](https://docs.vllm.ai/en/latest/examples/others/tensorize_vllm_model.html) takes care of this process. Let's walk through a basic example by serializing `facebook/opt-125m` using the script, and then loading it for inference. From c67e148c6d916f9087b5cb5adc75f4ff6455e1c3 Mon Sep 17 00:00:00 2001 From: Sanger Steel Date: Wed, 23 Jul 2025 14:25:20 -0400 Subject: [PATCH 7/7] docs: Resolve suggested changes from review Signed-off-by: Sanger Steel --- docs/models/extensions/tensorizer.md | 2 +- examples/others/tensorize_vllm_model.py | 6 ++---- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/models/extensions/tensorizer.md b/docs/models/extensions/tensorizer.md index ffb3a7d55f85..f70ab0c6f4e5 100644 --- a/docs/models/extensions/tensorizer.md +++ b/docs/models/extensions/tensorizer.md @@ -14,7 +14,7 @@ To install `tensorizer`, run `pip install vllm[tensorizer]`. ## The basics To load a model using Tensorizer, the model first needs to be serialized by -Tensorizer. [The example script](https://docs.vllm.ai/en/latest/examples/others/tensorize_vllm_model.html) takes care of this process. +Tensorizer. [The example script](../../examples/others/tensorize_vllm_model.md) takes care of this process. Let's walk through a basic example by serializing `facebook/opt-125m` using the script, and then loading it for inference. diff --git a/examples/others/tensorize_vllm_model.py b/examples/others/tensorize_vllm_model.py index 8c33114780e3..559c7c493aca 100644 --- a/examples/others/tensorize_vllm_model.py +++ b/examples/others/tensorize_vllm_model.py @@ -88,8 +88,7 @@ from vllm import LLM llm = LLM( "s3://my-bucket/vllm/facebook/opt-125m/v1", - load_format="tensorizer", - enable_lora=True + load_format="tensorizer" ) ``` @@ -99,8 +98,7 @@ ``` vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ - --load-format tensorizer \ - --enable-lora + --load-format tensorizer ``` In order to see all of the available arguments usable to configure