Skip to content

Conversation

@kyuyeunk
Copy link
Collaborator

@kyuyeunk kyuyeunk commented Nov 13, 2025

Description

Fixes [tpu_platform.py:154] The model dtype is not properly set for JAX backend. Overwriting it to jnp.bfloat16.

Properly translates model dtype set by vLLM (only model config) into jax dtype.

Tests

https://buildkite.com/tpu-commons/tpu-inference-ci/builds/6131

Checklist

Before submitting this PR, please make sure:

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have made or will make corresponding changes to any relevant documentation.

@github-actions
Copy link

Description

Start with a short description of what the PR does and how this is a change from
the past.

The rest of the description includes relevant details and context, examples:

  • why is this change being made,
  • the problem being solved and any relevant context,
  • why this is a good solution,
  • some information about the specific implementation,
  • shortcomings of the solution and possible future improvements.

If the change fixes a bug or a Github issue, please include a link, e.g.,:
FIXES: b/123456
FIXES: #123456

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure:

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have made or will make corresponding changes to any relevant documentation.

@kyuyeunk kyuyeunk force-pushed the fix_model_dtype branch 12 times, most recently from df4f5a4 to 8b37ed4 Compare November 19, 2025 08:21
@kyuyeunk kyuyeunk marked this pull request as draft November 20, 2025 02:42
@kyuyeunk kyuyeunk force-pushed the fix_model_dtype branch 2 times, most recently from 34afdd5 to 386d1d3 Compare November 29, 2025 13:24
Signed-off-by: Kyuyeun Kim <kyuyeunk@google.com>
@kyuyeunk kyuyeunk marked this pull request as ready for review November 29, 2025 17:07
@kyuyeunk kyuyeunk added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 29, 2025
@kyuyeunk
Copy link
Collaborator Author

@py4, @sixiang-google, @wenxindongwork can anyone take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants