[Data] support num_cpus, memory, concurrency, batch_size for preprocess

[xingyu-long]

Why are these changes needed?

As title, we'd like to support above parameters for preprocess

Related issue number

Close #52448

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[xingyu-long]

cc @richardliaw to review this change set.

Please note that I only update the Tokenizer with above arguments, if we are good with this solution, I will update other subclass of preprocess.

[richardliaw]

OK, thinking through this a little more. I think maybe better if we just expose:

• num_cpus
• memory
• batch_size
• concurrency

And avoid exposing ray_remote_args and ray_remote_args_fn generally.

This will be simpler to maintain and will allow it to be easily serializable

[xingyu-long]

OK, thinking through this a little more. I think maybe better if we just expose:

• num_cpus
• memory
• batch_size
• concurrency

And avoid exposing ray_remote_args and ray_remote_args_fn generally.

This will be simpler to maintain and will allow it to be easily serializable

Make sense, will update code accordingly. thanks!

[richardliaw]

Another thought - 1. This should only be for transform(), not init 2. for fit_transform() we can add “transform_num_cpus” or something

…

On Fri, Apr 25, 2025 at 8:13 PM Xingyu Long ***@***.***> wrote: *xingyu-long* left a comment (ray-project/ray#52574) <#52574 (comment)> OK, thinking through this a little more. I think maybe better if we just expose: - num_cpus - memory - batch_size - concurrency And avoid exposing ray_remote_args and ray_remote_args_fn generally. This will be simpler to maintain and will allow it to be easily serializable Make sense, will update code accordingly. thanks! — Reply to this email directly, view it on GitHub <#52574 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCRZZMEUURY3MWILDRJG3323L2VFAVCNFSM6AAAAAB3X7PFZWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMZRG44DSNRVGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[xingyu-long]

Another thought -

1. This should only be for transform(), not init
2. for fit_transform() we can add “transform_num_cpus” or something

sure, we can do this too. just updated it. also I was wondering if we should remove function _get_transform_config since we used these configs explicitly, let me know what do you thin k on this. Thanks!

[richardliaw]

nice work, left some comments

[xingyu-long]

Thanks for reviewing it, @richardliaw, I just updated the changes.

[xingyu-long]

with this change (f52f057), fixed the test_chain test and also tested it all preprocessor tests.

$ pytest -v -s python/ray/data/tests/preprocessors/
Results (25.86s):
     119 passed

[richardliaw]

1 small nit

[richardliaw]

oh i just realized, this looks good - you're planning to extend this to all preprocessors right?

[xingyu-long]

oh i just realized, this looks good - you're planning to extend this to all preprocessors right?

That was the idea (with constructor approach I tried), however, since we modified common functions in parent class, so we don't have to touch them any more.

I've ran all tests under preprocess test dir. let me know if I am missing any other tests.

btw, I will fix the above comments later today. Thanks!