Also this one, which is similar to #51379 -- cc @xingyu-long

Hi @richardliaw, I did some investigating on this.

if I understand this issue correctly, we'd like to have the ray_remote_args/ray_remote_args_fn in preprocessors

I looked the code, and it seems users can implement

and then while we are calling map_batches, we load the config

i.e., they can add ray_remote_args/ray_remote_args_fn into the dict within def _get_transform_config(self) if they are implementing subclass

def _get_transform_config(self) -> Dict[str, Any]:
    return {
      "ray_remote_args": ...,
      "ray_remote_args_fn": ...,
      # other args...
   }

maybe we can have the corresponding setter for transform_config? and then we only update preprocessor. or you are looking for something different in terms of API design here?

or we modify the constructor of preprocessor with default value for both two arguments, all existing subclass also need to be updated?

for example with tokenizer

>>> from ray.data.preprocessors import Tokenizer
>>> tokenizer = Tokenizer(columns=["text"], ray_remote_args=..., ray_remote_args_fn=...)

btw, feel free to assign this issue to me too. Thanks!

I decided to give a try with constructor approach above. feel free to check out linked PR for details. Thanks!