Allow OptunaSearch to wait for n_startup_trials to finish before launching additional trials

[fkgruber]

Description

Hi
I'm using Ray Tune with OptunaSearch (TPESampler) for hyperparameter optimization. While I understand that Optuna uses n_startup_trials to determine how many trials to run before switching from random to TPE-based sampling, I’ve run into a practical limitation:

• When running in parallel, more than n_startup_trials can be launched as random trials if early trials have not completed yet. This can lead to dozens of random trials, which defeats the purpose of a quick switch to TPE sampling.

It would be useful to enforce the following behavior:

• Launch exactly n_startup_trials trials using random sampling.

• Wait for all of them to finish.

• Only then allow new trials to be submitted using the TPE sampler.

• After that, allow full parallelism again for TPE-based trials.

Currently I'm using ConcurrencyLimiter to limit the number of concurrent trials to reduce over-submission of random trials, but there is no guarantee that only n_startup_trials total will run before TPE kicks in — especially if early trials are slow.

This would be really useful because it would avoid wasted compute on excessive random trials and improve the sample efficiency of Optuna within Ray Tune in many practical use cases.

thanks
FKG

Use case

No response