How to predict the feature importance for complex ML model by SHAP?

[Darker1095]

I found that many times the output model of TPOT is a combination of multiple models, or feature processing，For example, this model ：StackingEstimator(estimator=DecisionTreeRegressor(max_depth=7, min_samples_leaf=13, min_samples_split=15)),
PolynomialFeatures(degree=2, include_bias=False, interaction_only=False),
XGBRegressor(learning_rate=0.1, max_depth=5, min_child_weight=4, n_estimators=100, n_jobs=1, objective="reg:squarederror", subsample=0.6000000000000001, verbosity=0)
Then it is difficult to predict the importance of the original features using SHAP values. Is there any way to restrict TPOT to outputting only relatively simple models?

[perib]

How to predict the feature importance for complex ML model by SHAP

This is not built into TPOT. How you approach this depends on the questions you are trying to ask. There are two options

1. you can pass the entire pipeline in the shap values function. TPOT pipelines are complex, but are still sklearn BaseEstimators and should be compatible with all the same shap functions. This will give you the feature importance of the pipeline as a whole (which is a valid approach).

2. If you are interested in the importance of an individual component given what has come before, You can manually extract the components of the pipeline, and apply the shap package to the combination of features/steps you are interested in. For example, you can take out the feature selectors, and only pass in the selected features + downstream components. Or you can take the everything up until the final classifier, transform your data, then pass that transformed data and the final classifier into the shap functions.

Is there any way to restrict TPOT to outputting only relatively simple models?

Your pipeline starts with a StackingEstimator so I'm assuming you might be using an older version on TPOT. The new version of TPOT (formally TPOT2, now just the current release of TPOT) gives you a lot more flexibility in defining pipelines. This allows you to restrict pipelines to be linear for example, if you want something easier to work with. You can also customize what categories of components can go in each step of the pipeline. See more information here: https://github.com/EpistasisLab/tpot/blob/f2f15f43799ec3bf1a146b4a2d069d26391b65c5/Tutorial/2_Search_Spaces.ipynb

For example, if you use SequentialPipeline or DynamicLinearPipeline, the result will be a simple linear sklearn.pipeline.Pipeline. From there you can use Pipeline.steps to access each object in sequence. (You could then use that to create the intermediate transformations you are interested in. If you use the GraphPipeline, there's a function get_inputs_to_node which will return the data transformed by all previous incoming nodes.

Note that this version of TPOT is a complete rewrite, so some things may not be backwards compatible. paper here: Ribeiro, P. et al. (2024). TPOT2: A New Graph-Based Implementation of the Tree-Based Pipeline Optimization Tool for Automated Machine Learning. In: Winkler, S., Trujillo, L., Ofria, C., Hu, T. (eds) Genetic Programming Theory and Practice XX. Genetic and Evolutionary Computation. Springer, Singapore. https://doi.org/10.1007/978-981-99-8413-8_1

[Darker1095]

Thank you very much. I try to train the model in the same case using tpot1.0(tpot = TPOTRegressor(search_space="linear", generations=5, population_size=100, verbose=2, random_state=random_state, n_jobs=10, scorers='neg_mean_squared_error', cv=5). But it takes more time, what happens, why the time consumed increases many times over？

[perib]

There are several changes to TPOT in the last update which can affect it's behavior. It was completely rewritten from scratch so many things are different. Here are some changes and tips on improving runtime in this version of TPOT:

• One change is that the default maximum time for evaluating a single pipeline has increased from 5 minutes to 10 minutes. You can reduce this by setting max_eval_time_mins to 5.
• The default number of cross validation folds has also been increased from 5 to 10. Decreasing number of folds increases speed (at the cost of accuracy if the dataset is small.
• The default population size has decreased from 100 to 50. Smaller population sizes are evaluated faster. In my testing around 50 seemed sufficient (sometimes with increased generations to compensate.) Early stopping is evaluated once per generation, so allowing for more generations means that the time to early stop is shorter. I saw that your setting it to 100, as was default in TPOT1, this could probably be decreased.
• By Default TPOT using maximum time limit to stop rather than a set number of generations. You have to set max_time_mins to None if you want to disable the timer (otherwise it will time out even if it has not reached the total number of generations).
• Early stopping is disabled by default, you can set this to a value of like 5-10 with a population size of around 20-50 is what I normally use.
• Some ml models (particularly SVC and SVR) are very slow algorithms that almost always wait take longer than max_eval_time_mins for certain parameters that are almost guaranteed to appear in your run. This effectively makes every generation take at least as long, sometimes longer if you have fewer threads and they run sequentially. I find that removing these some the search spaces drastically improves run time. Two options for removing this
  • you can use the 'linear-light' template which only includes the faster running models (and excludes things like SVR or MLP).
  • You can create your own custom search space with only the models you are interested in. You would need to create your own search space as described in Tutorial 2: https://github.com/EpistasisLab/tpot/blob/main/Tutorial/2_Search_Spaces.ipynb . If you want to model it off the default search spaces you can find them in this file: https://github.com/EpistasisLab/tpot/blob/main/tpot/config/template_search_spaces.py#L158
• TPOT also by default allows classifiers and regressors as inner nodes in the pipeline. In some cases this can improve performance, but it significantly slows training. the default 'linear' search space includes inner ml models, but the 'linear-light' does not. (if you want all ml models AND no inner classifiers/regressors, you'd have to modify the search space definitions manually https://github.com/EpistasisLab/tpot/blob/main/tpot/config/template_search_spaces.py#L158)
  • The default search space also allows for multiple transformers, but you can simplify it to include only one or None if you'd prefer. (The default linear search space with multiple transformers did perform better than pipelines with only 1 or no transformers).
• Having some sort or secondary objective to minimize complexity may also improve performance. The previous version of TPOT used number of nodes by default, but this is not set by default in the new version of TPOT.

I recommend using the complexity scorer here (This should probably have been made default):

scorers=['neg_mean_squared_error',tpot.objectives.complexity_scorer],
scorers_weights=[1,-1],

The other option is number of nodes objective, but I find that number of nodes is not a great measure of complexity (it makes LR and Xgboost equivalent).

scorers=['neg_mean_squared_error'],
scorers_weights=[1],
other_objective_functions=[tpot.objectives.number_of_leaves_objective],
other_objective_functions_weights=[-1],

more info in tutorial 1:

tpot/Tutorial/1_Using_TPOT.ipynb

Line 152 in f2f15f4

" scorers=['roc_auc_ovr',tpot.objectives.complexity_scorer],\n",

• In some cases, the steady TPOTEstimatorSteadyState can run faster (but performance may be higher or lower depending on the dataset). This estimator also supports early_stop_mins, to early stop if performance isn't increasing after a certain time period. This would be a good feature to bring into TPOTEstimator as well.
• you can also make use of periodic_checkpoint_folder to save your progress.

When I run tpot, I I like setting population size to 50 (20 to 50 should work fine), setting generations to None, setting a maximum max_time_mins, setting max_eval_time_mins to 10 (depending on the dataset size), and setting early stop to something between 5 and 10. When using early stop, make sure to look at the training curve in evaluated_individuals to make sure that it is more or less converging (you don't want to stop if the model is still improving).

For debugging, you can set verbosity to 5 to print all logs and errors to see if there's anything fishy happening. You can also take a look at the evaluated_individuals dataframe to check if the models are actually training correctly.

[perib]

hope this helps, let me know if you have any questions of any of these