Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

[MuvvaThriveni]

I am facing an issue when i am trying to build multiclass classification model
here is my code from starting

import pandas as pd
data=pd.read_csv('/content/Normalized_Data_PBLD.csv')

y=data['label'].tolist()
X_train, X_test, y_train, y_test = train_test_split(data['comment'].tolist(), y, random_state=5, test_size=0.2) #train, test split
#validation split
from sklearn.utils import class_weight
class_weights = class_weight.compute_class_weight(class_weight="balanced",
classes=np.unique(y_train),
y=y_train)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, random_state=5, test_size=0.1)

list_of_class={'NEG':0,'NTL':1,'POS':2}
y_val=[list_of_class[i.strip()]for i in y_val]
y_train=[list_of_class[i.strip()]for i in y_train]
y_test=[list_of_class[i.strip()]for i in y_test]

d1 = {'comment': X_train, 'label': y_train}
df_train = pd.DataFrame(d1)

d2 = {'comment': X_val, 'label': y_val}
df_val = pd.DataFrame(d2)

d3 = {'comment': X_test, 'label': y_test}
df_test = pd.DataFrame(d3)

calling bert model

model = ClassificationModel('bert', 'bert-base-multilingual-cased', num_labels=3, args={'learning_rate':1e-5, 'num_train_epochs': 2, 'reprocess_input_data': True, 'overwrite_output_dir': True})

model.train_model(df_train)

result, model_outputs, wrong_predictions = model.eval_model(df_val)
when running this line facing below error

ERROR:
ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

n tried another way
from sklearn.metrics import f1_score, accuracy_score

def f1_multiclass(labels, preds):
return f1_score(labels, preds, average='weighted')

result, model_outputs, wrong_predictions = model.eval_model(df_val, f1=f1_multiclass, acc=accuracy_score)

even though same error

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

anyone please help to solve

[lnash94]

I am also currently encountering a similar ValueError [1] while evaluating the model for multi-class classification using the DistilBERT model.

As a workaround, I have attempted the direct computation of the F1 score outside of the eval_model method.

[1] Error

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

[2] Workaround (tried the direct computation of the F1 score outside of the eval_model method)

from sklearn.metrics import f1_score
import numpy as np

# This part depends on how your model outputs predictions
predictions, raw_outputs = model.predict(valid_df['text'].tolist())

# This assumes `valid_df['labels']` contains the true class labels for each sample
true_labels = valid_df['labels'].values

# Calculate F1 Score
f1 = f1_score(true_labels, predictions, average='weighted')
print(f"Weighted F1 Score: {f1}")

It would be greatly appreciated if someone could provide input regarding this issue.

[pvcastro]

I'm downgrading to 0.64.3

[isdeniz]

I am getting the same error as below:

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

There are 3 classes in train and test datasets.
Below are some related code snippets:

def calc(p1, p2, func, **kwargs):
return func(p1, p2, **kwargs)

metrics_recom = {
"accuracy": partial(calc,func=sklearn.metrics.accuracy_score) ,
"p_micro": partial(calc,func=sklearn.metrics.precision_score,average='micro'),
"r_micro": partial(calc,func=sklearn.metrics.recall_score,average='micro'),
"f_micro": partial(calc,func=sklearn.metrics.f1_score,average='micro'),
"classificationReport": partial(calc,func=sklearn.metrics.classification_report, output_dict=True)}

results, model_outputs, wrong_pred = model.eval_model(test, verbose = True, **metrics_recom)
results

I am able to train the model and get predictions without any problem.

Would appreciate if you have any suggestion @ThilinaRajapakse

[isdeniz]

I tried version 0.64.3, and it worked without problem.
pip install simpletransformers==0.64.3

[swati-rajwal]

I faced the exact same problem and downgrading to 0.64.3 helped!
pip install simpletransformers==0.64.3
Thanks, folks for the potential solutions:)