Different baseline scripts log performances (and other metrics) under different keys

[abaisero]

As per title; I don't understand how am I supposed to compare methods appropriately if each script does its own thing and uses completely separate conventions.

I use wandb; if I run some of the value decomposition methods, I get performances under keys like the ones below:

If I run methods like IPPO or MAPPO, I get performances under different keys like below:

I understand that there can be come metrics that only make sense for some methods and not others; but when it comes to things like performances, winrates, it seems impossible to compare methods because each is written is its own standalone script that does things its own way.

I genuinely cannot understand: how do I use this library to compare methods?

[jselvaraaj]

I think for comparing JAX baselines, Mava might be a better option.

They are specifically designed for benchmarking MARL algorithms in JAX, and they also support jaxmarl (at least for some homogeneous scenarios). They also have more responsive code owners.

[abaisero]

Hi J; funny seeing you here. That looks like an interesting resource; though I'm under a tight timeline and probably won't be able to explore it fully any time soon. In the end, I figured that the baseline implementations are not really the main focus of this repo, but more like an extra to showcase how these environments could be used.

[amacrutherford]

Hey @abaisero this is a very fair point, will add to our todo list to get these all aligned.