integrations: Evidently

[daavoo]

Probably no need to build anything custom but just make an example using the Python API.

Example with MLFlow: https://docs.evidentlyai.com/integrations/evidently-and-mlflow

[francesco086]

Hi @daavoo , would be happy to contribute to this one.

From what I see, I agree that an example with dvc is sufficient. But this example should appear on the evidently doc webpage, right?

[daavoo]

Hi @daavoo , would be happy to contribute to this one.

From what I see, I agree that an example with dvc is sufficient. But this example should appear on the evidently doc webpage, right?

Yes.

I think it would not harm to have content both in dvc.org (i.e. in https://dvc.org/doc/user-guide/integrations/ml-frameworks) and in the evidently website.

[shcheklein]

@mnrozhkov can share btw some materials on this I think.

[mnrozhkov]

I would be happy to help with this! There is an example with DVC pipelines + Evidently. But, DVCLive can be used as well.

[francesco086]

Thank you! I will come with a PR that adds a new evidently.md file here towards the end of next week. Once that is merged, I will then try to push it to the evidently doc webpage too.

[francesco086]

Hi guys, sry for being late, last week I was sick.

So, I picked this up and what looks to me as being most meaningful is to mimic the evidently page, using dvclive instead of mlflow.
As soon as I started writing down something I realized that it would be useful to have something that one can replicate.
And this is why I ended up on a colab notebook (I barely used it in the past, and I am not an enthusiast, but I guess it serves the purpose here):
https://colab.research.google.com/drive/14usegPOSArF9tdO7NUOPndNk5vuUaYaX?usp=sharing

I think I could replicate everything, except that I am not able to show all the steps at the end. Any hint?

[dberenbaum]

Awesome work @francesco086!

There are two possible ways to handle the steps here:

1. In a single experiment (the way you do it). In this case, you can't see all steps in the table. It only shows the latest. However, you can use plots to compare them (try dvc plots show) or see the results in the VS Code extension or Studio.
2. Make each comparison a separate experiment. This will give you each step as a separate row in the table but without having them inherently grouped as steps.

[francesco086]

Thanks for the clarification @dberenbaum, I thought there was a way I was unaware of.

I then adjusted the notebook, and actually show both approaches: all in one experiment, and plots, or multiple experiments and visualization via dvc exp show. I think we can leave both, right?

Ok, now that the formalities are done, let's get concrete. I can easily write a markdown, but every time an update would happen it will be a pain to update it... I think it would be easier if one could keep a notebook next to the markdown in the documentation, and have a script or something that turns the notebook into a doc page. There are quite many tools, perhaps you are already using one. Or perhaps you think it is not a good idea and you prefer to have a plain md.

Could you please share with me what is the best way (technically) to turn this notebook into a doc page?

PS: of course I will do some final lifting in the notebook and get rid of the warnings :)

[dberenbaum]

Thanks @francesco086! Looks great!

I think it would be easier if one could keep a notebook next to the markdown in the documentation, and have a script or something that turns the notebook into a doc page.

It's a good idea, but we don't have capacity now to set this up in our existing docs framework. We have had many discussions around it, but for now the cost of manually updating is lower than changing our docs framework.

Could you please share with me what is the best way (technically) to turn this notebook into a doc page?

It's fine to include the notebook as a link in the docs and as an example in dvclive itself, but regardless we will want to have the basics in a markdown page here. See the guide for contributing.

Are you also planning to add something to the evidently docs?

[francesco086]

Thanks @dberenbaum !

Yes, I plan to add the same in the evidently docs.

Alright, I made the last lifting to the notebook, and now I am going to prepare the md document (will read how-to-contribute first).

[francesco086]

Et voila': iterative/dvc.org#4918

By the way, if it is not too much work, I would really appreciate if you could add the hacktoberfest tags for this PR, as I am participating.