Integrate darts models with MLflow (logging, autologging, loading)

[pelekhs]

Is your feature request related to a current problem? Please describe.
Missing API to log / load / autolog ML models to MLflow.

Describe proposed solution
Provide an API that integrates darts models with MLflow models and provides model logging and loading capabilities. (check this as an example for pytorch)

Describe potential alternatives
Pyfunc models and model flavors can be used right now but this quite a time consuming process to handle darts models with MLflow.

Additional context
Would it be maybe good to handle darts models as pytorch lightning models during this process? And maybe build up on the prexisting pytorch integration of MLflow?

[dennisbader]

@pelekhs, acknowledged. Give us some time to go over this and come back to you.

[turbotimon]

I would really appreciate that too. I am currently using MLFlow together with Darts for a project, but have to do it manually

[dennisbader]

Hi again. We discussed and think this would be a great idea :)
Can you help us understand what such an integration would take?

In general we are open to have something like this in place as long as it doesn't disrupt our current API. Some compromises might be fine though if they make sense.

From [FR] U8darts Integration I see that you're already in contact with the MLFlow team. We would be happy to offer guidance/supervision as well!

As an extension to the feature request description from [FR] U8darts Integration:

• All of Darts TorchForecastingModels (TFM, deep learning) are built on top of PyTorch Lightning (and underlying PyTorch). So callbacks such as MLFlow loggers are available to the user.
• In Darts, the TFM models perform the PyTorch-Lightning-related setup (trainer, datasets, etc) behind the hood. This means that our API is different to the standard PyTorch-Lightning API.

[turbotimon]

In my opinion, the main advantage is to get the mlflow.autolog() functionality working

• As it already works with Pytorch Lightning, it maybe needs just a thin wrapper
• When I run it with e.g. TCNModel, I get the following error (I didn't investigate further):
  WARNING mlflow.utils.autologging_utils: Encountered unexpected error during pytorch autologging: 'LightningDoublePrecisionModule' object has no attribute 'example_input_array'.
• Customizations I can think of:
  • Log (save/export) the full darts model, not only the underlying Pytorch model.
  • Logging some darts-specific or useful metrics and parameters.

[Sonta]

Same here, would appreciate this.
@turbotimon if it helps you, I had the exact same error until i converted my time series to 32-bit.

Getting another error about a NoneType instead, but the artifact logging seems to work nevertheless.

My main issue is the loading of the model which I would need to implement manually through a custom mlflow pyfunc (haven't done it because I have a workaround through another model registry).

Can offer some assistance if someone wants to take the lead on this, don't think I'll have the capacity to do it by myself though. Looking at the responses from the mlflow guys it would need to be some kind of 3rd-party package I guess.

[turbotimon]

@Sonta Thanks for the hint with 32-bit! I'll try it out

[solalatus]

@Sonta when you talk about the artefact logger working, you mean, you "just" instantiated a Lightning based MLFlowLogger, and that was it? Or did you have to do anything special?

Any help is much appreciated!

[chartsengrafs]

We need mlflow for Darts and time series in general so badly.

[GitHunter0]

We need mlflow for Darts and time series in general so badly.

Yes, this is essential to enable darts to be used in a production setting.

PS: For reference, sktime has this function https://www.sktime.net/en/latest/api_reference/deployment.html#mlflow

[GitHunter0]

In the meanwhile, could some please provide an example converting a darts model into an MLflow model using mlflow.pyfunc.PythonModel?

[kusumy]

I would really appreciate that too. I am currently using MLFlow together with Darts for a project, but have to do it manually

Hi, can you give me an example of this? I need it so badly, primarily when registering the model.

Many thanks for considering my request.

[turbotimon]

@kusumy i used it only for logging parameters, metrics and artifacts (manually with e.g. mlflow.log_metric(...)). I didn't used it to register models, because i guess that needs intrinsic support from darts. What i did however, was saving the darts-model as artifact. Hope that helps..

darts_model.save(MODEL_PATH+"/mymodel")
mlflow.log_artifact(MODEL_PATH, "mymodel")
...
loaded_model= TCNModel.load(MODEL_PATH+"/mymodel")

[d-sooter]

Hi there we are looking to set up a solution using mlflow and darts. Does anyone here have any tips or sample projects. Or a general structure we could follow.

We would also be very interested in this intergration and would be willing to help make it happen and/or test it.

[pelekhs]

Hi, This is an example file of an initial "darts" flavor for MLflow created within our project DeepTSF. This is used for inference and could be a starting point for creating an official Darts flavor in MLflow. Best regards, Sotiris Στις Κυρ 8 Οκτ 2023 στις 4:35 μ.μ., ο/η David Sooter < ***@***.***> έγραψε:

…

Hi there we are looking to set up a solution using mlflow and darts. Does anyone here have any tips or sample projects. Or a general structure we could follow. We would also be very interested in this intergration and would be willing to help make it happen and/or test it. — Reply to this email directly, view it on GitHub <#1618 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKXVR2Z43SOPIJTI3S5G7Z3X6KT2FAVCNFSM6AAAAAAVQCAIVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJSGAZTAMZXGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[turbotimon]

Hi there we are looking to set up a solution using mlflow and darts. Does anyone here have any tips or sample projects. Or a general structure we could follow.

We would also be very interested in this intergration and would be willing to help make it happen and/or test it.

As mlflow is not integrated in Darts, the convenient autolog will not work, so logging has to be done manually (with log_param. log_metric, log_artifact and so one). But that's no too complicated either, just take the official example from mlflow and substitute the sklearn part with Darts. Model registry will probably also not work, but i posted a workaround for that above.

So a simplified structure would be:

# 0. setup
import mlflow
import os

os.environ["MLFLOW_TRACKING_USERNAME"] = "<user>"
os.environ["MLFLOW_TRACKING_PASSWORD"] = "<pw>"
mlflow.set_tracking_uri("") # local, else set url here
mlflow.set_experiment("My_Experiment")

# 1. start run
RUN_NAME = "MyRun"
mlflow.start_run(run_name=RUN_NAME)

# 2. fit&evaluate your model
....

# 3. log your stuff
mlflow.log_param("epochs", 15)
mlflow.log_metric("accuracy", 0.99)
mlflow.log_artifact("mymodel.pickle") # if saved e.g. with model.save("mymodel.pickle")
...

# 4. end
mlflow.end_run() 

Does this help or where do you need help in particular?

[d-sooter]

Thanks a lot for the answer. Still getting going with both. I was also looking into storing the model with mlflow. While i wont be using mlflow to serve the model, i like their way of storing it. I still haven’t looked at storing and retrieving a darts model with mlflow. But since they both use picke i cant Imagine it would be an issue.

[Houcemderbel]

@pelekhs do you have an example of reading and writing models to mlflow? we are currently trying to set up tracking in mlflow but would like to continue to use Darts.

We want to use external storage to store the artifacts (s3/azure blob) and these are the general steps im thinking but if you have an example it would be very helpful.

Training step:

• create pickle model using darts. (does it have to be created as a local file or can i create it on the storage directly)
• log artifact ex: mlflow.log_artifact("mymodel.pkl")

Prediction step:

• load model
• build darts model
• run prediction

if you have any sample code it would be awesome.
PS im new to mlflow

[thuyng-ing]

Hi, is there any update on this integration? Thank you very much!

[cargecla1]

Hello everyone,

I have a solution to the MLflow integration via additional code on your notebook/ .py file making pytorch calls and activating tensorboard collection on Darts' s pytortch models. Could anyone from the team reach me out to let me know where I can add an MLflow notebook example that solves this issue? I have examples with NBEATS, TCN, NHiTS, DLinear, and NLinear doing autologging with MLflow code added. Please reach out.

[madtoinou]

Hi @cargecla1,

If you think that there is enough content to make a whole notebook, you can create a new one in the examples/ folder.

If the code is rather generic and similar across the models, it would probably be better to just add a section in the example user guide (source is in docs/userguide/) or directly in the notebook of the torch-based models (RNN, TCN, Transformer, NBeats, Tide, ...).

It would be great if you could open a draft PR as soon as possible so that we can start collecting feedback on the integration with MLflow :)

[cargecla1]

Hi @cargecla1,

If you think that there is enough content to make a whole notebook, you can create a new one in the examples/ folder.

If the code is rather generic and similar across the models, it would probably be better to just add a section in the example user guide (source is in docs/userguide/) or directly in the notebook of the torch-based models (RNN, TCN, Transformer, NBeats, Tide, ...).

It would be great if you could open a draft PR as soon as possible so that we can start collecting feedback on the integration with MLflow :)

I will create a draft PR over the weekend, thank you for your feedback!

[cargecla1]

Hi @cargecla1,

If you think that there is enough content to make a whole notebook, you can create a new one in the examples/ folder.

If the code is rather generic and similar across the models, it would probably be better to just add a section in the example user guide (source is in docs/userguide/) or directly in the notebook of the torch-based models (RNN, TCN, Transformer, NBeats, Tide, ...).

It would be great if you could open a draft PR as soon as possible so that we can start collecting feedback on the integration with MLflow :)

Hello,

I added Preview PR here: cargecla1:feature/MLflow_1618.

Let me know what you think.

[dennisbader]

Hi @cargecla1, and thanks for adding the example. I created Draft PR #2092 for this.

[cargecla1]

Hi @cargecla1, and thanks for adding the example. I created Draft PR #2092 for this.

Hello @dennisbader, excellent, thank you. Glad to be able to support.

[dnerini]

Hi there, I looked into #2092 but I don't understand how the DARTS model can be loaded from MLflow once logged as a pytorch model, am I missing on something obvious here? Thanks for the help!

[cargecla1]

Hi there, I looked into #2092 but I don't understand how the DARTS model can be loaded from MLflow once logged as a pytorch model, am I missing on something obvious here? Thanks for the help!

Hello @dnerini

Please refer to this link to see how to load the model after you register it: https://mlflow.org/docs/latest/model-registry.html

This page shows you how to use the MLflow UI to manage your registered models, there are two ways to log a model: using the mlflow.<model_flavor>.log_model() method or using the mlflow.register_model() method, above link explains this.

Please pay special attention to the subtitle:
Fetching an MLflow Model from the Model Registry

and

Serving an MLflow Model from Model Registry

Hope this helps!

Cheers

[turbotimon]

Hi there, I looked into #2092 but I don't understand how the DARTS model can be loaded from MLflow once logged as a pytorch model, am I missing on something obvious here? Thanks for the help!

Hello @dnerini

Please refer to this link to see how to load the model after you register it: https://mlflow.org/docs/latest/model-registry.html

...

I don't think that solve the issue of @dnerini. Because in the current state of #2092 (13d8113) there is no call to log_model(), which saves the model as an artifact, there is no sense in register_model as the given model_uri doesn not point to a valid model artifact. I tried it out and was not abel to log_model as there is currenty no flavor for darts (pytorch flavor does not work eather). There are some other issues with the current code as well (e.g. missing or unused import, error in L508). But as #2092 ist still a draft, let's hope there will be a full example...