{ "cells": [ { "cell_type": "markdown", "id": "852578c4", "metadata": {}, "source": [ "#### Small example how to use MFlow for model validation
\n", "We load existing model and log evaluator outputs using SHAP.
\n", "More in the documentation: https://mlflow.org/docs/latest/python_api/mlflow.html?highlight=mlflow%20evaluate#mlflow.evaluate" ] }, { "cell_type": "code", "execution_count": 1, "id": "1a4a3dcc", "metadata": {}, "outputs": [], "source": [ "import mlflow\n", "\n", "from sklearn.model_selection import train_test_split\n", "from sklearn import datasets\n", "\n", "tracking_uri = 'https://dev-orch-mlflow-service.dev.theorchard.io'\n", "mlflow.set_tracking_uri(tracking_uri)" ] }, { "cell_type": "code", "execution_count": 4, "id": "697603bb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "model_uri uri: s3://test-orcd-bucket/dev-aadamu/mlflow_artifacts/0/aa1d672c14d24a8dab871a3cf10a2667/artifacts/model\n" ] } ], "source": [ "experiment = 'test-experiment-776'\n", "mlflow.set_experiment(experiment)\n", "\n", "# Just in case end existing active run and select new one.\n", "mlflow.end_run() \n", "mlflow.start_run(\"50bcc4b8c86340469749be68d6672362\")" ] }, { "cell_type": "markdown", "id": "66c90a60", "metadata": {}, "source": [ "Load existing model" ] }, { "cell_type": "code", "execution_count": 52, "id": "b321a7a6", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023/09/12 07:20:07 WARNING mlflow.pyfunc: Detected one or more mismatches between the model's dependencies and the current Python environment:\n", " - scipy (current: 1.11.1, required: scipy==1.10.1)\n", "To fix the mismatches, call `mlflow.pyfunc.get_model_dependencies(model_uri)` to fetch the model's environment and install dependencies using the resulting environment file.\n" ] } ], "source": [ "logged_model = 'runs:/50bcc4b8c86340469749be68d6672362/model'\n", "\n", "# Load model as a PyFuncModel.\n", "loaded_model = mlflow.pyfunc.load_model(logged_model)" ] }, { "cell_type": "markdown", "id": "9e045f81", "metadata": {}, "source": [ "Prepare data for validation" ] }, { "cell_type": "code", "execution_count": 36, "id": "ed59db28", "metadata": {}, "outputs": [], "source": [ "iris = datasets.load_iris()\n", "\n", "# Split the data into training and test sets\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " iris.data[:,2:], iris.target, test_size=0.33, random_state=42\n", ")\n", "\n", "# Build the Evaluation Dataset from the test set\n", "eval_data = X_test" ] }, { "cell_type": "markdown", "id": "ebfc9489", "metadata": {}, "source": [ "Evaluate the logged model" ] }, { "cell_type": "code", "execution_count": 71, "id": "dccc4cca", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023/09/12 07:41:40 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "2023/09/12 07:41:40 INFO mlflow.models.evaluation.default_evaluator: The evaluation dataset is inferred as multiclass dataset, number of classes is inferred as 3\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "2023/09/12 07:41:42 INFO mlflow.models.evaluation.default_evaluator: Shap explainer Exact is used.\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "X has feature names, but DecisionTreeClassifier was fitted without feature names\n", "Unable to serialize underlying model using MLflow, will use SHAP serialization\n", "No data for colormapping provided via 'c'. Parameters 'vmin', 'vmax' will be ignored\n", "No data for colormapping provided via 'c'. Parameters 'vmin', 'vmax' will be ignored\n" ] } ], "source": [ "result = mlflow.evaluate(\n", " loaded_model,\n", " eval_data,\n", " targets=y_test,\n", " model_type=\"classifier\",\n", " evaluators=[\"default\"],\n", ")" ] }, { "cell_type": "markdown", "id": "d296913b", "metadata": {}, "source": [ "Notes" ] }, { "cell_type": "markdown", "id": "2dffd5ed", "metadata": {}, "source": [ "Ideally we should log evaluations during the model log using mlflow.start_run().
\n", "Otherwise we end up with different artifact folders:" ] }, { "cell_type": "markdown", "id": "d2f764bf", "metadata": {}, "source": [ "![image.png](artifact_folders.png)" ] }, { "cell_type": "markdown", "id": "3aa4ab49", "metadata": {}, "source": [ "Otherwise you can see new charts that give extra explanability into the model.
\n", "For example SHAP feature importance plot below:" ] }, { "cell_type": "markdown", "id": "249a4bad", "metadata": {}, "source": [ "![image.png](shap_example.png)" ] } ], "metadata": { "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 5 }