tutorials/azureml-getting-started (credits-default-model) fails to deploy with: "Expected 96 from C header, got 88 from PyObject"

[RoFz]

Operating System

MacOS

Version Information

Python Version: 3.12.7
azure-ai-ml package version: 1.21.0

Steps to reproduce

1. Load azureml-getting-started-studio.ipynb into VS Code
2. Add the code block below before the first original block (otherwise, authentication fails)

from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential

try:
   credential = DefaultAzureCredential()
   # Check if given credential can get token successfully.
   credential.get_token("https://management.azure.com/.default")
except Exception as ex:
   # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work
   # This will open a browser page for
   credential = InteractiveBrowserCredential()

3. Run everything successfully until step/item 7 (Deploy the model as an online endpoint)
4. Follow the Endpoint creation & deployment via wizard UI instructions, receive a Failed under the deployment's Provisioning State and the error log below:

Instance status:
SystemSetup: Succeeded
UserContainerImagePull: Succeeded
ModelDownload: Succeeded
UserContainerStart: InProgress

Container events:
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:12.478572Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:22.494146Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:32.498042Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:42.497885Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:52.478491Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:02.494329Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:12.494084Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:22.494119Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:32.494126Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:42.497857Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:52.494157Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:02.49442Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:12.494019Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:22.494261Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:32.494194Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:42.49788Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:52.494265Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:06:02.49823Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:06:12.494149Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502
Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:06:22.494347Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502

Container logs:
2024-10-10T11:00:05,636783121+00:00 - rsyslog/run 
2024-10-10T11:00:05,660534689+00:00 - nginx/run 
2024-10-10T11:00:05,666009973+00:00 - gunicorn/run 
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
2024-10-10T11:00:05,699105686+00:00 | gunicorn/run | 
2024-10-10T11:00:05,706316197+00:00 | gunicorn/run | ###############################################
2024-10-10T11:00:05,710913868+00:00 | gunicorn/run | AzureML Container Runtime Information
2024-10-10T11:00:05,717056363+00:00 | gunicorn/run | ###############################################
2024-10-10T11:00:05,720074210+00:00 | gunicorn/run | 
2024-10-10T11:00:05,723216459+00:00 | gunicorn/run | 
2024-10-10T11:00:05,729737860+00:00 | gunicorn/run | AzureML image information: mlflow-ubuntu20.04-py38-cpu-inference:20240805.v5
2024-10-10T11:00:05,732576004+00:00 | gunicorn/run | 
2024-10-10T11:00:05,735530049+00:00 | gunicorn/run | 
2024-10-10T11:00:05,738387494+00:00 | gunicorn/run | PATH environment variable: /opt/miniconda/envs/amlenv/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-10-10T11:00:05,741136636+00:00 | gunicorn/run | PYTHONPATH environment variable: 
2024-10-10T11:00:05,795602279+00:00 | gunicorn/run | 
2024-10-10T11:00:11,635188772+00:00 | gunicorn/run | CONDAPATH environment variable: /opt/miniconda

# conda environments:
#
base                     /opt/miniconda
amlenv                   /opt/miniconda/envs/amlenv

2024-10-10T11:00:14,920278755+00:00 | gunicorn/run | 
2024-10-10T11:00:14,925375932+00:00 | gunicorn/run | Pip Dependencies (before dynamic installation)

annotated-types==0.7.0
azure-core==1.30.2
azure-identity==1.17.1
azureml-inference-server-http==1.3.0
blinker==1.8.2
cachetools==5.4.0
certifi==2024.7.4
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cryptography==43.0.0
Flask==2.3.2
Flask-Cors==3.0.10
google-api-core==2.19.1
google-auth==2.32.0
googleapis-common-protos==1.63.2
gunicorn==22.0.0
idna==3.7
importlib_metadata==8.2.0
inference-schema==1.8
itsdangerous==2.2.0
Jinja2==3.1.4
MarkupSafe==2.1.5
msal==1.30.0
msal-extensions==1.2.0
opencensus==0.11.4
opencensus-context==0.1.3
opencensus-ext-azure==1.1.13
packaging==24.1
portalocker==2.10.1
proto-plus==1.24.0
protobuf==5.27.3
psutil==6.0.0
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycparser==2.22
pydantic==2.7.4
pydantic-settings==2.4.0
pydantic_core==2.18.4
PyJWT==2.9.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2024.1
requests==2.32.3
rsa==4.9
six==1.16.0
typing_extensions==4.12.2
urllib3==2.2.2
Werkzeug==3.0.3
wrapt==1.16.0
zipp==3.19.2

2024-10-10T11:00:20,606892801+00:00 | gunicorn/run | 
2024-10-10T11:00:20,610213452+00:00 | gunicorn/run | Entry script directory: /var/mlflow_resources/.
2024-10-10T11:00:20,613244298+00:00 | gunicorn/run | 
2024-10-10T11:00:20,618317077+00:00 | gunicorn/run | ###############################################
2024-10-10T11:00:20,621522426+00:00 | gunicorn/run | Dynamic Python Package Installation
2024-10-10T11:00:20,624719075+00:00 | gunicorn/run | ###############################################
2024-10-10T11:00:20,627784922+00:00 | gunicorn/run | 
2024-10-10T11:00:20,631190575+00:00 | gunicorn/run | Updating conda environment from /var/azureml-app/azureml-models/credit_defaults_model/1/credit_defaults_model/conda.yaml !
Retrieving notices: ...working... done
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
    current version: 24.7.1
    latest version: 24.9.1

Please update conda by running

    $ conda update -n base -c conda-forge conda



Downloading and Extracting Packages: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/opt/miniconda/envs/userenv/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.v_2c6wn8.requirements.txt', '--exists-action=b']
Pip subprocess output:
Collecting mlflow==2.16.2
  Downloading mlflow-2.16.2-py3-none-any.whl (26.7 MB)
Collecting backports-tempfile==1.0
  Downloading backports.tempfile-1.0-py2.py3-none-any.whl (4.4 kB)
Collecting cloudpickle==2.2.1
  Downloading cloudpickle-2.2.1-py3-none-any.whl (25 kB)
Collecting jaraco-collections==5.1.0
  Downloading jaraco.collections-5.1.0-py3-none-any.whl (11 kB)
Collecting pandas==1.5.3
  Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB)
Collecting psutil==5.8.0
  Downloading psutil-5.8.0.tar.gz (470 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting scikit-learn==1.5.2
  Downloading scikit_learn-1.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.3 MB)
Collecting graphene<4
  Downloading graphene-3.3-py2.py3-none-any.whl (128 kB)
Collecting sqlalchemy<3,>=1.4.0
  Downloading SQLAlchemy-2.0.35-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting alembic!=1.10.0,<2
  Downloading alembic-1.13.3-py3-none-any.whl (233 kB)
Collecting markdown<4,>=3.3
  Downloading Markdown-3.7-py3-none-any.whl (106 kB)
Collecting matplotlib<4
  Downloading matplotlib-3.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB)
Collecting Flask<4
  Downloading flask-3.0.3-py3-none-any.whl (101 kB)
Collecting mlflow-skinny==2.16.2
  Downloading mlflow_skinny-2.16.2-py3-none-any.whl (5.6 MB)
Collecting numpy<3
  Downloading numpy-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.3 MB)
Collecting Jinja2<4,>=2.11
  Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
Collecting gunicorn<24
  Downloading gunicorn-23.0.0-py3-none-any.whl (85 kB)
Collecting pyarrow<18,>=4.0.0
  Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (39.9 MB)
Collecting docker<8,>=4.0.0
  Downloading docker-7.1.0-py3-none-any.whl (147 kB)
Collecting scipy<2
  Downloading scipy-1.14.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.2 MB)
Collecting backports.weakref
  Downloading backports.weakref-1.0.post1-py2.py3-none-any.whl (5.2 kB)
Collecting jaraco.text
  Downloading jaraco.text-4.0.0-py3-none-any.whl (11 kB)
Collecting python-dateutil>=2.8.1
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Collecting pytz>=2020.1
  Downloading pytz-2024.2-py2.py3-none-any.whl (508 kB)
Collecting joblib>=1.2.0
  Downloading joblib-1.4.2-py3-none-any.whl (301 kB)
Collecting threadpoolctl>=3.1.0
  Downloading threadpoolctl-3.5.0-py3-none-any.whl (18 kB)
Collecting sqlparse<1,>=0.4.0
  Downloading sqlparse-0.5.1-py3-none-any.whl (44 kB)
Collecting opentelemetry-api<3,>=1.9.0
  Downloading opentelemetry_api-1.27.0-py3-none-any.whl (63 kB)
Collecting click<9,>=7.0
  Downloading click-8.1.7-py3-none-any.whl (97 kB)
Collecting gitpython<4,>=3.1.9
  Downloading GitPython-3.1.43-py3-none-any.whl (207 kB)
Collecting importlib-metadata!=4.7.0,<9,>=3.7.0
  Downloading importlib_metadata-8.5.0-py3-none-any.whl (26 kB)
Collecting pyyaml<7,>=5.1
  Downloading PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB)
Collecting cachetools<6,>=5.0.0
  Downloading cachetools-5.5.0-py3-none-any.whl (9.5 kB)
Collecting databricks-sdk<1,>=0.20.0
  Downloading databricks_sdk-0.34.0-py3-none-any.whl (565 kB)
Collecting opentelemetry-sdk<3,>=1.9.0
  Downloading opentelemetry_sdk-1.27.0-py3-none-any.whl (110 kB)
Collecting requests<3,>=2.17.3
  Downloading requests-2.32.3-py3-none-any.whl (64 kB)
Collecting protobuf<6,>=3.12.0
  Downloading protobuf-5.28.2-cp38-abi3-manylinux2014_x86_64.whl (316 kB)
Collecting packaging<25
  Downloading packaging-24.1-py3-none-any.whl (53 kB)
Collecting Mako
  Downloading Mako-1.3.5-py3-none-any.whl (78 kB)
Collecting typing-extensions>=4
  Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Collecting urllib3>=1.26.0
  Downloading urllib3-2.2.3-py3-none-any.whl (126 kB)
Collecting itsdangerous>=2.1.2
  Downloading itsdangerous-2.2.0-py3-none-any.whl (16 kB)
Collecting Werkzeug>=3.0.0
  Downloading werkzeug-3.0.4-py3-none-any.whl (227 kB)
Collecting blinker>=1.6.2
  Downloading blinker-1.8.2-py3-none-any.whl (9.5 kB)
Collecting graphql-relay<3.3,>=3.1
  Downloading graphql_relay-3.2.0-py3-none-any.whl (16 kB)
Collecting aniso8601<10,>=8
  Downloading aniso8601-9.0.1-py2.py3-none-any.whl (52 kB)
Collecting graphql-core<3.3,>=3.1
  Downloading graphql_core-3.2.4-py3-none-any.whl (203 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-3.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20 kB)
Collecting pyparsing>=2.3.1
  Downloading pyparsing-3.1.4-py3-none-any.whl (104 kB)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.54.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
Collecting cycler>=0.10
  Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB)
Collecting kiwisolver>=1.3.1
  Downloading kiwisolver-1.4.7-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
Collecting pillow>=8
  Downloading pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB)
Collecting contourpy>=1.0.1
  Downloading contourpy-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (322 kB)
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting greenlet!=0.4.17
  Downloading greenlet-3.1.1-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (599 kB)
Collecting jaraco.functools
  Downloading jaraco.functools-4.1.0-py3-none-any.whl (10 kB)
Collecting more-itertools
  Downloading more_itertools-10.5.0-py3-none-any.whl (60 kB)
Collecting autocommand
  Downloading autocommand-2.2.2-py3-none-any.whl (19 kB)
Collecting jaraco.context>=4.1
  Downloading jaraco.context-6.0.1-py3-none-any.whl (6.8 kB)
Collecting google-auth~=2.0
  Downloading google_auth-2.35.0-py2.py3-none-any.whl (208 kB)
Collecting gitdb<5,>=4.0.1
  Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)
Collecting zipp>=3.20
  Downloading zipp-3.20.2-py3-none-any.whl (9.2 kB)
Collecting backports.tarfile
  Downloading backports.tarfile-1.2.0-py3-none-any.whl (30 kB)
Collecting importlib-metadata!=4.7.0,<9,>=3.7.0
  Downloading importlib_metadata-8.4.0-py3-none-any.whl (26 kB)
Collecting deprecated>=1.2.6
  Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)
Collecting opentelemetry-semantic-conventions==0.48b0
  Downloading opentelemetry_semantic_conventions-0.48b0-py3-none-any.whl (149 kB)
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2024.8.30-py3-none-any.whl (167 kB)
Collecting idna<4,>=2.5
  Downloading idna-3.10-py3-none-any.whl (70 kB)
Collecting wrapt<2,>=1.10
  Downloading wrapt-1.16.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB)
Collecting smmap<6,>=3.0.1
  Downloading smmap-5.0.1-py3-none-any.whl (24 kB)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.4.1-py3-none-any.whl (181 kB)
Collecting pyasn1<0.7.0,>=0.4.6
  Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB)
Building wheels for collected packages: psutil
  Building wheel for psutil (setup.py): started
  Building wheel for psutil (setup.py): finished with status 'done'
  Created wheel for psutil: filename=psutil-5.8.0-cp310-cp310-linux_x86_64.whl size=229611 sha256=72ee154aa9763cd0ed84a9a5f203ae36bf1a6e13ecb7d149cdc2a8f3c86f8a37
  Stored in directory: /home/dockeruser/.cache/pip/wheels/12/a3/6d/615295409067d58a62a069d30d296d61d3ac132605e3a9555c
Successfully built psutil
Installing collected packages: zipp, wrapt, pyasn1, importlib-metadata, deprecated, urllib3, smmap, rsa, pyasn1-modules, opentelemetry-api, idna, charset-normalizer, certifi, cachetools, typing-extensions, six, requests, opentelemetry-semantic-conventions, numpy, more-itertools, MarkupSafe, greenlet, graphql-core, google-auth, gitdb, backports.tarfile, Werkzeug, threadpoolctl, sqlparse, sqlalchemy, scipy, pyyaml, pytz, python-dateutil, pyparsing, protobuf, pillow, packaging, opentelemetry-sdk, Mako, kiwisolver, joblib, Jinja2, jaraco.functools, jaraco.context, itsdangerous, graphql-relay, gitpython, fonttools, databricks-sdk, cycler, contourpy, cloudpickle, click, blinker, autocommand, aniso8601, scikit-learn, pyarrow, pandas, mlflow-skinny, matplotlib, markdown, jaraco.text, gunicorn, graphene, Flask, docker, backports.weakref, alembic, psutil, mlflow, jaraco-collections, backports-tempfile
Successfully installed Flask-3.0.3 Jinja2-3.1.4 Mako-1.3.5 MarkupSafe-3.0.1 Werkzeug-3.0.4 alembic-1.13.3 aniso8601-9.0.1 autocommand-2.2.2 backports-tempfile-1.0 backports.tarfile-1.2.0 backports.weakref-1.0.post1 blinker-1.8.2 cachetools-5.5.0 certifi-2024.8.30 charset-normalizer-3.4.0 click-8.1.7 cloudpickle-2.2.1 contourpy-1.3.0 cycler-0.12.1 databricks-sdk-0.34.0 deprecated-1.2.14 docker-7.1.0 fonttools-4.54.1 gitdb-4.0.11 gitpython-3.1.43 google-auth-2.35.0 graphene-3.3 graphql-core-3.2.4 graphql-relay-3.2.0 greenlet-3.1.1 gunicorn-23.0.0 idna-3.10 importlib-metadata-8.4.0 itsdangerous-2.2.0 jaraco-collections-5.1.0 jaraco.context-6.0.1 jaraco.functools-4.1.0 jaraco.text-4.0.0 joblib-1.4.2 kiwisolver-1.4.7 markdown-3.7 matplotlib-3.9.2 mlflow-2.16.2 mlflow-skinny-2.16.2 more-itertools-10.5.0 numpy-2.1.2 opentelemetry-api-1.27.0 opentelemetry-sdk-1.27.0 opentelemetry-semantic-conventions-0.48b0 packaging-24.1 pandas-1.5.3 pillow-10.4.0 protobuf-5.28.2 psutil-5.8.0 pyarrow-17.0.0 pyasn1-0.6.1 pyasn1-modules-0.4.1 pyparsing-3.1.4 python-dateutil-2.9.0.post0 pytz-2024.2 pyyaml-6.0.2 requests-2.32.3 rsa-4.9 scikit-learn-1.5.2 scipy-1.14.1 six-1.16.0 smmap-5.0.1 sqlalchemy-2.0.35 sqlparse-0.5.1 threadpoolctl-3.5.0 typing-extensions-4.12.2 urllib3-2.2.3 wrapt-1.16.0 zipp-3.20.2

done
#
# To activate this environment, use
#
#     $ conda activate userenv
#
# To deactivate an active environment, use
#
#     $ conda deactivate

WARNING: Package(s) not found: azureml-inference-server-http
2024-10-10T11:05:40,620470267+00:00 | gunicorn/run | Install azureml-inference-server-http version 1.3.0
Collecting azureml-inference-server-http==1.3.0
  Downloading azureml_inference_server_http-1.3.0-py3-none-any.whl (42 kB)
Collecting inference-schema~=1.8.0
  Downloading inference_schema-1.8-py3-none-any.whl (21 kB)
Collecting flask-cors~=3.0.1
  Downloading Flask_Cors-3.0.10-py2.py3-none-any.whl (14 kB)
Collecting flask<=2.3.2
  Downloading Flask-2.3.2-py3-none-any.whl (96 kB)
Collecting opencensus-ext-azure~=1.1.0
  Downloading opencensus_ext_azure-1.1.13-py2.py3-none-any.whl (43 kB)
Requirement already satisfied: werkzeug>=3.0.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from azureml-inference-server-http==1.3.0) (3.0.4)
Collecting pydantic-settings
  Downloading pydantic_settings-2.5.2-py3-none-any.whl (26 kB)
Collecting gunicorn==22.0.0
  Downloading gunicorn-22.0.0-py3-none-any.whl (84 kB)
Collecting pydantic~=2.7.1
  Downloading pydantic-2.7.4-py3-none-any.whl (409 kB)
Requirement already satisfied: packaging in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from gunicorn==22.0.0->azureml-inference-server-http==1.3.0) (24.1)
Requirement already satisfied: click>=8.1.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (8.1.7)
Requirement already satisfied: itsdangerous>=2.1.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (2.2.0)
Requirement already satisfied: blinker>=1.6.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (1.8.2)
Requirement already satisfied: Jinja2>=3.1.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (3.1.4)
Requirement already satisfied: Six in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask-cors~=3.0.1->azureml-inference-server-http==1.3.0) (1.16.0)
Requirement already satisfied: python-dateutil>=2.5.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from inference-schema~=1.8.0->azureml-inference-server-http==1.3.0) (2.9.0.post0)
Requirement already satisfied: wrapt<=1.16.0,>=1.14.0 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from inference-schema~=1.8.0->azureml-inference-server-http==1.3.0) (1.16.0)
Requirement already satisfied: pytz>=2017.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from inference-schema~=1.8.0->azureml-inference-server-http==1.3.0) (2024.2)
Collecting azure-core<2.0.0,>=1.12.0
  Downloading azure_core-1.31.0-py3-none-any.whl (197 kB)
Requirement already satisfied: requests>=2.19.0 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2.32.3)
Collecting azure-identity<2.0.0,>=1.5.0
  Downloading azure_identity-1.19.0-py3-none-any.whl (187 kB)
Collecting opencensus<1.0.0,>=0.11.4
  Downloading opencensus-0.11.4-py2.py3-none-any.whl (128 kB)
Requirement already satisfied: psutil>=5.6.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (5.8.0)
Collecting pydantic-core==2.18.4
  Downloading pydantic_core-2.18.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)
Collecting annotated-types>=0.4.0
  Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB)
Requirement already satisfied: typing-extensions>=4.6.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from pydantic~=2.7.1->azureml-inference-server-http==1.3.0) (4.12.2)
Requirement already satisfied: MarkupSafe>=2.1.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from werkzeug>=3.0.3->azureml-inference-server-http==1.3.0) (3.0.1)
Collecting python-dotenv>=0.21.0
  Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Collecting msal-extensions>=1.2.0
  Downloading msal_extensions-1.2.0-py3-none-any.whl (19 kB)
Collecting cryptography>=2.5
  Downloading cryptography-43.0.1-cp39-abi3-manylinux_2_28_x86_64.whl (4.0 MB)
Collecting msal>=1.30.0
  Downloading msal-1.31.0-py3-none-any.whl (113 kB)
Collecting opencensus-context>=0.1.3
  Downloading opencensus_context-0.1.3-py2.py3-none-any.whl (5.1 kB)
Collecting google-api-core<3.0.0,>=1.0.0
  Downloading google_api_core-2.21.0-py3-none-any.whl (156 kB)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2.2.3)
Requirement already satisfied: idna<4,>=2.5 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2024.8.30)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (3.4.0)
Collecting cffi>=1.12
  Downloading cffi-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (446 kB)
Collecting proto-plus<2.0.0dev,>=1.22.3
  Downloading proto_plus-1.24.0-py3-none-any.whl (50 kB)
Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.19.5 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (5.28.2)
Requirement already satisfied: google-auth<3.0.dev0,>=2.14.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2.35.0)
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2
  Downloading googleapis_common_protos-1.65.0-py2.py3-none-any.whl (220 kB)
Collecting PyJWT[crypto]<3,>=1.0.0
  Downloading PyJWT-2.9.0-py3-none-any.whl (22 kB)
Collecting portalocker<3,>=1.4
  Downloading portalocker-2.10.1-py3-none-any.whl (18 kB)
Collecting pycparser
  Downloading pycparser-2.22-py3-none-any.whl (117 kB)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (5.5.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (4.9)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (0.4.1)
Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (0.6.1)
Installing collected packages: pycparser, cffi, PyJWT, cryptography, proto-plus, portalocker, msal, googleapis-common-protos, pydantic-core, opencensus-context, msal-extensions, google-api-core, azure-core, annotated-types, python-dotenv, pydantic, opencensus, flask, azure-identity, pydantic-settings, opencensus-ext-azure, inference-schema, gunicorn, flask-cors, azureml-inference-server-http
  Attempting uninstall: flask
    Found existing installation: flask 3.0.3
    Uninstalling flask-3.0.3:
      Successfully uninstalled flask-3.0.3
  Attempting uninstall: gunicorn
    Found existing installation: gunicorn 23.0.0
    Uninstalling gunicorn-23.0.0:
      Successfully uninstalled gunicorn-23.0.0
Successfully installed PyJWT-2.9.0 annotated-types-0.7.0 azure-core-1.31.0 azure-identity-1.19.0 azureml-inference-server-http-1.3.0 cffi-1.17.1 cryptography-43.0.1 flask-2.3.2 flask-cors-3.0.10 google-api-core-2.21.0 googleapis-common-protos-1.65.0 gunicorn-22.0.0 inference-schema-1.8 msal-1.31.0 msal-extensions-1.2.0 opencensus-0.11.4 opencensus-context-0.1.3 opencensus-ext-azure-1.1.13 portalocker-2.10.1 proto-plus-1.24.0 pycparser-2.22 pydantic-2.7.4 pydantic-core-2.18.4 pydantic-settings-2.5.2 python-dotenv-1.0.1
2024-10-10T11:06:18,005275637+00:00 | gunicorn/run | 
2024-10-10T11:06:18,008905681+00:00 | gunicorn/run | ###############################################
2024-10-10T11:06:18,012396323+00:00 | gunicorn/run | Checking if the Python package azureml-inference-server-http is installed
2024-10-10T11:06:18,015871665+00:00 | gunicorn/run | ###############################################
2024-10-10T11:06:18,019106304+00:00 | gunicorn/run | 
2024-10-10T11:06:22,619276944+00:00 | gunicorn/run | 
2024-10-10T11:06:22,622712786+00:00 | gunicorn/run | ###############################################
2024-10-10T11:06:22,625700722+00:00 | gunicorn/run | AzureML Inference Server
2024-10-10T11:06:22,628489655+00:00 | gunicorn/run | ###############################################
2024-10-10T11:06:22,631337490+00:00 | gunicorn/run | 
2024-10-10T11:06:22,634345926+00:00 | gunicorn/run | Starting AzureML Inference Server HTTP.
2024-10-10 11:06:23,311 I [12] azmlinfsrv - Loaded logging config from /opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/logging.json
2024-10-10 11:06:23,408 I [12] gunicorn.error - Starting gunicorn 22.0.0
2024-10-10 11:06:23,410 I [12] gunicorn.error - Listening at: http://0.0.0.0:31311 (12)
2024-10-10 11:06:23,410 I [12] gunicorn.error - Using worker: sync
2024-10-10 11:06:23,414 I [152] gunicorn.error - Booting worker with pid: 152
/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pydantic/_internal/_fields.py:160: UserWarning: Field "model_dc_storage_enabled" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ('settings_',)`.
  warnings.warn(

Azure ML Inferencing HTTP server v1.3.0


Server Settings
---------------
Entry Script Name: /var/mlflow_resources/mlflow_score_script.py
Model Directory: /var/azureml-app/azureml-models/credit_defaults_model/1
Config File: None
Worker Count: 1
Worker Timeout (seconds): 300
Server Port: 31311
Health Port: 31311
Application Insights Enabled: false
Application Insights Key: None
Inferencing HTTP server version: azmlinfsrv/1.3.0
CORS for the specified origins: None
Create dedicated endpoint for health: None


Server Routes
---------------
Liveness Probe: GET   127.0.0.1:31311/
Score:          POST  127.0.0.1:31311/score


Warnings
---------------
Azmlinfsrv will be migrating to Pydantic 2.0 on 1/15/24. This is a breaking change for any Pydantic 1.0 code.

2024-10-10 11:06:23,996 W [152] azmlinfsrv - Found extra keys in the config file that are not supported by the server.
Extra keys = ['AZUREML_ENTRY_SCRIPT', 'AZUREML_MODEL_DIR', 'HOSTNAME']
2024-10-10 11:06:25,124 I [152] azmlinfsrv - AML_FLASK_ONE_COMPATIBILITY is set. Patched Flask to ensure compatibility with Flask 1.
Initializing logger
2024-10-10 11:06:25,127 I [152] azmlinfsrv - Starting up app insights client
2024-10-10 11:06:26,402 E [152] azmlinfsrv - Traceback (most recent call last):
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/server/user_script.py", line 77, in load_script
    main_module_spec.loader.exec_module(user_module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/var/mlflow_resources/mlflow_score_script.py", line 5, in <module>
    import mlflow
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/__init__.py", line 34, in <module>
    from mlflow import (
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/data/__init__.py", line 5, in <module>
    from mlflow.data import dataset_registry
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/data/dataset_registry.py", line 137, in <module>
    from mlflow.data.pandas_dataset import from_pandas
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/data/pandas_dataset.py", line 6, in <module>
    import pandas as pd
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/__init__.py", line 22, in <module>
    from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/compat/__init__.py", line 18, in <module>
    from pandas.compat.numpy import (
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/compat/numpy/__init__.py", line 4, in <module>
    from pandas.util.version import Version
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/util/__init__.py", line 2, in <module>
    from pandas.util._decorators import (  # noqa:F401
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/util/_decorators.py", line 14, in <module>
    from pandas._libs.properties import cache_readonly
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/server/aml_blueprint.py", line 91, in setup
    self.user_script.load_script(config.app_root)
  File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/server/user_script.py", line 79, in load_script
    raise UserScriptImportException(ex) from ex
azureml_inference_server_http.server.user_script.UserScriptImportException: Failed to import user script because it raised an unhandled exception

2024-10-10 11:06:26,404 I [152] gunicorn.error - Worker exiting (pid: 152)
2024-10-10 11:06:27,130 E [12] gunicorn.error - Worker (pid:152) exited with code 3
2024-10-10 11:06:27,132 E [12] gunicorn.error - Shutting down: Master
2024-10-10 11:06:27,133 E [12] gunicorn.error - Reason: Worker failed to boot.

Azure ML Inferencing HTTP server v1.3.0


Server Settings
---------------
Entry Script Name: /var/mlflow_resources/mlflow_score_script.py
Model Directory: /var/azureml-app/azureml-models/credit_defaults_model/1
Config File: None
Worker Count: 1
Worker Timeout (seconds): 300
Server Port: 31311
Health Port: 31311
Application Insights Enabled: false
Application Insights Key: None
Inferencing HTTP server version: azmlinfsrv/1.3.0
CORS for the specified origins: None
Create dedicated endpoint for health: None


Server Routes
---------------
Liveness Probe: GET   127.0.0.1:31311/
Score:          POST  127.0.0.1:31311/score


Warnings
---------------
Azmlinfsrv will be migrating to Pydantic 2.0 on 1/15/24. This is a breaking change for any Pydantic 1.0 code.

2024-10-10T11:06:27,400529401+00:00 - gunicorn/finish 3 0
2024-10-10T11:06:27,404048457+00:00 - Exit code 3 is not normal. Killing image.

Expected behavior

The endpoint is provisioned successfully.

Actual behavior

Provisioning State = Failed

Addition information

According to this reference, the underlying error condition showed in the logs (ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject) is due to an incompatibility between the numpy and pandas package versions. Could this discrepancy be coming from the environment image (i.e. environment="azureml://registries/azureml/environments/sklearn-1.5/labels/latest")? Or is it somehow related to my local environment?

[riccardotrevisan]

As a workaround, I created a custom environment without specifying package versions:

name: sklearn-custom
channels:
  - conda-forge
  - anaconda
dependencies:
  - python=3.10
  - pip
  - pandas
  - scipy
  - numpy
  - pip:
      - scikit-learn-intelex
      - azureml-core
      - azureml-defaults
      - azureml-mlflow
      - azureml-telemetry
      - scikit-learn
      - joblib
      - py-spy
      - debugpy
      - ipykernel
      - tensorboard
      - psutil
      - matplotlib
      - tqdm
      - py-cpuinfo
      - torch-tb-profiler

Using it as training environment in the Getting Started notebook apparently allows to deploy the model on the endpoint correctly.

Neverthless, Azure dev team should look into this and fix the tutorial, at least.