KFserving ERROR when running 'Kubeflow E2E MNIST Case'

[SeibertronSSS]

Environment:

• Fairing version: 1.0.1:
• Kubeflow version: I deploy the Kubeflow by kustomize (https://github.com/kubeflow/manifests/tree/v1.3.0-rc.0#cert-manager)
• Kubernetes version: v1.18.9:
• OS : Centos7

I have such an error when running "Kubeflow E2E MNIST Case"
‘’‘
Traceback (most recent call last):
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kfserving/api/kf_serving_client.py", line 107, in create
inferenceservice)
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/apis/custom_objects_api.py", line 178, in create_namespaced_custom_object
(data) = self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/apis/custom_objects_api.py", line 277, in create_namespaced_custom_object_with_http_info
collection_formats=collection_formats)
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 334, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
_request_timeout=_request_timeout)
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 377, in request
body=body)
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/rest.py", line 266, in POST
body=body)
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 05 May 2021 05:35:08 GMT', 'Content-Length': '391'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook "inferenceservice.kfserving-webhook-server.validator" denied the request: Canary rollout is no longer supported on v1alpha2, please convert to v1beta1 to use the feature.","reason":"Canary rollout is no longer supported on v1alpha2, please convert to v1beta1 to use the feature.","code":403}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "serving.py", line 11, in
isvc.deploy(isvc.generate_isvc())
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubeflow/fairing/deployers/kfserving/kfserving.py", line 120, in deploy
self.namespace, self.generate_isvc())
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubeflow/fairing/kubernetes/manager.py", line 159, in create_isvc
created_isvc = KFServing.create(isvc, namespace=namespace)
File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kfserving/api/kf_serving_client.py", line 111, in create
%s\n" % e)
RuntimeError: Exception when calling CustomObjectsApi->create_namespaced_custom_object: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 05 May 2021 05:35:08 GMT', 'Content-Length': '391'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook "inferenceservice.kfserving-webhook-server.validator" denied the request: Canary rollout is no longer supported on v1alpha2, please convert to v1beta1 to use the feature.","reason":"Canary rollout is no longer supported on v1alpha2, please convert to v1beta1 to use the feature.","code":403}
’‘’

I got this erroe when I was using KFserving to deploy the inferenceservice. The specific code is：

kfserving_client = KFServingClient()
kfserving_client.get(namespace=my_namespace))

I want to know why there is such a problem, whether it is related to the version of k8s

[nongmo677]

the same for me

[ycsk02]

this issue is occurred when you are using V1alpha2 canary spec which is deprecated，need to upgrade kf_serving_client.py to use v1beta1 canary spec in file fairing/kubeflow/fairing/deployers/kfserving/kfserving.py

[seoyeonhwng]

@ycsk02
could you explain more detail how to upgrade kf_serving_client.py to use v1beta1 canary spec?

[caiyuanji]

the same question for me.

[ycsk02]

@seoyeonhwng i think the way is: kf_serving_client.py import V1beta1InferenceServiceSpec and V1beta1InferenceService from kfserving, then generate_isvc generate the V1beta1InferenceService with the V1beta1InferenceServiceSpec, so the deploy will create the InferenceService in kubernetes whith V1beta1 version.

https://github.com/kserve/kserve/blob/45c3c2cf187b5255eb918fa365606f20d0e906d2/python/kserve/kserve/models/v1beta1_inference_service_spec.py#L34

https://github.com/kserve/kserve/blob/638fff64d833e3c5b378b16eda12d5e131417541/python/kserve/kserve/models/v1beta1_inference_service.py#L34

[caiyuanji]

Thank you very much @ycsk02
After I follow your recommended operation, the following prompt appears. Can you give me some more advice .

---

ApiException Traceback (most recent call last)
/usr/local/jupyter38/lib/python3.8/site-packages/kfserving/api/kf_serving_client.py in create(self, inferenceservice, namespace, watch, timeout_seconds, version)
104 try:
--> 105 outputs = self.api_instance.create_namespaced_custom_object(
106 constants.KFSERVING_GROUP,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py in create_namespaced_custom_object(self, group, version, namespace, plural, body, **kwargs)
224 kwargs['_return_http_data_only'] = True
--> 225 return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs) # noqa: E501
226

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py in create_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, body, **kwargs)
343
--> 344 return self.api_client.call_api(
345 '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'POST',

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api_client.py in call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, async_req, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host)
347 if not async_req:
--> 348 return self.__call_api(resource_path, method,
349 path_params, query_params, header_params,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api_client.py in __call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host)
179 # perform request and return response
--> 180 response_data = self.request(
181 method, url, query_params=query_params, headers=header_params,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api_client.py in request(self, method, url, query_params, headers, post_params, body, _preload_content, _request_timeout)
390 elif method == "POST":
--> 391 return self.rest_client.POST(url,
392 query_params=query_params,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/rest.py in POST(self, url, headers, query_params, post_params, body, _preload_content, _request_timeout)
274 body=None, _preload_content=True, _request_timeout=None):
--> 275 return self.request("POST", url,
276 headers=headers,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/rest.py in request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
233 if not 200 <= r.status <= 299:
--> 234 raise ApiException(http_resp=r)
235

ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '97912c25-28e9-4b41-978b-c5adbca30698', 'X-Kubernetes-Pf-Prioritylevel-Uid': '9d424c9f-1842-405a-86d2-b23fadc9c8e4', 'Date': 'Mon, 15 Nov 2021 09:49:59 GMT', 'Content-Length': '463'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook "inferenceservice.kfserving-webhook-server.v1beta1.validator" denied the request: Exactly one of [SKLearn, XGBoost, Tensorflow, PyTorch, Triton, ONNX, PMML, LightGBM, PodSpec] must be specified in PredictorSpec","reason":"Exactly one of [SKLearn, XGBoost, Tensorflow, PyTorch, Triton, ONNX, PMML, LightGBM, PodSpec] must be specified in PredictorSpec","code":403}