sidecar task does not work with "gpu_limit"

[giordyb]

I have an issue using "gpu_limit" with a sidecar task (i need this in order to enable "privileged mode" on the pod), it fails with the following error:

Workflow[yolotrain:development:train.simple_gpu.SimpleWorkflow] failed. RuntimeExecutionError: max number of system retry attempts [31/30] exhausted. Last known status message: failed at Node[a]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [sidecar]: [Invalid] failed to create resource, caused by: Pod "zxq9zn3huq-a-0" is invalid: [spec.containers[0].resources.limits[gpu]: Invalid value: "gpu": must be a standard resource type or fully qualified, spec.containers[0].resources.limits[gpu]: Invalid value: "gpu": must be a standard resource for containers, spec.containers[0].resources.requests[gpu]: Invalid value: "gpu": must be a standard resource type or fully qualified, spec.containers[0].resources.requests[gpu]: Invalid value: "gpu": must be a standard resource for containers]

this is the code that I tested it with:

def generate_pod_spec_for_task():
    pod_spec = generated_pb2.PodSpec()
    primary_container = generated_pb2.Container(name="primary")
    primary_container.securityContext.privileged = True
    pod_spec.containers.extend([primary_container])
    return pod_spec

@inputs(a=Types.Integer)
@outputs(b=Types.Integer)
@sidecar_task(
    pod_spec=generate_pod_spec_for_task(), primary_container_name="primary", gpu_limit="1"
)
def add_one(wf_params, a, b):
    # TODO lets add a test that works with tensorflow, but we need it to be in
    # a different container
    b.set(a + 1)

@workflow_class
class SimpleWorkflow(object):
    input_1 = Input(Types.Integer)
    input_2 = Input(Types.Integer, default=5, help="Not required.")
    a = add_one(a=input_1)
    output = Output(a.outputs.b, sdk_type=Types.Integer)

the same code using @python_task(gpu_limit="1") instead of the sidecar_task works fine.

as a workaround I had to specify the limits inside the generate_pod_spec_for_task() like this:

def generate_pod_spec_for_task():
    pod_spec = generated_pb2.PodSpec()
    # pod_spec.nodeSelector.update({"accelerator": "nvidia-gtx1080ti"})

    resource_requirements = generated_pb2.ResourceRequirements()

    resource_requirements.requests["nvidia.com/gpu"].CopyFrom(
        _lazy_k8s.io.apimachinery.pkg.api.resource.generated_pb2.Quantity(string="1")
    )
    resource_requirements.limits["nvidia.com/gpu"].CopyFrom(
        _lazy_k8s.io.apimachinery.pkg.api.resource.generated_pb2.Quantity(string="1")
    )
    resource_requirements.requests["cpu"].CopyFrom(
        _lazy_k8s.io.apimachinery.pkg.api.resource.generated_pb2.Quantity(string="4")
    )
    resource_requirements.limits["cpu"].CopyFrom(
        _lazy_k8s.io.apimachinery.pkg.api.resource.generated_pb2.Quantity(string="4")
    )
    resource_requirements.requests["memory"].CopyFrom(
        _lazy_k8s.io.apimachinery.pkg.api.resource.generated_pb2.Quantity(string="20Gi")
    )
    resource_requirements.limits["memory"].CopyFrom(
        _lazy_k8s.io.apimachinery.pkg.api.resource.generated_pb2.Quantity(string="20Gi")
    )
    primary_container = generated_pb2.Container(name="primary")
    primary_container.resources.CopyFrom(resource_requirements)
    primary_container.securityContext.privileged = True

    pod_spec.containers.extend([primary_container])

    return pod_spec

I'm running this on baremetal hardware, k8s v1.16.1 and nvidia k8s device plugin 1.0.0-beta4

[giordyb]

Hi,

I just tried the example again after upgrading flytekit and I'm getting the same error but uppercase instead of lowercase...do I need to update anything else?

Workflow[yolotrain:development:object_detection.simple_gpu.SimpleWorkflow] failed. RuntimeExecutionError: max number of system retry attempts [31/30] exhausted. Last known status message: failed at Node[a]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [sidecar]: [Invalid] failed to create resource, caused by: Pod "v7c7qb7ldf-a-0" is invalid: [spec.containers[0].resources.limits[CPU]: Invalid value: "CPU": must be a standard resource type or fully qualified, spec.containers[0].resources.limits[CPU]: Invalid value: "CPU": must be a standard resource for containers, spec.containers[0].resources.limits[GPU]: Invalid value: "GPU": must be a standard resource type or fully qualified, spec.containers[0].resources.limits[GPU]: Invalid value: "GPU": must be a standard resource for containers, spec.containers[0].resources.requests[CPU]: Invalid value: "CPU": must be a standard resource type or fully qualified, spec.containers[0].resources.requests[CPU]: Invalid value: "CPU": must be a standard resource for containers, spec.containers[0].resources.requests[GPU]: Invalid value: "GPU": must be a standard resource type or fully qualified, spec.containers[0].resources.requests[GPU]: Invalid value: "GPU": must be a standard resource for containers]

Update: I've done some more troubleshooting, it looks like removing the .lowercase() also makes the cpu_limit invalid (before the fix it would work and only the GPU limit would be invalid)

[katrogan]

Thanks for pointing this out, will try a different fix again. Sorry for the trouble 😓

[kumare3]

@katrogan i think we will need to create a new lyft/flyte release so that he can deploy the new propeller

[katrogan]

@kumare3 lyft/flyte has already been updated with the propeller fix

[kumare3]

Closing the issue