[BUG] Running make dev-requirements.txt fails due to incompatible versions of protobuf

[dennisobrien]

Describe the bug

I'm trying to so all the setup necessary to work on some features for flytekit but I'm getting hung up on running make dev-requirements.txt I was able to do the initial setup described in the Contributing guide, but as I understand it, for extras and plugins, we should add these optional requirements to dev-requirements.in. Here's the error I'm getting:

make dev-requirements.in
...
Could not find a version that matches protobuf<4.0.0,>=3.5.0,>=3.6.1,>=4.21.6 (from flytekit==0.0.0+develop->-r requirements.in (line 1))
Tried: 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0, 3.0.0, 3.1.0, 3.1.0.post1, 3.1.0.post1, 3.2.0, 3.2.0, 3.3.0, 3.4.0, 3.4.0, 3.5.0.post1, 3.5.0.post1, 3.5.1, 3.5.1, 3.5.2, 3.5.2, 3.5.2.post1, 3.5.2.post1, 3.6.0, 3.6.0, 3.6.1, 3.6.1, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.8.0, 3.8.0, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.9.2, 3.9.2, 3.10.0, 3.10.0, 3.11.0, 3.11.0, 3.11.1, 3.11.1, 3.11.2, 3.11.2, 3.11.2, 3.11.3, 3.11.3, 3.11.3, 3.12.0, 3.12.1, 3.12.2, 3.12.2, 3.12.2, 3.12.4, 3.12.4, 3.12.4, 3.13.0, 3.13.0, 3.13.0, 3.14.0, 3.14.0, 3.14.0, 3.15.0, 3.15.0, 3.15.0, 3.15.1, 3.15.1, 3.15.1, 3.15.2, 3.15.2, 3.15.2, 3.15.3, 3.15.3, 3.15.3, 3.15.4, 3.15.4, 3.15.4, 3.15.5, 3.15.5, 3.15.5, 3.15.6, 3.15.6, 3.15.6, 3.15.7, 3.15.7, 3.15.7, 3.15.8, 3.15.8, 3.15.8, 3.16.0, 3.16.0, 3.16.0, 3.17.0, 3.17.0, 3.17.0, 3.17.1, 3.17.1, 3.17.1, 3.17.2, 3.17.2, 3.17.2, 3.17.3, 3.17.3, 3.17.3, 3.18.0, 3.18.0, 3.18.0, 3.18.1, 3.18.1, 3.18.1, 3.18.3, 3.18.3, 3.18.3, 3.19.0, 3.19.0, 3.19.0, 3.19.1, 3.19.1, 3.19.1, 3.19.2, 3.19.2, 3.19.2, 3.19.3, 3.19.3, 3.19.3, 3.19.4, 3.19.4, 3.19.4, 3.19.5, 3.19.5, 3.19.5, 3.19.6, 3.19.6, 3.19.6, 3.20.0, 3.20.0, 3.20.0, 3.20.1, 3.20.1, 3.20.1, 3.20.2, 3.20.2, 3.20.2, 3.20.3, 3.20.3, 3.20.3, 4.21.0, 4.21.0, 4.21.0, 4.21.0, 4.21.1, 4.21.1, 4.21.1, 4.21.1, 4.21.2, 4.21.2, 4.21.2, 4.21.2, 4.21.3, 4.21.3, 4.21.3, 4.21.3, 4.21.4, 4.21.4, 4.21.4, 4.21.4, 4.21.5, 4.21.5, 4.21.5, 4.21.5, 4.21.6, 4.21.6, 4.21.6, 4.21.6, 4.21.7, 4.21.7, 4.21.7, 4.21.7, 4.21.8, 4.21.8, 4.21.8, 4.21.8
Skipped pre-versions: 2.0.0b0, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.2.0rc1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0rc2, 3.7.0rc2, 3.7.0rc2, 3.7.0rc3, 3.7.0rc3, 3.8.0rc1, 3.8.0rc1, 3.9.0rc1, 3.9.0rc1, 3.10.0rc1, 3.10.0rc1, 3.11.0rc1, 3.11.0rc1, 3.11.0rc2, 3.11.0rc2, 3.12.0rc1, 3.12.0rc2, 3.13.0rc3, 3.13.0rc3, 3.13.0rc3, 3.14.0rc1, 3.14.0rc1, 3.14.0rc1, 3.14.0rc2, 3.14.0rc2, 3.14.0rc2, 3.14.0rc3, 3.14.0rc3, 3.14.0rc3, 3.15.0rc1, 3.15.0rc1, 3.15.0rc1, 3.15.0rc2, 3.15.0rc2, 3.15.0rc2, 3.16.0rc1, 3.16.0rc1, 3.16.0rc1, 3.16.0rc2, 3.16.0rc2, 3.16.0rc2, 3.17.0rc1, 3.17.0rc1, 3.17.0rc1, 3.17.0rc2, 3.17.0rc2, 3.17.0rc2, 3.18.0rc1, 3.18.0rc1, 3.18.0rc1, 3.18.0rc2, 3.18.0rc2, 3.18.0rc2, 3.19.0rc1, 3.19.0rc1, 3.19.0rc1, 3.19.0rc2, 3.19.0rc2, 3.19.0rc2, 3.20.0rc1, 3.20.0rc1, 3.20.0rc1, 3.20.0rc2, 3.20.0rc2, 3.20.0rc2, 3.20.1rc1, 3.20.1rc1, 3.20.1rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc2, 4.0.0rc2, 4.0.0rc2, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2
There are incompatible versions in the resolved dependencies:
  protobuf<4.0.0,>=3.5.0 (from flyteidl==1.1.22->flytekit==0.0.0+develop->-r requirements.in (line 1))
  protobuf>=4.21.6 (from grpcio-status==1.50.0->flytekit==0.0.0+develop->-r requirements.in (line 1))
  protobuf<4,>=3.6.1 (from flytekit==0.0.0+develop->-r requirements.in (line 1))
make: *** [Makefile:62: requirements.txt] Error 2

I'm using a conda environment instead of using the recommended virtualenv, but this should work, and it did for the initial setup. I have tried this in a fresh environment using

conda create -n flyte-dev python=3.8
conda activate flyte-dev
make dev-requirements.txt

But still get this error.

From a discussion on Slack, it was suggested that this is due to a new version of grpcio (1.50.0) is pulling in a version of protobuf >= 4. The suggested fix is to include grpcio<=1.47.0 and grpcio-status<=1.47.0 to requirements.in.

I'll try this in a PR.

Expected behavior

Running make dev-requirements.txt should run without failure.

Additional context to reproduce

conda create -n flyte-dev python=3.8
conda activate flyte-dev
make dev-requirements.txt

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes