Unusable Buildpacks If Upload Fails

[johha]

Issue

If a buildpack is uploaded (via CC API or CF-CLI) and the bits upload fails (e.g. due to network issues) the buildpack will be still listed by cf buildpacks and seems usable. However the buildpack is not usable and push operations will fail with Unsupported buildpack type.
This can also lead to issues when updating an existing buildpack. In that case the error can be even more confusing as the buildpack worked before.

Context

This is relevant for foundries in landscapes where there is only limited internet access and thus offline buildpacks are needed.

Steps to Reproduce

1. Create Buildpack with CF CLI (cancel while uploading bits)

   $ cf create-buildpack test_buildpack bp.zip 100
   
   Creating buildpack test_buildpack as provisioned_user_cf_admin...
   OK
   
   Uploading buildpack test_buildpack as provisioned_user_cf_admin...
   1.56 MiB / 1.01 GiB [>----------------------------------------------------------------------------------------------------------------------------] 0.15%^C

   Alternatively buildpack can be created with v3 API (v2 shows the same behavior).
   Uploading of bits can be neglected as the error is the same.

   $ curl "$CF_API/v3/buildpacks" \
     -X POST \
     -H "Authorization: $(cf oauth-token | grep bearer)" \
     -H "Content-type: application/json" \
     -d '{
       "name": "cf7_buildpack",
       "position": 42,
       "enabled": true,
       "locked": false,
       "stack": "cflinuxfs3"
     }'

2. Push app

   $ cf push test_app  -b test_buildpack --no-route
   Pushing app test_app to org test-org / space my-test as provisioned_user_cf_admin...
   Getting app info...
   Creating app with these attributes...
   + name:         test_app
     path:         /tmp
     buildpacks:
   +   test_buildpack
   
   Creating app test_app...
   Comparing local files to remote cache...
   Packaging files to upload...
   Uploading files...
   110.73 KiB / 110.73 KiB [=======================================================================================================================] 100.00% 1s
   
   Waiting for API to complete processing files...
   
   Staging app and tracing logs...
   Unexpected Response
   Response code: 500
   CC code:       0
   CC error code:
   Request ID:    110e7004-0ac5-46b0-492b-92e6512a3a79::a0c9ac76-d220-4559-87a4-fed3a7ef78d6
   Description:   {
     "description": "Stager error: Unsupported buildpack type: 'test_buildpack'",
     "error_code": "CF-StagerError",
     "code": 170011
   }

Expected result

I would expect that the buildpack can't be used at all and is not shown under cf-buildpacks.
Furthermore the error message does not indicate an issue with the buildpack but instead rather indicates an app issue.

Current result

Stager error: Unsupported buildpack type: 'test_buildpack'

Possible Fix

Correct/better behavior would be to make a buildpack first available for usage (cf-buildpacks) after the bits are uploaded/available.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/173038631

The labels on this github issue will be updated when the story is started.

[piyalibanerjee]

Hello @johha ! Thanks for bringing this to our attention. We (@matt-royal and I) are currently working on introducing an error message if a user attempts to push an app with an unavailable buildpack. We're also planning to introduce a filter to the GET /v3/buildpacks endpoint to enable listing buildpacks that are ready for use. We may not be able to make changes to the output of the CLI command cf buildpacks since it may be a breaking change for other users. Let us know if there's anything else we can do to help.

[cwlbraa]

we've fixed the error to be more specific:

cf push test_app  -b cf7_buildpack --no-route
Pushing app test_app to org org / space space as admin...
For application 'test_app': buildpack "cf7_buildpack" must be in ready state
FAILED

[cwlbraa]

^ we created this issue on the CLI to further improve unready buildpack visibility

[stephanme]

Hello @cwlbraa , thanks for taking care.

Better error messages are good first step. But is it also possible to to handle cf create/update-buildpack in a sort-of transactional manner? This is especially important for cf update-buildpack because otherwise there is a risk of cf-push outages when e.g. the upload of a new version of the java-buildpack fails (which happens for us as part of an unattended landscape update).

[cwlbraa]

I think the way large file uploads are handled (by nginx and separate worker process) would make it very difficult to get those uploads inside of a DB transaction.

We hopefully won't have this problem in cf-for-k8s, though, as we're decoupling the registry upload and the buildpacks list configuration change. Neither will be edited via the CF API, but in the container registry and k8s directly.

[matt-royal]

The changes we made to cloud_controller_ng were causing database deadlock errors, so we reverted the original work. We should be able to revisit this next week and hopefully will be able to find a way to add the validation without causing deadlocks.

[Gerg]

This issue should be resolved in capi-release 1.96

[johha]

Thanks for improving the error message!
One thing which could be improved is that the broken buildpack is still listed when running cf buildpacks without indication if it's broken or not. Therefore as a app developer I'd assume that this buildpack is usable. Is this a bug for the cf-cli?

[cwlbraa]

in the comments above: cloudfoundry/cli#1952