panic generating crd: invalid schema type TypeInvalid

[jaylevin]

What happened?

I am trying to generate a new Upjet provider from the Elastic Cloud Terraform Provider.

The first resource I would like to generate is ec_deployment but I am encountering an error during make generate:

panic: cannot generate crd for resource ec_deployment: 
cannot build types for ElasticCloudDeployment: 
cannot build the Types for resource "ec_deployment": 
cannot infer type from schema of field apm: invalid schema type TypeInvalid

The only related issue I could find was #121, but it looks like that issue was fixed by #130 and released in the latest version of upjet (that I'm using): v1.1.0.

How can we reproduce it?

Try running make generate in the repository: https://github.com/jaylevin/provider-elastic-cloud

Schema.json: https://github.com/jaylevin/provider-elastic-cloud/blob/main/config/schema.json

Any help would be greatly appreciated, thanks in advance 😃

[haarchri]

did you tried v1.2.3 for upjet ? today its only as tag created

[smileisak]

@haarchri I have the same issue with provider-ovh, trying to update the terraform provider to last version I got the same error, because of a new created resource that have a nested_type.

make generate
14:23:32 [ .. ] generating provider schema for ovh/ovh 0.39.0
14:23:33 [ OK ] generating provider schema for ovh/ovh 0.39.0
14:23:33 [ .. ] verify go modules dependencies have expected content
all modules verified
14:23:33 [ OK ] go modules dependencies verified
14:23:34 [ .. ] go generate linux_arm64
panic: cannot generate crd for resource ovh_cloud_project_alerting: cannot build types for ProjectAlerting: cannot build the Types for resource "ovh_cloud_project_alerting": cannot infer type from schema of field formatted_monthly_threshold: invalid schema type TypeInvalid

go.md

module github.com/edixos/provider-ovh

go 1.22.0

toolchain go1.22.1

require (
	github.com/crossplane/crossplane-runtime v1.14.0-rc.0.0.20231011070344-cc691421c2e5
	github.com/crossplane/crossplane-tools v0.0.0-20230925130601-628280f8bf79
	github.com/crossplane/upjet v1.2.3
	github.com/pkg/errors v0.9.1
	gopkg.in/alecthomas/kingpin.v2 v2.2.6
	k8s.io/apimachinery v0.30.0-beta.0
	k8s.io/client-go v0.28.2
	sigs.k8s.io/controller-runtime v0.16.2
	sigs.k8s.io/controller-tools v0.13.0
)
...

[jaylevin]

did you tried v1.2.3 for upjet ? today its only as tag created

I am still getting the same error on v1.2.3 as well.

[erhancagirici]

@jaylevin @smileisak unfortunately upjet does not support protov6 schemas yet, that have nested attributes. Currently it is not possible to generate such resources.

[jaylevin]

@jaylevin @smileisak unfortunately upjet does not support protov6 schemas yet, that have nested attributes. Currently it is not possible to generate such resources.

Thanks for the update, is this feature already in progress or are we still looking for a contributor?

[danielsinai]

Hi 👋
Joining to @jaylevin question, this can be really helpful

[littlejo]

Same problem here

[jonasz-lasut]

Problem still persists in 1.4.0, joining the question

[ma-ble]

I get the same error message when creating the gitlab_group_access_token crds using upjet v1.4.1.

cannot infer type from schema of field rotation_configuration

rotation_configuration is from typ "Attributes".

[tschlaepfer]

I get the same error message when creating the platform_oidc_identity_mapping crds using upjet v1.4.0. Is there already a road map for supporting nested attributes?

[jeanduplessis]

We will discuss this topic at the next Upjet SIG meeting on Aug 14: https://docs.google.com/document/d/1QJfsAk-pdo_j2cKsJtG6W-BOQh_3ilhiRVTYVIGkDXM/edit

[ekarlso]

@jeanduplessis was anything done / mentioned on the issue on the meeting ? I can't see any updates in the document about it.

[mergenci]

We discussed the topic off-channel with @ulucinar and @erhancagirici separately. Let me summarize as best as I can. Upjet deserializes provider schema from schema.json into a Terraform Plugin SDK schema. Obviously, Terraform Plugin SDK resources have no problem being deserialized. Most of the Terraform Plugin Framework resources don't have any problems either.

Unfortunately, Plugin Framework resources that contain blocks or nested attributes in their schemas cannot be deserialized. Handling such a deserialization doesn't seem to be straightforward. The ideal solution would be to deserialize each resource type into its own schema type, i.e., deserialize Plugin SDK resources into a Plugin SDK schema and Plugin Framework resources into a Plugin Framework schema. Doing so would require architectural changes to code generation pipeline, which is currently oblivious to the resource types — code generation works with only the Plugin SDK schema, which is why all resources are deserialized into one.

A less-substantial solution would involve finding a way of deserializing blocks and nested attributes into a Plugin SDK schema. Doing so would still require special handling of those fields in code generation, but at least the effort would, hopefully, be less compared to the ideal solution.

We didn't discuss the topic in the meeting, because we prioritized the topics of the attendees. We welcome anyone to the meetings for further discussion.

[wejdross]

hello, I'm blocked as well here while trying to generate crossplane-provider from this terraform provider: https://registry.terraform.io/providers/exoscale/exoscale/latest can be used as a test case :)

I'd like to ask You to put some more attention into this issue. Best regards

[tomaszkiewicz]

+1 to the list of blocked people :)

Maybe you can suggest some workaround, like manually generating missing types? Or annotating somehow these nested blocks so code pipeline can reason from them?

Or... just assume it's an object type with "preserve unknown fields" annotation in CRD, this way we can at least generate most of the work and then do workarounds manually.

[ssch1337]

+1

[vietanhtwdk]

+1

[hown3d]

Or... just assume it's an object type with "preserve unknown fields" annotation in CRD, this way we can at least generate most of the work and then do workarounds manually.

I think it might be useful to just generate these types are interface{} in the CRD as it would at least not block generating the CRDs at all.

[bczoma]

+1