Consistent pagination experience across all APIs

[nfx]

Goal: present consistent entity listing for the end-user

TODO:

• token-based pagination & fix Add utility function to get all repos #58 in a fundamental way
• annotate remaining APIs

Databricks REST API has diverse pagination style:

Only 30% of APIs have pagination, though there are two different implementations of it. This PR proposes generating wrapper methods to return consistent type for a collection of all results. It also adds frequently used wrappers, like name-to-id mapping, which could later be used to generate client-level upserts.

Decisions to discuss:

Return type: slice vs iterator

• Current implementation proposes returning slice + error
• theoretically, with added complexity, we can move to iterator + error implementation, though Go ecosystem didn't settle on the standard iterator yet: discussion: standard iterator interface golang/go#54245
• other ecosystems, like Python, Java, and JavaScript should lean towards iterators

Naming: ListQueriesAll (suffix) vs ListAllQueries (infix) vs all methods returning slices (might be invasive)

• current template has "All" name suffix for methods returning all results
• 18% of APIs return slice from the platform.
• 36% of APIs return all results, but in a single-field response.

Out of scope of this PR

• splitting high-level generated interface from low-level direct mapping Split high-level generated interface from low-level direct API mapping #87
• telemetry for higher-level generated code Telemetry for generated high-level wrappers #88

[nfx]

@shreyas-goenka

[pietern]

I tried to check the 1-based vs 0-based pagination assumption with the following test:

func TestAccQueriesList(t *testing.T) {
	ctx := context.Background()
	wsc := workspaces.New()

	var qs1, qs2 []dbsql.Query

	{
		result, err := wsc.Queries.ListQueries(ctx, dbsql.ListQueriesRequest{PageSize: 10})
		require.NoError(t, err)
		qs1 = result.Results
	}

	{
		result, err := wsc.Queries.ListQueriesAll(ctx, dbsql.ListQueriesRequest{})
		require.NoError(t, err)
		qs2 = result
	}

	for i := 0; i < len(qs1) && i < len(qs2); i++ {
		assert.Equal(t, qs1[i], qs2[i], "Query at index %d not equal", i)
	}
}

It fails with an unrelated unmarshalling error:

        	Error:      	Received unexpected error:
        	            	json: cannot unmarshal object into Go struct field Parameter.results.options.parameters.value of type string
        	Test:       	TestAccQueriesList

[pietern]

This todo is very important for all pagination that isn't based on tokens.

[nfx]

no duplicates :)