[BUG] Using additional labels in prometheus metrics can cause panic

[hamersaw]

Describe the bug

Creating multiple metrics with additional labels may cause the Inc function to panic. This is contingent on the number of metric keys set at the global level.

Expected behavior

Metric labels should be set and maintained independent of each other so that additional metrics do not cause panics.

Additional context to reproduce

package main

import (
	"context"

	"github.com/flyteorg/flytestdlib/contextutils"
	"github.com/flyteorg/flytestdlib/promutils"
 	"github.com/flyteorg/flytestdlib/promutils/labeled"
)

func main() {
	labeled.UnsetMetricKeys()
	labeled.SetMetricKeys(contextutils.ProjectKey, contextutils.DomainKey, contextutils.WorkflowIDKey)

	scope := promutils.NewTestScope()
	c1 := labeled.NewCounter(
		"handle_count1",
		"c1",
		scope,
		labeled.AdditionalLabelsOption{Labels: []string{"foo"}},
	)
	c2 := labeled.NewCounter(
		"handle_count2",
		"c2",
		scope,
		labeled.AdditionalLabelsOption{Labels: []string{"tasktype"}}, // this corrupts c1
	)

	ctx := context.TODO()
	ctx = contextutils.WithProjectDomain(ctx, "project", "domain")
	ctx = contextutils.WithWorkflowID(ctx, "workflow")
	ctx = contextutils.WithTaskID(ctx, "task")
	ctx = contextutils.WithTaskType(ctx, "type")
	ctx = context.WithValue(ctx, "foo", "foo")

	c1.Inc(ctx) // <--- panic here
	c2.Inc(ctx)
}

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes