Upgrade Azure SDK from azure-storage-blob-go to azure-sdk-for-go

[LasseHels]

What this PR does:
This pull request updates Tempo's azure package to use the azure-sdk-for-go SDK instead of the deprecated azure-storage-blob-go SDK.

This pull request aims to be fully backwards compatible, and is mainly a refactor. At least one functional change is included:

• Requests to Azure Blob Storage will now be retried once by default instead of zero time.

See https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/storage/azblob/migrationguide.md for a non-exhaustive list of the differences between the two SDKs.

Which issue(s) this PR fixes:
Fixes #2129
Fixes #2835

Checklist

• Tests updated - Note that no new tests have been added, as this pull request should be functionally equivalent to main.
• Documentation added - I'm not sure documentation updates are necessary for this change.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[LasseHels]

This pull request is not ready for review, but anyone interested is welcome to take a look at the work in progress.

[LasseHels]

Tests are passing locally, and this should be ready for review. I can't run the pull request workflows myself, and am thus unable to verify that the CI tests will also pass.

[joe-elliott]

So far so good. I'm going to build this image and run in our azure dev environment. Will report back if there are issues.

[LasseHels]

@joe-elliott Thanks for your review! I've got some internal tasks I need to prioritise, after which I will take a look at this again. ETA a couple of days.

[LasseHels]

@joe-elliott Comments have been addressed. Are you able to take another look?

[LasseHels]

@joe-elliott Thank you for the review. All comments have been addressed. Can you take another look? 👀

[joe-elliott]

Amazing contribution. Thank you 🙏

Looks like there's a changelog conflict. Clean that up and we'll merge!

[LasseHels]

Conflicts have been fixed and this should be ready for a merge!

[LasseHels]

Looks like a test is failing, I'm taking a look.

[LasseHels]

Hmm, looks like the TestHedge/hedge_disabled test panicked. I can't reproduce the issue locally, neither when running the test directly nor when running make test-with-cover-tempodb. @joe-elliott any insight? Should we re-run the pipeline?

[zalegrala]

@LasseHels If there is a failing test that you don't think is a result of your change, always feel free to re-run the failed job if you have the permission. Thanks for the PR.

[zalegrala]

I see now that the test failed on tempodb/backend/azure.TestHedge.

[zalegrala]

@LasseHels Have you been able to run this code change in your environment? We continue to see this test failure pop up intermittently and wondering about next steps. It would help to know if we believe this was only a test failure, or if the change is also experiencing issues in an Azure environment.

[zalegrala]

go test -race -cover ./tempodb/backend/azure/ -count=1000 -failfast -run='(TestHedge)' -timeout=30m is reproducing this pretty reliably for me.

[zalegrala]

The failure looks specific to the Write test and a large buffer to force the multiple calls. #2948 to work around the test. Would still like to know how this is performing in a real environment.

[LasseHels]

@zalegrala We are not running this change in our own environments. As such, I can't say whether the test failure is a false or true positive.

I might be able to spend some investigation time over the next couple of days to try to figure out what's going on. If you deem this to be an urgent issue, then you may be better off not relying on me, as I can't guarantee my availability.

[LasseHels]

@zalegrala I spent a couple of hours looking into the issue and got nowhere. I can reproduce it locally, but I don't know the root cause yet. I can't tell if the issue is in our implementation of the Azure SDK, our test, the SDK itself or somewhere else.

I also see we created an issue in the SDK repository, will link that here for visibility: Azure/azure-sdk-for-go#21555

[zalegrala]

Thanks for the info @LasseHels. I also spent some time researching and wasn't able to nail it down. I've got #2952 up to bring back the old code as v1, and use the new code as v2 with a config option to switch. This will allow us to gain some confidence as we move forward and give time for the Azure folks to dig into that issue. Keep an eye on the changelog and docs for the final name of the config option in case it changes before merge.