Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #8
Adds support for Azure Blob Storage.
There's a weird quirk in how azblob seems to work... Running
fs.ReadDir
on it reports two DirEntries - one which has dir mode, and another that doesn't...This is also obvious with the following command:
$ az storage blob list --delimiter / --account-name azureopendatastorage --container-name citydatacontainer --prefix Crime/Processed/2020/1/20/
One tell-tale sign that a certain item is a directory is that it'll have no ContentType. There is a default otherwise, so for now I'm simply ignoring the file entry with no content type.
As for unit tests, it may not be as simple to test this as it was for s3/gs - I can't find an appropriate fake Azure Blob package. There is "Azurite", but that's not written in Go, and seems designed to run as a separate process. Maybe I can write a decent integration test at some point.
For now I'll test manually with Azure's Open Datasets, and wire it off in CI.
Signed-off-by: Dave Henderson [email protected]