Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add azblob support to BlobFS #13

Merged
merged 1 commit into from
May 2, 2021
Merged

Add azblob support to BlobFS #13

merged 1 commit into from
May 2, 2021

Conversation

hairyhenderson
Copy link
Owner

@hairyhenderson hairyhenderson commented May 2, 2021

Fixes #8

Adds support for Azure Blob Storage.

There's a weird quirk in how azblob seems to work... Running fs.ReadDir on it reports two DirEntries - one which has dir mode, and another that doesn't...

This is also obvious with the following command:

$ az storage blob list --delimiter / --account-name azureopendatastorage --container-name citydatacontainer --prefix Crime/Processed/2020/1/20/

One tell-tale sign that a certain item is a directory is that it'll have no ContentType. There is a default otherwise, so for now I'm simply ignoring the file entry with no content type.

As for unit tests, it may not be as simple to test this as it was for s3/gs - I can't find an appropriate fake Azure Blob package. There is "Azurite", but that's not written in Go, and seems designed to run as a separate process. Maybe I can write a decent integration test at some point.

For now I'll test manually with Azure's Open Datasets, and wire it off in CI.

Signed-off-by: Dave Henderson [email protected]

@hairyhenderson hairyhenderson self-assigned this May 2, 2021
@hairyhenderson hairyhenderson added this to the v1.0 milestone May 2, 2021
@hairyhenderson hairyhenderson force-pushed the azblob-support-8 branch 2 times, most recently from ba06f4b to dede2d3 Compare May 2, 2021 20:43
@hairyhenderson hairyhenderson marked this pull request as ready for review May 2, 2021 20:55
Signed-off-by: Dave Henderson <[email protected]>
@github-actions github-actions bot added the documentation Improvements or additions to documentation label May 2, 2021
@hairyhenderson
Copy link
Owner Author

While I'm not confident that this implementation works flawlessly, I'm going to merge this anyway... In theory it should work pretty well most of the time 😉

@hairyhenderson hairyhenderson merged commit 72329b0 into main May 2, 2021
@hairyhenderson hairyhenderson deleted the azblob-support-8 branch May 2, 2021 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Azure Blob Store (azblob) support
1 participant