Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue-22: Set up a project documentation site using mkdocs and publish it to github.io #70

Merged
merged 25 commits into from
Sep 13, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
cdf0025
issue-22: Initial documentation.
steve-todorov Sep 1, 2020
5c06fb4
issue-22: Increasing debug level (netlify)
steve-todorov Sep 1, 2020
a67ef89
issue-22: Adding mkdocs==1.1.2 to requirements.txt
steve-todorov Sep 1, 2020
7c48610
issue-22: Polishing docs.Dockerfile.
steve-todorov Sep 1, 2020
c6b583b
issue-22: Updating requirements.txt
steve-todorov Sep 1, 2020
bbd9031
issue-22: Updating example section.
steve-todorov Sep 1, 2020
13371ba
Issue-22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 1, 2020
0c9d93a
issue-22: Bulk fixing links.
steve-todorov Sep 1, 2020
2ae1f36
issue-22: Don't trigger GH actions on documentation changes.
steve-todorov Sep 1, 2020
99f90f9
issue-22: Don't trigger GH actions on documentation changes.
steve-todorov Sep 1, 2020
d64201d
issue-22: Added SonarQube badge and changed project key in SonarQube …
steve-todorov Sep 1, 2020
14b0ae5
issue-21: Set up badges
steve-todorov Sep 1, 2020
2936379
issue-21: Added JDK support badge
steve-todorov Sep 1, 2020
b718edf
issue-21: Import the beginning of README.md directly into index.md to…
steve-todorov Sep 3, 2020
3ad9403
Issue-22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 8, 2020
ca408b4
Issue-22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 9, 2020
54fb150
Issue-22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 9, 2020
b72558c
Issue-22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 9, 2020
5725006
Issue-22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 10, 2020
9809389
Issue 22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 10, 2020
03c2676
Issue 22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 11, 2020
90fc7b2
Issue 22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 11, 2020
35edccd
issue-22: Fixing front page example.
steve-todorov Sep 11, 2020
7d41e30
issue-22: Fixing broken asset link.
steve-todorov Sep 11, 2020
7a544aa
Issue 22: Set up a project documentation site using mkdocs and publis…
carlspring Sep 12, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
name: Build and test workflow
on: [push, pull_request]
name: Build and test
on:
# Build PRs
pull_request: ~
# On push event
push:
# Build only changes pushed into the `master` branch.
branches:
- master
# Ignore documentation changes (Netlify will handle this)
paths-ignore:
- 'docs/**'
- '-*.md'
- '-**/*.md'

jobs:
build-and-test:
runs-on: ${{ matrix.os }}
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ target/
tmp/
*.versionsBackup
.m2/
site/

# Java Files #
*.jar
Expand Down
1 change: 1 addition & 0 deletions CODE-OF-CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
For our code of conduct, please check [here](https://s3fs.carlspring.org/contributing/code-of-conduct.md).
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
For details on how to contribute, please check [here](https://s3fs.carlspring.org/contributing/index.md).
215 changes: 38 additions & 177 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,196 +1,57 @@
An **Amazon AWS S3** FileSystem Provider **JSR-203** for Java 7 (NIO2)
# General

Amazon Simple Storage Service provides a fully redundant data storage infrastructure for storing and retrieving any amount of data, at any time.
NIO2 is the new file management API, introduced in Java version 7.
This project provides a first API implementation, little optimized, but "complete" to manage files and folders directly on Amazon S3.
This is an implementation of an **Amazon AWS S3** `FileSystem` provider using **[JSR-203]** (a.k.a. NIO2) for Java 8.

[![Build Status](https://travis-ci.org/Upplication/Amazon-S3-FileSystem-NIO2.svg?branch=master)](https://travis-ci.org/Upplication/Amazon-S3-FileSystem-NIO2/builds) [![Coverage Status](https://coveralls.io/repos/Upplication/Amazon-S3-FileSystem-NIO2/badge.png?branch=master)](https://coveralls.io/r/Upplication/Amazon-S3-FileSystem-NIO2?branch=master) [![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.upplication/s3fs/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.upplication/s3fs)
Amazon Simple Storage Service provides a fully redundant data storage infrastructure for storing and retrieving any
amount of data, at any time.

#### How to use
[NIO2] is the new file management API, introduced in Java version 7.

##### Download from Maven Central
This project provides a complete API implementation, for managing files and folders directly in Amazon S3.

```XML
<dependency>
<groupId>com.upplication</groupId>
<artifactId>s3fs</artifactId>
<version>2.2.2</version>
</dependency>
```
[![Master Build Status][master-build-status-badge]][master-build-status-link]
[![Docs][master-docs-badge]][master-docs-link]
[![License][license-badge]][license-link]
[![Help Contribute to Open Source][codetriage-badge]][codetriage-link]
[![GitHub issues by-label][good-first-issue-badge]][good-first-issue-link]
[![GitHub issues by-label][help-wanted-badge]][help-wanted-link]
[![GitHub issues by-label][hacktoberfest-badge]][hacktoberfest-link]
[![GitHub issues by-label][stackoverflow-badge]][stackoverflow-link]

# Compatibility

And add to your META-INF/services/java.nio.file.spi.FileSystemProvider (create if not exists yet) a new line like this: com.upplication.s3fs.S3FileSystemProvider.
We support both JDK 8 and 11.

##### S3FileSystem and AmazonS3 settings
## Documentation

All settings for S3FileSystem and for the underlying AmazonS3 connector library can be set through System properties or environment variables.
Possible settings can be found in com.upplication.s3fs.AmazonS3Factory.
You can check out our documentation [here](https://s3fs.carlspring.org).

#### Using service locator and system vars
[<--# Links -->]: #

Check that s3fs_access_key and s3fs_secret_key system vars are present with the correct values to have full access to your amazon s3 bucket.
[NIO2]: https://jcp.org/en/jsr/detail?id=203
[JSR-203]: https://jcp.org/en/jsr/detail?id=203

Use this code to create the fileSystem and set to a concrete endpoint.
[<--# Badges -->]: #
[master-build-status-link]: https://github.com/carlspring/s3fs-nio2/actions?query=branch%3Amaster
[master-build-status-badge]: https://github.com/carlspring/s3fs-nio2/workflows/Build%20and%20test%20workflow/badge.svg

```java
FileSystems.newFileSystem("s3:///", new HashMap<String,Object>(), Thread.currentThread().getContextClassLoader());
```
[master-docs-link]: https://carlspring.github.io/s3fs-nio2
[master-docs-badge]: https://img.shields.io/badge/docs-current-brightgreen.svg

##### Using service locator and amazon.properties in the classpath
[license-link]: https://opensource.org/licenses/Apache-2.0
[license-badge]: https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg

Add to your resources folder the file amazon.properties with the content:
s3fs_access_key=access-key
s3fs_secret_key=secret-key
[codetriage-link]: https://www.codetriage.com/carlspring/s3fs-nio2
[codetriage-badge]: https://www.codetriage.com/carlspring/s3fs-nio2/badges/users.svg

Use this code to create the fileSystem and set to a concrete endpoint.
[good-first-issue-link]: https://github.com/carlspring/s3fs-nio2/issues?q=is%3Aissue+is%3Aopen+label%3A%22good%20first%20issue%22
[good-first-issue-badge]: https://img.shields.io/github/issues-raw/carlspring/s3fs-nio2/good%20first%20issue.svg?label=good%20first%20issue

```java
FileSystems.newFileSystem("s3:///", new HashMap<String,Object>(), Thread.currentThread().getContextClassLoader());
```
[help-wanted-link]: https://github.com/carlspring/s3fs-nio2/issues?q=is%3Aissue+is%3Aopen+label%3A%22help%20wanted%22
[help-wanted-badge]: https://img.shields.io/github/issues-raw/carlspring/s3fs-nio2/help%20wanted.svg?label=help%20wanted&color=%23856bf9&

##### Using service locator and programatically authentication
[hacktoberfest-link]: https://github.com/carlspring/s3fs-nio2/issues?q=is%3Aissue+is%3Aopen+label%3A%22hacktoberfest%22
[hacktoberfest-badge]: https://img.shields.io/github/issues-raw/carlspring/s3fs-nio2/hacktoberfest.svg?label=hacktoberfest&color=orange

Create a map with the authentication and use the fileSystem to create the fileSystem and set to a concrete endpoint.

```java
Map<String, ?> env = ImmutableMap.<String, Object> builder()
.put(com.upplication.s3fs.AmazonS3Factory.ACCESS_KEY, "access key")
.put(com.upplication.s3fs.AmazonS3Factory.SECRET_KEY, "secret key").build()
FileSystems.newFileSystem("s3:///", env, Thread.currentThread().getContextClassLoader());
```

Complete settings lists:

* s3fs_access_key
* s3fs_secret_key
* s3fs_request_metric_collector_class
* s3fs_connection_timeout
* s3fs_max_connections
* s3fs_max_retry_error
* s3fs_protocol
* s3fs_proxy_domain
* s3fs_proxy_host
* s3fs_proxy_password
* s3fs_proxy_port
* s3fs_proxy_username
* s3fs_proxy_workstation
* s3fs_socket_send_buffer_size_hint
* s3fs_socket_receive_buffer_size_hint
* s3fs_socket_timeout
* s3fs_user_agent
* s3fs_amazon_s3_factory
* s3fs_signer_override
* s3fs_path_style_access

##### Set endpoint to reduce data latency in your applications

```java
// Northern Virginia or Pacific Northwest
FileSystems.newFileSystem("s3://s3.amazonaws.com/", env, Thread.currentThread().getContextClassLoader());
// Northern Virginia only
FileSystems.newFileSystem("s3://s3-external-1.amazonaws.com/", env, Thread.currentThread().getContextClassLoader());
// US West (Oregon) Region
FileSystems.newFileSystem("s3://s3-us-west-2.amazonaws.com/", env, Thread.currentThread().getContextClassLoader());
// US West (Northern California) Region
FileSystems.newFileSystem("s3://s3-us-west-1.amazonaws.com/", env, Thread.currentThread().getContextClassLoader());
// EU (Ireland) Region
FileSystems.newFileSystem("s3://s3-eu-west-1.amazonaws.com/", env, Thread.currentThread().getContextClassLoader());
```

For a complete list of available regions look at: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

##### How to use in Apache MINA

```java
public FileSystemFactory createFileSystemFactory(String bucketName) throws IOException, URISyntaxException {
FileSystem fileSystem = FileSystems.newFileSystem(new URI("s3:///"), env, Thread.currentThread().getContextClassLoader());
String bucketPath = fileSystem.getPath("/" + bucketName);

return new VirtualFileSystemFactory(bucketPath);
}
```

##### How to use in Spring

Add to classpath and configure:

```java
@Configuration
public class AwsConfig {

@Value("${upplication.aws.accessKey}")
private String accessKey;

@Value("${upplication.aws.secretKey}")
private String secretKey;

@Bean
public FileSystem s3FileSystem() throws IOException {
Map<String, String> env = new HashMap<>();
env.put(com.upplication.s3fs.AmazonS3Factory.ACCESS_KEY, accessKey);
env.put(com.upplication.s3fs.AmazonS3Factory.SECRET_KEY, secretKey);

return FileSystems.newFileSystem(URI.create("s3:///"), env, Thread.currentThread().getContextClassLoader());
}
}
```

Now you can inject in any spring component:

```java
@Autowired
private FileSystem s3FileSystem;
```

##### What is new 2.0.0

* Preserve URI with end slash #76
* Removed META-INF/services/java.nio.file.spi.FileTypeDetector #78
* Bucket are filestores and root directories for a bucket is the bucket itself.
* getFileName for a root Path is ```null```
* Added new method toURL to S3Path and can be customized with the property s3fs_path_style_access #83
* Improved S3Path Tests

#### Features:

* Copy and create folders and files
* Delete folders and files
* Copy paths between different providers
* Walk file tree
* Works with virtual s3 folders (not really exists and are element's subkeys)
* List buckets for the client
* Multi endpoint fileSystem

#### Roadmap:

* Performance issue (slow querys with virtual folders, add multipart submit...)
* Disallow upload binary files with same name as folders and vice versa

#### Out of Roadmap:

* Watchers

#### How to contribute

Clone the github repository:

```java
git clone https://github.com/Upplication/Amazon-S3-FileSystem-NIO2.git
```

To run the tests:

First, you must copy the file `src/test/resources/amazon-test-sample.properties` and paste in the same directory with the name amazon-test.properties. In your copy you must edit all the keys:

```
bucket_name=/your-bucket-name for test
# http://docs.aws.amazon.com/general/latest/gr/rande.html
s3fs_secret_key= your secret key for test
s3fs_access_key=your access key for test
```

Thats all, now you can run the test with the command: `mvn test` or `mvn integration-test -Pintegration-tests`

#### LICENSE:

Amazon S3 FileSystem NIO2 is released under the MIT License.
[stackoverflow-link]: https://stackoverflow.com/tags/s3fs-nio2/
[stackoverflow-badge]: https://img.shields.io/badge/stackoverflow-ask-orange.svg
69 changes: 69 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
## Documentation

This directory contains the S3FS NIO2 documentation.

## Pre-requisites

* [Docker][docker-install]
* [Docker Compose][docker-compose-install]

## Getting started

1. `docker-compose up`
2. Open [`http://localhost:8000`](http://localhost:8000)

## Debugging

1. Build image
```
docker-compose build
```

2. Log into the container
```
docker-compose run --rm -p 8000:8000 -v $(pwd)/../:/workspace --entrypoint /bin/sh mkdocs
```

2. Manually start the server (i.e. test plugins, cwd should be ./docs!)
```
mkdocs serve -a 0.0.0.0:8000
```

## Notes

* Imported snippets (`--8<-- "./filename"`) are using relative paths to `./docs`
(i.e. `./content/something` would be searched for in `./docs/content/something`)

## Used tools and extensions

| Tool | Documentation | Sources |
| ------------------------------- | ------------------------------------- | ----------------------------------- |
| mkdocs | [documentation][mkdocs] | [Sources][mkdocs-src] |
| mkdocs-material | [documentation][mkdocs-material] | [Sources][mkdocs-material-src] |
| pymdown-extensions | [documentation][pymdown-extensions] | [Sources][pymdown-extensions-src] |
| mdx_gh_links | [documentation][mdx_gh_links] | [Sources][mdx_gh_links] |
| mkdocs-redirects | [documentation][mkdocs-redirects] | [Sources][mkdocs-redirects] |
| mkdocs-markdownextradata-plugin | [documentation][mkdocs-markdownextradata-plugin] | [Sources][mkdocs-markdownextradata-plugin] |
| mkdocs-pom-parser-plugin | [documentation][mkdocs-pom-parser-plugin] | [Sources][mkdocs-pom-parser-plugin] |
| mkdocs-minify-plugin | [documentation][mkdocs-minify-plugin] | [Sources][mkdocs-minify-plugin] |


[docker-install]: https://docs.docker.com/get-docker/ "Docker Installation"
[docker-compose-install]: https://docs.docker.com/compose/install/ "Docker Compose Installation"

[mkdocs]: https://www.mkdocs.org "Mkdocs"
[mkdocs-src]: https://github.com/mkdocs/mkdocs "Mkdocs - Sources"

[mkdocs-material]: https://squidfunk.github.io/mkdocs-material/ "Material for MkDocs"
[mkdocs-material-src]: https://github.com/squidfunk/mkdocs-material "Material for MkDocs - Sources"

[pymdown-extensions]: https://facelessuser.github.io/pymdown-extensions "PyMdown Extensions"
[pymdown-extensions-src]: https://github.com/facelessuser/pymdown-extensions "PyMdown Extensions - Sources"


[mdx_gh_links]: https://github.com/Python-Markdown/github-links/
[mkdocs-redirects]: https://github.com/datarobot/mkdocs-redirects
[mkdocs-markdownextradata-plugin]: https://github.com/rosscdh/mkdocs-markdownextradata-plugin
[mkdocs-pom-parser-plugin]: https://github.com/steve-todorov/mkdocs-pom-parser-plugin "A pom.xml file parser for mkdocs"

[mkdocs-minify-plugin]: https://github.com/byrnereese/mkdocs-minify-plugin
Binary file added docs/content/assets/fixed-the-code-will-be.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading