Implement support for deleting directories recursively

[carlspring]

Task Description

We need to implement support for deleting directories recursively in the org.carlspring.cloud.storage.s3fs.S3FileSystemProvider class, as this currently only works for regular files, or empty directories and does not recurse into directories, nor remove any files that exist in them.

Tasks

The following tasks will need to be carried out:

• Study the code of the org.carlspring.cloud.storage.s3fs.S3FileSystemProvider and propose the most efficient way to do this.
• Implement the necessary changes.
• Implement test cases.

Useful links

• Blog - Multi-Object Delete Announcement
• Dev Guide - Multi-Object Delete Docs and Examples
• Dev Guide - Deleting objects
• Dev Guide - Listing Keys Hierarchically Using a Prefix and Delimiter
• API Delete Objects

Help

• Our chat channel
• Points of contact:
  • @carlspring
  • @ptirador
  • @steve-todorov
  • @sbespalov

[steve-todorov]

The docs state there's a limit for deleting multiple objects:

Delete multiple objects—Amazon S3 also provides the Multi-Object Delete API that you can use to delete up to 1000 objects in a single HTTP request.

"Directories" can contain much more than 1000 objects and they can be nested (which also counts as an object within the parent). The way I see this working would be to have something like this:

1. A method which Lists Keys Hierarchically Using a Prefix and Delimiter should return a portion of the records for the path which is going to be deleted. The docs don't state it, but the source code mentions there's a limit of 1000 objects as well.

// s3-2.15.57.jar/codegen-resources/service-2.json 
"ListObjectsV2Request" { 
   ...
   "MaxKeys":{
      "shape":"MaxKeys",
      "documentation":"<p>Sets the maximum number of keys returned in the response. By default the API returns up to 1,000 key names. The response might contain fewer keys but will never contain more.</p>",
      "location":"querystring",
      "locationName":"max-keys"
   }
   ...
},

2. A method which does multi-object deletion following the Deleting Multiple Objects example

3. while(continue != false) deleteObjects; listObjects = getObjects(); continue = listObjects.size() > 0;

4. If there are errors during the deletion - I would assume the operation should stop (unless a force=true argument was provided?)

[edmang]

@steve-todorov concerning the 4.
As the delete(...) method from S3FileSystemProvider is inherited from java.nio.file.spi.FileSystemProvider:
public abstract void delete(Path path) throws IOException;
the boolean force=true could not be used as argument in the method ... :/

Maybe we could :

• throw the SdkException at S3FileSystemProvider::delete level (hence, let the user to decide how to handle that case) ?
• or wrap it and throw it as an IOException in order to keep only 1 base exception for the delete method ?

any idea/suggestion is welcomed.