Minio/Amazon S3 Storage Adapter

Minio adapter for receiving upstreams. Particularly useful for streaming multipart file uploads from the Skipper body parser.

Installation

$ npm install skipper-minio --save

If you're using this module outside of Sails (e.g. Express or a vanilla Node.js server), make sure you have skipper itself installed as your body parser.

Buyer Beware I probably would not recommend using this package in production applications just yet, and I'm sorry I didn't namespace the package from the beginning. That said it seems to do what it says on the box, but streams are still black magic as far as I'm concerned.

Usage

Using Minio

In the route(s) / controller action(s) where you want to accept file uploads, do something like:

req.file('avatar')
.upload({
  // ...options here...
  // These *could* be better off in `sails.config.uploads`
  adapter: require('skipper-minio'),
  bucket: 'avatars',
  endPoint: 'minio.mydomain.com',
  port: 9000,
  accessKey: 'ABCDEFGH123456789',
  secretKey: 'ABCDEFGH123456789ABCDEFGH123456789',
  useSSL: false,
  maxBytes: 1024 * 1024,
  maxBytesPerFile: 1024 * 1024 / 3,

  allowedFileTypes: ['image/jpeg', 'image/png', 'image/gif'],
  transformer: () => { return someDataModifyingPipeableToCropAvatars }
}, function whenDone(err, uploadedFiles) {
  if (err) return res.negotiate(err);
  else return res.ok({
    files: uploadedFiles,
    textParams: req.params.all()
  });
});

You can also use the nifty sails-hook-uploads for async/await-able upload processing in Sails v1.

Using Amazon S3

req.file('avatar')
.upload({
  adapter: require('skipper-minio'),
  bucket: 'avatars',
  accessKey: 'ABCDEFGH123456789',
  secretKey: 'ABCDEFGH123456789ABCDEFGH123456789'
}, function whenDone(err, uploadedFiles) {
  if (err) return res.negotiate(err);
  else return res.ok({
    files: uploadedFiles,
    textParams: req.params.all()
  });
});

Only allow files of type x

mmmagic is used to detect the MIME type of incoming upload streams.

You're able to restrict the types of files that will be accepted by passing an array of MIME types to the allowedFileTypes setting.

Transforming an incoming upload

You're able to specify a transformer method (or array of methods) to modify the incoming upload.

The following example accepts jpeg/png/gif uploads which it crops & resizes and stores as a jpeg using sharp.

const uuid = require('uuid/v4');
const sharp = require('sharp');

req.file('avatar')
.upload({
  // Ensure the resulting file always has a `.jpg` extension regardless of the filename of the original upload
  saveAs: (__incomingFileStream, cb) => { return cb(null, uuid() + '.jpg'); },

  // Specify allowed input file types
  allowedFileTypes: ['image/jpeg', 'image/png', 'image/gif'],

  // Uploaded files will be piped through the `transformer`s returned value before being sent to minio
  // It will be passed a single argument - the detected MIME type of the incoming stream, or `null`
  transformer: (detectedMimeType) => {
    return sharp()
      .flatten({ background: {r: 255, g: 255, b: 255, alpha: 1} })
      .resize({
        height: 100,
        width: 100,
      })
      .jpeg({ progressive: true });
  }
}, function whenDone(err, uploadedFiles) {
  if (err) return res.negotiate(err);
  else return res.ok({
    files: uploadedFiles,
    textParams: req.params.all()
  });
});

File extension normalisation

If you specify the runSaveAsAfterMimeDetection boolean option then the saveAs method will be called once again just before uploading the file to minio. This could allow you to normalise the final filename of an upload. The use case that prompted this was

• A user uploads a jpeg file with a .png extension
• A user uploads a jpeg file with a .JPEG extension, but other moving parts of the system are looking for and will only work on files with .jpg file extension

If this option is true then your saveAs method will be run multiple times.

You probably shouldn't use this (it's more for my own reference when I come back in 12 months and wonder wtf is going on) but hey, check out our config/uploads.js

module.exports.uploads = {
  // ...
  allowedFileTypes: ['image/jpeg', 'image/png', 'image/gif'],
  fileTypeMap: {
    'image/jpeg': '.jpg',
    'image/png': '.png',
    'image/gif': '.gif',
  },

  runSaveAsAfterMimeDetection: true,
  saveAs: function saveUploadAs(__file, cb) {
    const uuid = require('uuid/v4')
    const extension = (sails.config.uploads.fileTypeMap[__file.mimeType] || __file.filename.split('.').pop() || '.upload').toLowerCase()

    // Because this saveAs method is gonna run probably twice, we attach  the generated uuid to __file to save CPU cycles the next time we run
    let filename
    if (__file.__userlandFileName) {
      filename = __file.__userlandFileName
    } else {
      filename = uuid()
      __file.__userlandFileName = filename
    }

    return cb(null, filename + extension)
  }
}

TODO

• Run tests against minio &~ s3mock

Contribute

Please! I have no idea what I'm doing here 😉

To run the tests:

$ npm test

License

MIT

Based on work by Jarrod Linahan, Mike McNeil, Balderdash Design Co., Sails Co.

See LICENSE.md.

This module is intended to be uesd with the Sails framework, and is free and open-source under the MIT License.