Compability problem with ngx_aws_auth and nginx-vod-module

[xird]

Hi

Here's some background about our situation: We're running a VOD streaming service. We've been successfully using the ngx_aws_auth module (https://github.com/anomalizer/ngx_aws_auth/) to authenticate proxy_passed requests to S3 API end points with parts of our video platform.

Now, we decided to try using Nginx and the Kaltura nginx-vod-module (https://github.com/kaltura/nginx-vod-module) as our streamers. The idea was to proxy_pass the S3 requests like we've been doing. Note that we're not using the S3 "static website hosting" feature, but instead the S3 API end points. This is so that we can use signed requests for authentication(which is what ngx_aws_auth is for).

After installing both modules, we quickly discovered that we're getting nothing but 403s from S3. Further digging revealed that the AWS signature headers were missing from the requests sent to S3, even though the Nginx configuration had the "aws_sign" command in the appropriate location.

After debugging for a while, we found the problem at ngx_aws_auth/ngx_http_aws_auth.c:150 . The line in question calls "ngx_http_get_module_loc_conf", which, according to documentation, "..gets the module’s http location block configuration object from the request object". The returned conf object is then checked with "if(!conf->enabled) {". For requests that have been processed by the nginx-vod-module, this check returns true. In other words:

When a request has been processed with "vod hls;", the conf object returned for the request doesn't have the "module enabled" flag set for ngx_aws_auth. Since that's the case, ngx_aws_auth thinks it's not enabled, and won't sign the request.

Now I don't know anything about how Nginx modules work, so I don't know if this is a bug in ngx_aws_auth or nginx-vod-module. As such, this bug report has been added to both projects' issue queues. Please let me know your thoughts about which module is the cause of the problem, so we can continue the discussion in the appropriate ticket.

Relevant parts of the Nginx configuration follow. The "test" location is to confirm that the authentication works without nginx-vod-module.

server {
  listen     80;

  vod_mode remote;
  vod_upstream_location /s3;

  location /s3/ {
    aws_access_key "REDACTED";
    aws_key_scope "20171005/eu-west-1/s3/aws4_request";
    aws_signing_key "REDACTED";
    aws_s3_bucket my-example-video-bucket;
    aws_sign;

    rewrite ^/s3/hls/(.*) /$1 break;
    
    proxy_pass http://my-example-video-bucket.s3.amazonaws.com;    
  }

  location /hls/ {
    vod hls;
  }

  location /test/ {
    aws_sign;
    rewrite ^/test/(.*) /$1 break;
    proxy_pass http://my-example-video-bucket.s3.amazonaws.com;
  }
}

[xird]

The author of the nginx-vod-module commented on the matter in that module's issue queue: kaltura/nginx-vod-module#717 (comment)

What are your thoughts on the best way to fix the issue?

[xird]

I've created a pull request based on instructions by @erankor. The only change is to move the auth module from NGX_HTTP_ACCESS_PHASE to NGX_HTTP_PREACCESS_PHASE. I've tested this on our server that uses both ngx_aws_auth and nginx-vod-module, and the requests to S3 are now correctly signed.

[anomalizer]

I need some more time to understand the best way to fix this for everybody. It appears that there are two plugins fighting over who controls the access phase.

The ideal solution would be some sort of chaining of access rules. The workaround that you have found seems to be to attach one of the plugins to a different phase. This doesn't sound like the correct way to do things as this could cause issues with some other plugin that wishes to operate in the pre-access phase.

[erankor]

That is inaccurate, nginx-vod-module does not use the access phase, and even if it did, there is no problem to register multiple access phase handlers. The problem here is that the access phase is skipped for subrequests, and nginx-vod-module uses subrequests to issue the upstream requests. Any use of subrequests will not work with this module, it's not specific to nginx-vod-module.