Added verbage for setting up AWS Lambda with Zappa. #167

fergyfresh · 2017-12-28T07:48:35Z

Added setup to get this working on AWS Lambda.

I used Zappa which is a command line utility I've used before and given that its a music related app I thought that having a dependency on Zappa would be pretty cool. I don't force it on anyone though with the requirements.txt file, but I get them from this code to AWS in < 5 minutes just like the Heroku example does.

If we want to add awscli and zappa to dependencies I can pip freeze my env and update the requirements.txt file.

digiltd · 2017-12-28T13:26:28Z

Cool, I will give this a go, I used to use Zappa for the Kodi Skill as well (but I have since changed my kodi skill to not be dependant on Zappa johnwheeler/flask-ask#197)

fergyfresh · 2017-12-28T14:26:34Z

So @digiltd did you use environment variables with that? I started down that path and got wrapped around a pole and stumbled upon Zappa. It doesn't add any code, just 2 dependencies. The Heroku deploy also adds a few dependencies (if you deploy from command line).

Also wouldn't the WSGI wrapper that's referenced in your hyperlink be conditional to only this deployment? Then we'd need feature flags for the entry point based on what we are using to deploy, right?

For reasons surrounding those concerns I would argue that this solution is perfectly fine, I found it enjoyable.

digiltd · 2017-12-28T14:57:38Z

Sorry, my fault, I wasn't clear enough. It was my Kodi skill I was talking about that no longer uses Zappa, not GeeMusic. I briefly tried to get GeeMusic working in the same way (without Zappa) but failed. The non-zappa way uses the standard Lambda "arn" address to connect to the Skill which is the thing I like about that method (compared to Zappa creating an API Gateway). But I don't think it would work with GeeMusic anyway given the way GeeMusic works so I didn't pursue it.

I'm not against Zappa at all, my beef I had with it with the Kodi Skill was mainly due to all the extra bits it creates (API Gateway, time based triggers, additional users) which are fine and all above board.

But I found that a lot of beginners (certainly on the Kodi forums) would zappa deploy once, then discover it didn't work, so would try to zappa deploy multiple times from different locations. Because they didn't zappa undeploy their aws account would be a mess of unused lambda functions, gateways, users, logs and everything else created by zappa each time they deployed.

Plus trying to find something in CloudWatch logs was always a challenge with the 4 min "keep warm" feature of zappa :)

Note that none of the above is me commenting on your idea of using Zappa with GeeMusic, just some ramblings. I am gonna try GeeMusic using Zappa right now, would rather do that than my current solution of installing it on one of my digitalocean vps :)

fergyfresh · 2017-12-28T16:25:25Z

I didn't take it as snark one bit @digiltd so you're fine on that aspect. I was just challenging you to make sure I understood what I was talking about, with the validation of some external third party. I wanted to make sure I understood what it took to implement parts of what you were talking about. Considering there is an optional setup to use docker I cannot see this setup being too hard for anyone that already has this app deployed and working in production for themselves. That's just my $0.02.

If you have trouble following my instructions let me know. I can clarify anything or help you debug.

digiltd · 2017-12-28T23:18:31Z

No starkyness read or taken :)

But I think I need some hints as I'm stuck.

A few questions...

Is your install using aws/lambda/zappa to actually serve the files back to Alexa? Whilst I have been having some success getting it to work (logging into google music, finding tracks and then supposedly "streaming" them) I am not able to actually hear anything. When I try playing the alexa/stream/31232131 url from the log in a browser, I get {"message": "Internal server error"}
with previous installation methods (local using ngrok & self hosted on a server) I am able to get the audio file this way.

Did you do any additional tweaking (increased memory/timeout) to the Lambda function?

When you ran zappa init what did you enter when this popped up:

It looks like this is a Flask application.
What's the modular path to your app's function?
This will likely be something like 'your_module.app'.
We discovered: geemusic.__init__.app
Where is your app's function? (default 'geemusic.__init__.app'):

Did you adjust any other files in the repo or was it a direct git clone of the stevenleeg repo?

Anything out of the ordinary here:

{
    "dev": {
        "app_function": "geemusic.app",
        "aws_region": "eu-west-1",
        "profile_name": "default",
        "project_name": "geemusic",
        "runtime": "python3.6",
        "keep_warm": false,
        "use_precompiled_packages": true,
        "s3_bucket": "zappa-geemusic",
        "environment_variables": {
          "APP_URL":"https://123123123.execute-api.eu-west-1.amazonaws.com/dev",
          "DEBUG_MODE": "False",
          "GOOGLE_EMAIL":"[email protected]",
          "GOOGLE_PASSWORD":"abc123",
          "ANDROID_ID": "123123",
          "USE_LIBRARY_FIRST": "True"
        }
    }
}

I am deploying from a digitalocean vps (Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-104-generic x86_64))

I am guessing you are deploying from a linux box? My main machine is a Mac, and as with previous zappa deployments there are issues due to when you pip install on a Mac it compiles key dependancies like Cryptography and cffi for the Mac platform. Zappa then uploads the Mac executables which don't work on Lambda. Zappa is supposed to spot these and swap them out with versions from lambda-packages, it is a bit hit and miss, I usually have to replace these manually (which I have done) but it is something to be aware of and possibly mention in the readme.

When using lambda how does the server (in server.py) get initialised? When I say the `app_function" is "geemusic.app" this successfully starts the Skill (for handling the request from Alexa) but what starts the server that serves the audio files?

To summarise, I have it working where I can call the skill "play playlist sleepy" and she responds "playing songs from Sleepy" looking at the logs it looks like everything is working, I can see the stream url (which looks something like: https://123123123.execute-api.eu-west-1.amazonaws.com/dev/alexa/stream/4432hjk2-2233-1111-4566-123123123) but I can't play it.

Any ideas?

fergyfresh · 2017-12-28T23:42:52Z

You're spot on. The second half of the whole thing doesn't work. Thanks for spelling it out for me though. Thats gonna be a tommorrow thing as I'm busy for the remainder of today/tonight.

digiltd · 2017-12-29T01:28:02Z

Lol, no problem, though I had assumed it was a tried and tested method.

Good luck

fergyfresh · 2017-12-29T04:26:36Z

I admittedly thought it was a google account issue on my end and not the AWS integration. I changed my google password not too long ago and forgot it yesterday and locked my account. Just figured it was that. Oops.

fergyfresh · 2017-12-29T16:52:46Z

I think I figured it out and will reopen once I fully verify.

fergyfresh · 2018-01-18T19:38:43Z

@digiltd I'm going to revisit this. The fix to flask-ask that we need is a PR from 10/17 last year and the most recent update to PyPi was 8/17 for flask-ask. I'm going to revisit this when I get home for sanities sake and just use the master for https://github.com/johnwheeler/flask-ask and see where that gets me.

fergyfresh · 2018-01-19T04:38:35Z

So i swapped out the requirements.txt Flask-Ask with the master on github and I still have the same error, it processes the voice fine off the server but never streams the song.

[1516336514491] [INFO] 2018-01-19T04:35:14.491Z 23c6c4bb-fcd2-11e7-9258-992c0901da8c 72.21.217.105 - - [19/Jan/2018:04:35:14 +0000] "POST /alexa HTTP/1.1" 200 669 "" "Apache-HttpClient/4.5.x (Java/1.8.0_131)" 0/1291.081
[1516336518889] [INFO] 2018-01-19T04:35:18.889Z 252c6e5e-fcd2-11e7-9391-c7ed87680229 208.103.74.1 - - [19/Jan/2018:04:35:18 +0000] "GET /alexa/stream/Thsodn7cumeozy4ig6xpondmdju HTTP/1.1" 200 5822170 "" "Echo/1.0(APNG)" 0/3299.862
[1516336519305] [INFO] 2018-01-19T04:35:19.305Z 276065c4-fcd2-11e7-b0e7-71cb22415519 72.21.217.105 - - [19/Jan/2018:04:35:19 +0000] "POST /alexa HTTP/1.1" 400 0 "" "Apache-HttpClient/4.5.x (Java/1.8.0_131)" 0/56.832
[1516336519474] [INFO] 2018-01-19T04:35:19.474Z 277f1180-fcd2-11e7-8899-817c054ab0c9 72.21.217.105 - - [19/Jan/2018:04:35:19 +0000] "POST /alexa HTTP/1.1" 400 0 "" "Apache-HttpClient/4.5.x (Java/1.8.0_131)" 0/40.431

But it also installs 0.9.7 even when I do pip install -e git+git://github.com/johnwheeler/flask-ask/@master#egg=Flask-Ask. So I'm lost.

digiltd · 2018-01-19T18:43:13Z

Yup, exactly what I was seeing. Testing does have its benefits :)

The Skill query side of it is not a problem for a Lambda, but they are not really suited to staying up and running in order to stream content. Though there might be a way, never say never :)

fergyfresh · 2018-01-23T15:22:11Z

@digiltd it streams from google play though doesn't it? We just pass the link through via a callback. That's my understanding so far at least.

I have a breakthrough in why it might not work pip -e and pip freeze don't work that well with git repositories I guess, I'm still trying to digest some related issues I'm seeing on the pip repo, but here's one that might be causing our issues pypa/pip#2037 or pypa/pip#1735. I'm going to try to have an editable repo as that is the currently specified work around, besides getting https://github.com/johnwheeler/flask-ask to bump a version. The install when using pip -e of the flask-ask master installs 0.9.7 which is 4 commits behind fergyfresh/flask-ask@5119634.

fergyfresh · 2018-03-06T16:51:13Z

@digiltd hey man, I jumped back into this for fun.

I am going to try to do this the old fashioned way, since you did it with kodi-alexa. So the geemusic app works similarly to kodi, where it takes the skill invocation, gets info from a google play music/kodi music/movie URL and streams it to your alexa device.

What did you specify in the Lambda console for the Handler for your kodi-alexa app? It keeps saying the file geemusic.py doesn't exist and various other iterations that I have tried.

Things I have tried for Handler:

geemusic.lambda_handler
geemusic.__init__.lambdahandler
geemusic/__init__.lambdahandler

All of them give me the same issues. I don't know what to do anymore. I'm stuck again. Any references or ideas you could throw out would be great!

digiltd · 2018-03-07T09:10:51Z

Good luck, it is on my list of things to investigate.

Looking at my Lambda function, I have the handler set to lambda_function.lambda_handler

I renamed server.py to lambda_function.py and left the contents as they were, I put the extra lines mentioned in johnwheeler/flask-ask#197 in the geemusic/__init__.py file

from __future__ import absolute_import
from flask import Flask, json, render_template
from flask_ask import Ask, session, question, statement, audio, request, context
import logging

from .utils.music import GMusicWrapper
from .utils.music_queue import MusicQueue

app = Flask(__name__)
ask = Ask(app, '/alexa')
logging.getLogger("flask_ask").setLevel(logging.INFO)

api = GMusicWrapper.generate_api(logger=app.logger)
queue = MusicQueue(api)

def lambda_handler(event, _context):
  return ask.run_aws_lambda(event)

from . import intents
from . import controllers

Hope it helps

fergyfresh · 2018-05-09T11:56:03Z

I jumped back into this one.

[1525866367187] [INFO] 2018-05-09T11:46:07.187Z 8f0a1f52-537e-11e8-97ce-d914754c0202 208.103.74.1 - - [09/May/2018:11:46:07 +0000] "GET /alexa/stream/Tyrxm6cyldk2tgwb4vxqpucha4q HTTP/1.1" 200 6921403 "" "Echo/1.0(APNG)" 0/596.483
[1525866367310] body size is too long

You need the def lambda_handler, I have skeletoned out the zappa_settings.json file I used. Some of the requirements are different BytesIO in flask-ask core.py changed back to StringIO.

Now we have to figure out what we can do to get around the body size being too long. All the other requests seem to go through fine.

fergyfresh · 2018-05-09T11:58:12Z

Gonna read through this tonight https://seancoates.com/blogs/lambda-payload-size-workaround/

fergyfresh · 2018-05-09T14:02:25Z

@digiltd I saw that kanzi or kodi_voice (I know it's one of these) uses S3 buckets. Are any of these used for S3 bucket redirects for bypassing the 6MB request limit?

digiltd · 2018-05-09T15:03:24Z

I don't think so, it uses it to store a cached version of the json requested from Kodi. I don't think an S3 redirect will help. Same developer created a way to listen to your music stored in Kodi through an Echo device. I haven't tried setting it up, I know it uses a db on his server (though he has provided the source) to link the accounts. Could be a starting point though https://lexigr.am/docs/what-is-koko.html

fergyfresh · 2018-05-09T15:17:45Z

But the current error message I have is basically screaming S3 redirect. Currently its failing on the redirect with body size too large error. Everything smells like that blog I linked 3 hours ago. Thanks for the reference into their S3 usage though.

fergyfresh · 2018-05-09T17:01:19Z

I really think this is the solution because this is what I'm trying to fix and it seems like its basically a drop in replacement:
https://github.com/scoates/access-es/blob/8800d0cfb11d75c888a88680f93f1f0ef2fc6260/app.py#L193-L197

https://github.com/fergyfresh/geemusic/blob/master/geemusic/controllers.py#L23-L25

fergyfresh added 3 commits December 27, 2017 17:22

Added verbage for setting up AWS Lambda with Zappa.

b612ca9

Fixed missing quotation.

e323218

Fixed typos.

71d0659

fergyfresh mentioned this pull request Dec 28, 2017

AWS Lambda integration #166

Closed

fergyfresh closed this Dec 29, 2017

fergyfresh reopened this Mar 6, 2018

fergyfresh closed this Mar 6, 2018

fergyfresh mentioned this pull request May 10, 2018

Zappa deploy #214

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added verbage for setting up AWS Lambda with Zappa. #167

Added verbage for setting up AWS Lambda with Zappa. #167

fergyfresh commented Dec 28, 2017

digiltd commented Dec 28, 2017 •

edited

Loading

fergyfresh commented Dec 28, 2017 •

edited

Loading

digiltd commented Dec 28, 2017

fergyfresh commented Dec 28, 2017 •

edited

Loading

digiltd commented Dec 28, 2017

fergyfresh commented Dec 28, 2017

digiltd commented Dec 29, 2017

fergyfresh commented Dec 29, 2017

fergyfresh commented Dec 29, 2017

fergyfresh commented Jan 18, 2018 •

edited

Loading

fergyfresh commented Jan 19, 2018 •

edited

Loading

digiltd commented Jan 19, 2018

fergyfresh commented Jan 23, 2018 •

edited

Loading

fergyfresh commented Mar 6, 2018

digiltd commented Mar 7, 2018

fergyfresh commented May 9, 2018

fergyfresh commented May 9, 2018

fergyfresh commented May 9, 2018

digiltd commented May 9, 2018

fergyfresh commented May 9, 2018 •

edited

Loading

fergyfresh commented May 9, 2018

Added verbage for setting up AWS Lambda with Zappa. #167

Added verbage for setting up AWS Lambda with Zappa. #167

Conversation

fergyfresh commented Dec 28, 2017

digiltd commented Dec 28, 2017 • edited Loading

fergyfresh commented Dec 28, 2017 • edited Loading

digiltd commented Dec 28, 2017

fergyfresh commented Dec 28, 2017 • edited Loading

digiltd commented Dec 28, 2017

fergyfresh commented Dec 28, 2017

digiltd commented Dec 29, 2017

fergyfresh commented Dec 29, 2017

fergyfresh commented Dec 29, 2017

fergyfresh commented Jan 18, 2018 • edited Loading

fergyfresh commented Jan 19, 2018 • edited Loading

digiltd commented Jan 19, 2018

fergyfresh commented Jan 23, 2018 • edited Loading

fergyfresh commented Mar 6, 2018

digiltd commented Mar 7, 2018

fergyfresh commented May 9, 2018

fergyfresh commented May 9, 2018

fergyfresh commented May 9, 2018

digiltd commented May 9, 2018

fergyfresh commented May 9, 2018 • edited Loading

fergyfresh commented May 9, 2018

digiltd commented Dec 28, 2017 •

edited

Loading

fergyfresh commented Dec 28, 2017 •

edited

Loading

fergyfresh commented Dec 28, 2017 •

edited

Loading

fergyfresh commented Jan 18, 2018 •

edited

Loading

fergyfresh commented Jan 19, 2018 •

edited

Loading

fergyfresh commented Jan 23, 2018 •

edited

Loading

fergyfresh commented May 9, 2018 •

edited

Loading