Thumbor 6.5.2 #45

leviwilson · 2018-09-03T13:44:24Z

Originally I'd opened up #44 as I was having troubles being able to successfully build the lambda package from source. Some of the issues that I encountered are documented in there.

Description

This pull request bumps the thumbor dependency from 6.4.2 ➡️ 6.5.2. The version jump additionally fixes #43 in that it is now able to work with CCITT Fax4 TIFF images.

Additionally, in light of the issues I encountered with being able to successfully build the lambda function I dockerized the build so that it is repeatable.

In addition to the base dependencies that were originally in the README, I had to also pull in other dependencies into the docker image (pngcrust, gifsicle, pngquant, etc.) so that the lambda function wouldn't complain in the CloudWatch logs. Feedback welcome!

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

thumbor-community/aws#100

https://hub.docker.com/r/naftulikay/circleci-lambda-rust/builds/b2quu7btj8md9zlbf36fd59/ gave me a little bit of a hint in which image to lock. Just need to find which version of it uses 7.51.0. This is the error I get now: pycurl: libcurl link-time version (7.51.0) is older than compile-time version (7.53.1) Where 7.53.1 is the same as what curl --version shows.

Noticed in the build output that it was failing to locate some of the imaging libraries that it was trying to yum install. Realized that on amazonlinux 2 that the command to enable epel is slightly different.

This reverts commit c1bb993.

…t.sh

…d-s3-dist.sh

pip.req was removed in pip >= 10

leviwilson · 2018-09-03T13:46:32Z

Dockerfile

+FROM amazonlinux:2017.03.1.20170812
+
+# lock yum to the same repository version
+RUN sed -i 's/releasever=.*/releasever=2017.03/g' /etc/yum.conf


So that the build is repeatable, I locked the releasever to 2017.03 vs. latest. In particular, the issue was when building the pycurl library from source since that when you yum install libcurl-devel if it pulls latest, when the lambda function runs you get an error that the linked version of libcurl differs from the compiled version of libcurl.

To get around that, I just pinned the yum repository to that version.

leviwilson · 2018-09-03T13:47:43Z

Dockerfile

+
+# pycurl
+RUN yum install -y nss-devel
+ENV PYCURL_SSL_LIBRARY=nss


pycurl is built from source. I also has to specify nss as the SSL library. Originally when I tried openssl, when it ran in lambda it complained that the compiled version used openssl but the runtime used nss.

leviwilson · 2018-09-03T13:51:17Z

Dockerfile

+RUN wget http://dl.fedoraproject.org/pub/epel/6/x86_64/Packages/l/libimagequant-2.5.2-5.el6.x86_64.rpm && \
+    yum localinstall libimagequant-2.5.2-5.el6.x86_64.rpm -y && rm libimagequant*rpm && \
+    wget http://dl.fedoraproject.org/pub/epel/6/x86_64/Packages/p/pngquant-2.5.2-5.el6.x86_64.rpm && \
+    yum localinstall pngquant-2.5.2-5.el6.x86_64.rpm -y && rm pngquant*rpm


I was not able to find optipng or pngquant in yum search. In order to get around errors in including those when running the build script, I grabbed them directly from here. Now when I run it doesn't complain in the build script at the end.

zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip pngquant zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip jpegtran zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip optipng zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip pngcrush zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip gifsicle zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip mozjpeg zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip imgmin

When I didn't have these dependencies, the output would say something to the effect of nothing to do here (when it couldn't find the dependencies).

leviwilson · 2018-09-03T13:52:42Z

Dockerfile

+WORKDIR /lambda/deployment
+
+ENTRYPOINT ["./build-s3-dist.sh"]
+CMD ["source-bucket-base-name"]


This defaults the bucket to source-bucket-base-name; passing any other value when you docker run will override this.

leviwilson · 2018-09-03T13:53:16Z

source/image-handler-custom-resource/setup.py

@@ -1,7 +1,7 @@
 # coding: utf-8

 from setuptools import setup, find_packages
-from pip.req import parse_requirements
+from pip._internal.req import parse_requirements


pip >= 10 does not expose pip.req and instead it was moved to _internal.

leviwilson · 2018-09-03T13:54:07Z

source/image-handler/lambda_function.py

@@ -213,7 +213,7 @@ def request_thumbor(original_request, session):

 def process_thumbor_responde(thumbor_response, vary):
     if thumbor_response.status_code != 200:
-         return response_formater(status_code=response.status_code)
+         return response_formater(status_code=thumbor_response.status_code)


I believe using response here was a mistake; at one point when running my lambda function when I had an issue with tc_aws it hit this code path and blew up saying that response wasn't a global.

I believe thumbor_response is what was intended here.

Looks like this was fixed in #34 as well

leviwilson · 2018-09-03T13:54:44Z

source/image-handler/setup.py

-        'botocore==1.3.7',
-        'tornado_botocore==1.0.2',
+        'botocore==1.8',
+        'tornado_botocore==1.3.2',


botocore, tornado_botocore as well as tc_aws needed to be bumped to get around the pip >= 10 issue

leviwilson · 2018-09-07T20:51:38Z

@stevemorad did you get a chance to look over this at all?

simllll · 2019-01-03T22:58:31Z

Any news on this?

leviwilson · 2019-01-03T23:07:07Z

@simllll I've not heard anything back from anyone whether this is a good idea or not.

I did this PR before there was new / recent activity in the repo. I realize there's conflicts here but hadn't addressed them because I was waiting to hear back on some feedback.

Particularly wanted to get some feedback on the docker setup as I struggled a lot with getting it able to build a working lambda function so I wanted to document it in the Dockerfile.

- [Lambda supported versions](https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html) - [Docker repo](https://hub.docker.com/_/amazonlinux/) Also modified the docker file to fix some libraries. A lot of help from aws-solutions#45

timkelty · 2019-05-01T11:31:48Z

Can we please move on this, or something like it?

Movement on any patches has been very slow, so it would great if it were easier for people to build their own. Dockerizing the dependencies helps immensely.

/cc @hyandell @hayesry

shsenior · 2019-06-13T16:23:50Z

Resolved with v4.0.0

leviwilson added 24 commits September 1, 2018 10:27

add .gitignore

eba5093

add Dockerfile for building thumbor

36d4640

get around pip.req issue with pip upgrade

b750ff4

include make in the Dockerfile

daa9d3b

bump thumbor to 6.5.2

0fd15e5

bump tc_aws to fix Botocore import error

ffcdbec

thumbor-community/aws#100

lock the yum repo down as well

d14a347

bump botocore and tornado_botocore to latest

c1bb993

enable epel on amazonlinux 2

a2e9c4c

Noticed in the build output that it was failing to locate some of the imaging libraries that it was trying to yum install. Realized that on amazonlinux 2 that the command to enable epel is slightly different.

compile nss instead of openssl

79b7521

add ImageMagick

36c7d3f

move ImageMagick up a little

7d51222

response ➡️ thumbor_response

58882ef

Revert "bump botocore and tornado_botocore to latest"

4e23e8a

This reverts commit c1bb993.

add pngcrush, libjpeg* and gifsicle

3387878

add wget so mozjpeg can be added

58948c6

install optipng and pngquant to the docker image

9800e89

add an ENTRYPOINT / CMD to the Dockerfile to perform the build-s3-dis…

110d6a0

…t.sh

remove .vimrc from the cache

844a73a

update the README.md to reflect leveraging docker to perform the buil…

33e08e8

…d-s3-dist.sh

remove the MAINTAINER line

3627e5a

specify pip._internal.req vs. pip.req

6bbc037

pip.req was removed in pip >= 10

ignore deployment/dist not deployment

0462412

leviwilson commented Sep 3, 2018

View reviewed changes

This was referenced Oct 29, 2018

How to deploy the customized (forked) code? #66

Closed

Add ability to build using docker #68

Closed

leviwilson mentioned this pull request Jan 10, 2019

Requirements for building the lambda function #44

Closed

leviwilson mentioned this pull request Feb 12, 2019

Optimize JPEGs #10

Closed

shsenior closed this Jun 13, 2019

leviwilson mentioned this pull request Jul 5, 2019

outputFormat option not honored #116

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Thumbor 6.5.2 #45

Thumbor 6.5.2 #45

leviwilson commented Sep 3, 2018

leviwilson Sep 3, 2018

leviwilson Sep 3, 2018

leviwilson Sep 3, 2018

leviwilson Sep 3, 2018

leviwilson Sep 3, 2018

leviwilson Sep 3, 2018

leviwilson Sep 3, 2018

leviwilson Sep 3, 2018

leviwilson commented Sep 7, 2018

simllll commented Jan 3, 2019

leviwilson commented Jan 3, 2019

timkelty commented May 1, 2019

shsenior commented Jun 13, 2019

Thumbor 6.5.2 #45

Thumbor 6.5.2 #45

Conversation

leviwilson commented Sep 3, 2018

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leviwilson commented Sep 7, 2018

simllll commented Jan 3, 2019

leviwilson commented Jan 3, 2019

timkelty commented May 1, 2019

shsenior commented Jun 13, 2019