Can no longer run IfcOpenShell on AWS Lambda

[MichaelBluetooth]

I previously was using python-ifcopenshell in my AWS Lambda function to perform some IFC file parsing. This package has been removed from PyPI and my builds fail because pip can no longer find the package. I tried upgrading to ifcopenshell, but I get the following error when my Lambda runs.

IfcOpenShell not built for 'linux/64bit/python3.9'

This is the powershell script I run to build my little deploy package, which would then be deployed to AWS. The pip install section comes from this aws knowledge center article. The lamba ran fine when python-ifcopenshell was available in PyPI, but no longer runs with ifcopenshell.

pip install --no-cache-dir --platform manylinux2014_x86_64 --target=./ --implementation cp --python 3.9 --only-binary=:all: --upgrade ifcopenshell
pip install --no-cache-dir --platform manylinux2014_x86_64 --target=./ --implementation cp --python 3.9 --only-binary=:all: --upgrade requests

$ZipFileResult="./ifc-parser.zip"
$DirToExclude=@(".git", "venv")

Get-ChildItem "./" -Directory  | 
           where { $_.Name -notin $DirToExclude} | 
              Compress-Archive -DestinationPath $ZipFileResult -Update
			  
Compress-Archive -Path "./*.py" -DestinationPath $ZipFileResult -Update
Compress-Archive -Path "./requirements.txt" -DestinationPath $ZipFileResult -Update

Is there some way I can install this package in a way that allows it to run in my Lambda? I found #629 which reports the same error, but there isn't much detail there on how the issue was resolved or why it no longer runs.

[Moult]

Is --platform manylinux2014_x86_64 necessary? Can you try remove it and have it autodetect (manylinux1_x86_64)?

In addition, is --implementation cp necessary? Try remove that too.

[MichaelBluetooth]

I have --platform manylinux2014_x86_64 and --implementation cp because this aws knowledge center article told me to.

Is --platform manylinux2014_x86_64 necessary?
I believe this is required because I am building on Windows. If I omit this it would grab the wrong version.

I tried the following three combinations, but they all yielded the same error.

1. Removing --implementation cp
2. Changing --platform manylinux2014_x86_64 to --platform manylinux1_x86_64
3. Removing --implementation cp and changing --platform manylinux2014_x86_64 to --platform manylinux1_x86_64

[Moult]

What happens if you run pip install --no-cache-dir --platform manylinux2014_x86_64 --target=./ --implementation cp --python 3.9 --only-binary=:all: --upgrade ifcopenshell? On my machine it successfully downloads and installs ifcopenshell.

If it does, then that means that the problem is with the build itself, i.e. the ifcopenbot builds (which are simply bundled again for pypi) are incompatible with AWS Lambda. You probably didn't run into this in the past because the old PyPI (python-ifcopenshell) package was probably built in a different way. You can test this by manually grabbing the appropriate file from here https://pypi.org/project/ifcopenshell/#files or from https://blenderbim.org/docs-python/ifcopenshell-python/installation.html#pre-built-packages

[MichaelBluetooth]

What happens if you run....

It successfully downloads and installs for me on my Windows 10 machine.

You can test this by manually grabbing the appropriate file from here...

I had previously tried to update my little script download the pre-built package and just unzip it to where pip would have put it. Unfortunately it didn't work and I get a different error. I suspect because it's not "installed" in the same way pip installs the package. I would have to put the contents directly in the python path dir, which I don't think I can do with Lambda.

AttributeError: module 'ifcopenshell' has no attribute 'open'

Here's how I had altered my script to download and unzip the contents:

pip install --no-cache-dir --platform manylinux2014_x86_64 --target=./ --implementation cp --python 3.9 --only-binary=:all: --upgrade requests

Invoke-WebRequest "https://s3.amazonaws.com/ifcopenshell-builds/ifcopenshell-python-39-v0.7.0-6180d73-linux64.zip" -OutFile ifcopenshell.zip
Expand-Archive ./ifcopenshell.zip ./ifcopenshell -Force
Remove-Item ./ifcopenshell.zip

$ZipFileResult="./ifc-parser.zip"
$DirToExclude=@(".git", "venv")

Get-ChildItem "./" -Directory  | 
           where { $_.Name -notin $DirToExclude} | 
              Compress-Archive -DestinationPath $ZipFileResult -Update
			  
Compress-Archive -Path "./*.py" -DestinationPath $ZipFileResult -Update
Compress-Archive -Path "./requirements.txt" -DestinationPath $ZipFileResult -Update

If it does, then that means that the problem is with the build itself, i.e. the ifcopenbot builds (which are simply bundled again for pypi) are incompatible with AWS Lambda
Do you have any recommendations on what I can try to remedy this?

The person who opened #629 was getting the same error - though they were running on AWS Lambda under a different setup. Do you think I'd be able to spin up a Docker container and run it there? I'd expect that the AWS base python image would have the same problem though. Just trying to think of ideas on how to work around the problem to get up and running.

[Moult]

Yeah, so potentially the IfcOpenBot build isn't happy on your platform.

I would personally rule out whether there is a fundamental incompatibility between AWS Lambda's Python / env and how IfcOpenBot builds things by packaging a zip (https://stackoverflow.com/questions/64161352/how-to-work-with-python-custom-packages-in-aws-lambda) with all the deps bundled. If that works, then it's not necessary to go the docker route.

Note there is a docker setup here: https://blenderbim.org/docs-python/ifcopenshell-python/installation.html#docker

I would have to put the contents directly in the python path dir

In your zip, can't you just have ifcopenshell as a subdir?

[aothms]

IfcOpenShell not built for 'linux/64bit/python3.9'

There's likely more information there in the stacktrace.

It's probably related to glibc version.

manylinux2014 is centos 7, which is glibc 2.17

https://distrowatch.com/table.php?distribution=centos

we build on ubuntu focal, which is glibc 2.31

https://distrowatch.com/table.php?distribution=ubuntu

The main issue though is that centos 7 comes with gcc (the compiler) 4.8 (originally released 2013, patches from later years) I doubt that will successfully compile the code base and its dependencies.

Is there are later Amazon Linux base image you can use? I think I recall others reporting something like that.

Otherwise what you can do is not install with pip, but extract the 0.6 ifcopenbot build in your site-packages "manually" IfcOpenBot@721fe47#commitcomment-56675315 0.6 is compiled on an older ubuntu distro.

[MichaelBluetooth]

There's likely more information there in the stacktrace.

There's actually not too much. This is directly copied from the CloudWatch logs. The lambda_function in this error is just my python script that calls IfcOpenShell to open a file.

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': IfcOpenShell not built for 'linux/64bit/python3.9'

Is there are later Amazon Linux base image you can use? I think I recall others reporting something like that.

I don't think I get an option to choose here. When you create a Lambda you just choose a runtime like "Python 3.9" or NodeJS 18.x" and an architecture - x86_64 or arm64. According to the AWS Lambda runtime docs the Lambda runs in a specific execution environment with the operating system being "Amazon Linux 2."

In the meantime, I'm going to pursue trying to create a "layer" and use the v0.6 build to see if it resolves my problem.

[aothms]

There's actually not too much. This is directly copied from the CloudWatch logs. The lambda_function in this error is just my python script that calls IfcOpenShell to open a file.

Oh that's unfortunate so the exception traceback is not printed only str(exception) which isn't ultimately as meaningful in this case.

It seems they are working on AL2022 https://aws.amazon.com/linux/amazon-linux-2022/faqs/ but I also don't seem to find any resources on using this as a lambda base image.

I don't know how much time you're willing to invest into this. If we can get the build script to succeed on Amazon Linux 2 that's basically all we need. I don't mind setting up another build worker for AL2 once we have it successfully compiling. So feel free to experiment with this. It's just running a single python script, but it takes a while to complete 5m - 2h depending on hardware.

[Moult]

Bump?

[MichaelBluetooth]

So I was able to get this to work with existing builds. Instead of bundling the lambda code along with the dependencies (in my case IfcOpenShell and Requests), I extracted my 3rd party dependencies into layers. The two big keys to success was to make sure the package are installed in python/lib/python3.9/site-packages and to use v0.6 as aothms mentioned in their comment.

Here's my powershell script I run to bundle everything up. It created a zip with three zips inside it.

• ifc-parser.zip - the actual lambda code
• requests.zip - all the stuff for requests
• ifcopenshell.zip - ifcopenshell

Then I just take the requests.zip and ifcopenshell.zip and created two layers for my lambda.

# Install stuff required for the "requests" library into a zip to be used as a layer
pip install --no-cache-dir --platform manylinux2014_x86_64 --target=./python/lib/python3.9/site-packages --implementation cp --python 3.9 --only-binary=:all: --upgrade requests
Compress-Archive "./python" -DestinationPath "requests.zip"
Remove-Item ./python -Force -Recurse


# Install stuff required for the "IfcOpenShell" library into a zip to be used as a layer
# pip install --no-cache-dir --platform manylinux2014_x86_64 --target=./python/lib/python3.9/site-packages --implementation cp --python 3.9 --only-binary=:all: --upgrade ifcopenshell
# Invoke-WebRequest "https://s3.amazonaws.com/ifcopenshell-builds/ifcopenshell-python-39-v0.7.0-6180d73-linux64.zip" -OutFile ifcopenshell.zip
Invoke-WebRequest "https://s3.amazonaws.com/ifcopenshell-builds/ifcopenshell-python-39-v0.6.0-721fe47-linux64.zip" -OutFile ifcopenshell.zip
Expand-Archive ./ifcopenshell.zip ./python/lib/python3.9/site-packages -Force
Remove-Item ./ifcopenshell.zip
Compress-Archive "./python" -DestinationPath "ifcopenshell.zip"
Remove-Item ./python -Force -Recurse


$ZipFileResult="./ifc-parser.zip"
$DirToExclude=@(".git", "venv", ".github", "requests.zip", "ifcopenshell.zip")

Get-ChildItem "./" -Directory  | 
           where { $_.Name -notin $DirToExclude} | 
              Compress-Archive -DestinationPath $ZipFileResult -Update
			  
Compress-Archive -Path "./*.py" -DestinationPath $ZipFileResult -Update
Compress-Archive -Path "./requirements.txt" -DestinationPath $ZipFileResult -Update

Compress-Archive "ifc-parser.zip" -DestinationPath "ifc-parser-pkg.zip"
Compress-Archive "ifcopenshell.zip" -DestinationPath "ifc-parser-pkg.zip" -Update
Compress-Archive "requests.zip" -DestinationPath "ifc-parser-pkg.zip" -Update

[Moult]

The key bit here is that you're reverting to v0.6.0 which is missing a lot of goodies from v0.7.0. Would it be possible to try get it working with Docker?

[MichaelBluetooth]

I will try to get it working with Docker, but it will likely take me some time. I'm still learning Docker and Lambda.

[Moult]

There is now a guide here: https://blenderbim.org/docs-python/ifcopenshell-python/installation.html#aws-lambda