Packaging for serverless: effortless? Doubtless!
A package-and-deploy pipeline is like plumbing: if it's working, you don't notice, but when it's not… you really do, and getting it working can be messy. We've tried several approaches for building Python artefacts to run on AWS Lambda, and finally settled on one that works well. I'll walk through how the Pants build system works in practice for getting our code running in production, and how it's improved upon other common practices we previously used.
See this talk and many more by getting your ticket to PyCon AU now!
I want a ticket!Across our system, we have dozens of serverless functions, most of which use PyPI dependencies. We previously used techniques like running pip install --target=... in a docker image, then copying in our source code: it mostly works, but… it sparks thumb-twiddling and pipeline-clogging more than it sparks joy.
Using Pants, we build our artefacts quickly, reliably, cross-platform, and without boilerplate! Our builds are quick, through caching, parallelism, and avoiding emulation. Dependency inference means each only includes exactly what it needs, without maintaining careful lists. We even benefit from these more generally, like testing and building non-serverless artefacts too.
Faster coding, faster builds, faster deploys, and faster cold-starts. It's like a working drain: Who doesn’t want that?
Code examples at https://github.com/huonw/serverless-effortless
Huon is a software engineer aiming to make others more effective. He's now writing and deploying Python extensively at ExoFlare, and has previously worked on dev tools and programming languages at Mozilla and Apple and machine learning research at CSIRO.