Tech Talk: Scaling with AWS Lambda
A deep dive into serverless architecture and how we leveraged AWS Lambda to scale our processing pipeline.
I had the pleasure of giving a tech talk about my experience with AWS Lambda at a hackathon hosted by Tech Night, a local tech community. With over 75 software developers in attendance, I shared my experience on building production ready applications with AWS Lambda.
The Video
Using AWS Lambda in the Real World
AWS Lambda is a serverless compute service which offers autoscaling and pay-per-use pricing. However, to move from a single lambda function to a production-ready application, Lambda must be integrated into a much larger ecosystem including CloudWatch, SQS, EventBridge, and Route 53.
While Lambda is seemingly as simple as uploading code and calling it a day, the path from starting a project to production is paved with non-obvious design decisions.
Key Edge Cases That Require Your Attention
When building for scale, "out-of-the-box" settings often fall short. In my presentation, I dive into three specific challenges I solved when working with AWS Lambda:
- Managing Back Pressure: While Lambda scales instantly, your downstream consumers might not. Utilize Reserved Concurrency to create a managed back-pressure system, ensuring we don't overwhelm external APIs.
- Architecting Long-Term Retries: Lambda’s built-in retry logic is limited. For enterprise reliability, I recommend a custom loop using SQS to store failed events and "Retry Lambdas" to handle transient failures and re-invoke the original lambda for up to 24 hours.
- Surviving AWS Outages: Regional outages are rare but inevitable. I discuss how we use Route 53 Global Endpoints and EventBridge to build a failover strategy that keeps the system alive even if an entire AWS region goes dark.
Building the Framework
You can’t just "upload code" and call it a day. To make Lambda reliable for enterprise use, we built a robust framework around it using:
- Infrastructure as Code (Terraform): For repeatable, environment-based deployments.
- Health Monitoring: Leveraging CloudWatch dashboards and automated alarms.
- CI/CD Pipelines: Ensuring every push to production is backed by automated end-to-end testing.
Checkout the presentation for more details!
Event Details
- Date of the talk: February 21, 2025