Here’s why Amazon’s new Availability Zone is great for Australian AWS customers
15 Feb 2016
Reader’s Letters: What’s with GorillaStack?
04 Mar 2016
Does your organization run batch jobs? I think I’d be hard pressed to find one that didn’t. Is the infrastructure running these batch jobs always on and highly available? Chances are you are spending a lot of money for servers that aren’t always active.
AWS Lambda is Amazon Web Services’ serverless execution offering, where customers can upload code, specify an execution role and execute code periodically or in response to events. The benefits are clear – high availability out of the box, simplicity in deployment and configuration as well as dramatic cost savings, with the user only paying for compute time to the nearest 100 milliseconds, rather than every hour their servers are running.
While Lambda has been an incredibly hyped and well received product, there were some limitations that have prevented deeper and widespread adoption. Until 2015’s AWS re:Invent conference, functions could only be fired in response to events and not on a periodic basis. This is attractive in cases where users have servers executing batch jobs via cron, Control-M or some other scheduler. The servers hosting the batch jobs can theoretically be replaced with lambda functions.
However one limitation was blocking broader adoption for this purpose (in this writer’s humble opinion). Lambda functions could only interact with resources placed in the default public network. Common scenarios where databases, data warehouses, caches and other resources were placed within a Virtual Private Cloud (VPC), these resources were off limits to code executing in your Lambda function. Until Now.
This week, AWS made another announcement regarding Lambda that makes it more appropriate than ever to replace servers running batch jobs. Lambda can now access resources inside a VPC. This means that regardless of how you arrange your organization’s networking and security, AWS Lambda is able to interact with your resources.
At GorillaStack, we are obsessed with improving our customer’s cloud and driving cost savings. In such a spirit, we decided to help guide our current and future customers down the path of enlightenment vis-à-vis cost savings, by writing an open source CloudFormation template to showcase this new functionality in the hope that it be adopted by those looking to migrate their batch jobs to Lambda.
Our intention is to lay down the groundwork on which you can migrate your batch logic to Lambda. Fork our repository and start saving!
This CloudFormation template creates:
- A VPC
- A highly-available AWS Aurora cluster in this VPC (selected in honor of Aurora’s recent launch in Sydney/ap-southeast-2)
- An example lambda function configured to connect to the Aurora database and that takes a count of the distinct entries in one table and inserts this result into another table with that day’s date
While this is a basic example, it demonstrates a new potential common use case for Lambda for enterprise customers of AWS and outlines how one would go about utilizing it for the execution of their batch jobs.
In the creation of this template we discovered some shortcomings of AWS Lambda that could act as a barrier to broader adoption. We share this in the hope that it saves the reader time in discovery and encourages our friends at AWS to address some of our pain points! There are three key issues:
- AWS CloudWatch events, the mechanism by which lambda functions can be executed periodically cannot be created in a CloudFormation template, but can be performed through the CLI or one of the SDKs.
- The selection of an AWS CloudWatch event as an event source for a lambda function is not possible via CloudFormation or even the AWS CLI or SDKs and is only possible through the AWS console
- CloudFormation does not yet support the VPC configuration parameters for Lambda functions that were just released. I think that in cases that resources are already covered by CloudFormation, new configuration options should be available there as they are released in the console or CLI.
This greatly limits our capability to automate anything around the automation and continuous delivery of scheduled lambda functions. This was a disappointing discovery and we definitely hope that AWS improve the coverage of their CLI, SDKs and CloudFormation over their available resources and configurations.
In summary, while Lambda is still maturing, it is clear that the good folks at AWS are trying hard to deliver enhancements that make Lambda a robust execution paradigm for small business, rogue developers and the enterprise alike.
[UPDATE 11/4/16: AWS CloudFormation now supports VPC Configuration for lambda – read more here]
