If laziness really is one of the virtues of a great programmer, then AWS Lambda is like a developer’s holy grail. With Lambda, you don’t have to spin up any servers, or configure or patch them, or pay for them to run hour after hour and week after week. You don’t even have to write much code.
AWS Lambda is about getting the job done with least effort—and little cost. You write simple functions and wire them to a request or an event, and Lambda executes those functions whenever that request or event occurs. You get charged only when your code actually runs.
Naturally, Lambda can tap into many other AWS services (S3, DynamoDB, Alexa, etc.), and you can draw on events in those services to trigger your functions. Lambda is an ideal way to experiment with the Amazon cloud and, perhaps, build something that’s exactly what everyone needs. Here’s how to get started.
Step 1: Create an AWS account
The first place to begin is by creating an AWS account. You probably have one because you can use the same Amazon account you use to buy soap or books. Once you have this, you can start renting computers and deploying anything you like.
Step 2: Design the architecture
It is tempting to jump to writing code immediately using any of Amazon’s examples as a starting point. Resist this temptation. Much of the real work will be imagining the way that data is created, analyzed, and transferred within your application. Sketching out this vision is a much bigger job than actually writing code.
Step 3: Choose an Amazon data store
Almost all applications require some kind of data storage. Once you determine how the data will flow, you can start to choose just how to store the data itself. This isn’t an easy decision because Amazon offers so many different options. If you’ve just got basic blocks of bits, then S3 storage is probably sufficient. If your data is more structured, then DynamoDB, Aurora, and other Amazon database services offer sophisticated options for organizing and analyzing the data.
Your choice will depend upon the structure of the data and the code you may already have for analyzing it. Aurora, for instance, is an SQL-capable database that operates like MySQL or PostgreSQL. You can take SQL code and apply it directly to take advantage of the sophisticated queries and tabular structure that already exist.
Step 4: Make a list of events
Lambda functions are triggered by events like the arrival of a new file or a user visiting a webpage. The list of these events is one of the best ways to understand and define a Lambda application. It illustrates just what the application will do to data and when it will do it.
There are dozens of possible events that you might choose. The simplest are just calls to a web address, but almost all of Amazon’s many services can generate Lambda events. I like the idea of waiting for events tied to data being stored in an S3 bucket. It’s easy to imagine a pipeline of S3 buckets joined by data processing functions that are triggered when the data arrives at the respective bucket.
Step 5: Consider additional AWS services
AWS offers a staggering number of services and some of them might do exactly what you need. It makes sense to pause and investigate whether you can get away without writing code by using an API run by Amazon instead. The less work you do yourself, the faster you get finished.
There are services for machine learning, analytics, security, game development, and a few other areas as well. In many cases, there are several different options. Spend some time poking around because you might be surprised.
Think of Lambda as a shell script for linking together all of the options under the umbrella of AWS. You might just be able to replace several different stops along the pipeline with an Amazon service.
Step 6: Choose a programming language
Okay, it’s finally time to think about coding. Lambda supports Java, JavaScript, C#, and Python directly. If you really must use some other code in another language, there’s a good chance that it can be emulated by JavaScript or Java. (See here or here to begin.)
After you choose a language, you should download the development tools and study the examples. It’s best not to think of Lambda as more of the same kind of development you’ve done before. In other words, it’s not another Java stack or Python program. You’ll really be writing just one function that responds to a very narrow range of parameters. You won’t have the same freedom as you do with another platform. The Lambda runtime will expect your code to accept a small packet of data, make a decision, and then move on.
Step 7: Set up a development environment
Choose the right environment for your language of choice. There is plenty of overlap, but Amazon has tried to meet developers halfway by extending the popular tools and models.
It’s worth noting that Amazon provides a pretty sophisticated web-based editor for your functions. You don’t need to do that much locally. I was generally able to make do with the online tool. There were some limitations and some places where it wasn’t as nice as a native editor, but it showed that you could pretty much do anything with a browser and an Internet connection.
Step 8: Write a function
Just one function. Really. That’s the whole gimmick. Okay, you can have this function call other functions, including many libraries, so it’s not that much different from writing a big application.
I think the best model is to write many small Lambda functions linked together with events. You may have a different idea. There are some steps in the data pipeline that might require plenty of computation and sophisticated modeling. Those might bundle lots of code into one Lambda invocation. But for the most part I think the platform lends itself to writing a chain of small functions that store their intermediate work in S3 buckets.
Step 9: Deploy the “app”
There’s a good chance your Lambda app is already deployed, in some sense, especially if you’re using the cloud development tools. Amazon offers several levels of staging and production so that you can set up a good development and testing workflow.
Step 10: Be mindful of limits
When you write software for your computer, your software is the boss. It controls the machine. Your code for Lambda is more of a guest who runs at the pleasure of Lambda. Your function should know its place and stay within the proper etiquette or AWS will simply terminate it.
There are a number of limits that will probably be more than adequate for most of the simple work that fits well in the Lambda model. Your function, for instance, gets only 300 seconds to answer or it’s shut down. It can only use 3008mb of memory, 1024 file descriptors and 1024 threads. Most of these won’t apply to you, but it’s best to study the limits so you can avoid making assumptions. Yes, Lambda will scale your application to accommodate demand but it won’t scale it in all possible ways.
Step 11: Debug the app
The simplest debugging can be done in your development environment on your local machine, be Visual Studio, Eclipse, or maybe even the command line. Many of the easiest bugs can be tackled here.
More elaborate debugging can make use of the logging functions built into Lambda. These logs are one of the simpler but most effective ways to track the flow of data and the way your code is reacting to events.
Step 12: Keep an eye on costs
The Lambda pricing model is very generous and forgiving, but at some point it does begin to cost real money. The first million requests and the first 400,000 GB-seconds of compute time are free. Then you pay tiny fractions of a cent when you exceed these limits. (A “GB-second” is a composite metric that measures the amount of RAM that you use and the time it takes for the function to finish.)
It’s difficult to anticipate just how much your application will cost in the long run. Just as one of the standard laws about programming says that premature optimization is a waste of time, the same is true about fretting about the costs of execution. When your code is up and running, you can test its behavior and then look for ways to reduce costs.
Much can be done by making simple architectural decisions about how often you generate events. If you’re, say, creating a network of sensors that phone in results to the AWS cloud, you can have them call in every second or every hour. While it’s tempting to offer near real time feedback and awesome service, it’s important to remember that the first option uses 3600 times as many invocations as the second. Think about the tradeoffs. Consider the needs. Measure the computation. Then make the right decisions to keep costs where the market will bear them.
Sit back and relax
Once your code is deployed and running, it’s time to relax, at least a bit. While many web applications require some monitoring and housekeeping, Lambda automates many of the biggest chores like watching for overburdened servers or full disks.
Still, you should watch behavior to see that the code is running cleanly and effectively. Once your application is running for a few weeks, you can begin to get a stable estimate of the cost of running it. If the costs are manageable, you can relax. If not, now is the time to look for ways to simplify the work your code does so the bills will be lower.