Sequence Multiple Actions With One Rule
05 Sep 2018
AWS Community Day 2018
12 Sep 2018
The serverless framework has really accelerated the development of APIs for new applications, particularly for mobile or web backends. It exposes existing systems via an API for integration. When combined with the AWS Lambda + API Gateway model for API development, it makes the value proposition easy for low-usage through its “pay only for the time your code runs” model.
Developers can get something running in just a few days, even in more ambitious cases to validate porting a monolithic application to Lambda. This is true for small applications, and most examples demonstrate smaller APIs well. However, more serious applications can have dozens or hundreds of APIs, which presents its own challenges. A bit of planning can prevent serious headaches down the line.
The dreaded CloudFormation 200 resources limit
Error -------------------------------------------------- The CloudFormation template is invalid: Template format error: Number of resources, 237, is greater than maximum allowed, 200
This problem will be familiar to anyone who has developed large applications on AWS using its native templating language, CloudFormation. The serverless framework uses CloudFormation underneath and offers no easy solution to this problem.
Each API endpoint can generate somewhere between 5-8 CloudFormation resources, which practically limits the number of APIs in a single serverless stack to somewhere around 24-39. The general solution to this problem is to split up your APIs over several stacks. As you will see, it’s smart to plan for this from the beginning of your project.
So what do I need to do?
Without using any special plugins, serverless provides a few helpers to make this task at least possible and its documentation suggests some plugins that might be useful. These rely on nested stacks, which are more advanced and which come with their own complications. I’m going to outline a method that uses multiple related stacks, but not nested stacks.
We will approach this by creating a base stack with our shared resources, and structuring it so that we can create dependent (child) stacks that contain our API implementations.
Then, we need to:
- plan our API paths
- declare our base stack
- implement shared resources like authorizers, IAM Roles and the REST API in the base stack
- manually declare base API paths
- exporting all the shared resources using CloudFormation Exports
- put together a template for the child stacks
I’m using an example of a to-do application project, structured as follows:
/serverless.yaml # Base stack /api # Dependent stacks /api/users # Sub-stack 1 - user management /api/users/serverless.yaml /api/posts # Sub-stack 2 Posts management /api/posts/serverless.yaml
Planning the API Paths
One of the most important things to consider is how the endpoint HTTP paths will be structured in your API. For example, say we are planning an API like the following:
/usersPOST: create user/users/meGET: get current user/users/mePUT: update current user settings/users/me/postsGET: list all the posts by the current user/posts/GET: list out or search the posts in the system/posts/POST: create a post/posts/{postId}GET: get a specific post
Our first plan for splitting would be to put all the /users* APIs in the users stack, and the /posts* APIs in the posts stack. Logically, however, the /users/me/posts API presents an issue because it really belongs in the /posts stack despite beginning with /users.
If we were to structure it like this, it also presents us with a practical issue. API Gateway requires a resource for each path element i.e. /users/{userId} requires two resources, users and {userId}, and then a resource for the method e.g. POST. These are hierarchical in that each one declares a parent resource.
serverless would generate something like the following for our /users/me PUT declaration.
Resources:
# /users (references the API Gateway as parent)
ApiGatewayResourceUsers:
Type: AWS::ApiGateway::Resource
Properties:
RestApiId: { Ref: "ApiGatewayRestApi" }
ParentId: { Fn::GetAtt: "ApiGatewayRestApi.RootResourceId" }
PathPart: users
# /me (references /users as parent)
ApiGatewayResourceUsersMe:
Type: AWS::ApiGateway::Resource
Properties:
RestApiId: { Ref: "ApiGatewayRestApi" }
ParentId: { Ref: "ApiGatewayResourceUsers" }
PathPart: "me"
# /users/me PUT (references /me as parent)
ApiGatewayResourceUsersMePosts:
Type: AWS::ApiGateway::Method
Properties:
RestApiId: { Ref: "ApiGatewayRestApi" }
ResourceId: { Ref: "ApiGatewayResourceUsersMe" }
HttpMethod: PUT
...When we declare our other endpoints that share the same path parts (e.g. /users GET or /users/me/posts GET), they will reference the same resources:
/usersGET :: referencesApiGatewayResourceUsers/users/me/postsGET :: referencesApiGatewayResourceUsersMe
If the endpoints are declared in different stacks, but we don’t reference the same parent, serverless will attempt to generate the path part resources twice (in this case ApiGatewayResourceUsers), and the second attempt will fail with a conflict. Only the AWS::ApiGateway::Method resources will be unique.
This means that we have to identify the path parts that are shared, declare and export them in our base stack, and then instruct serverless to use the shared ones in our child stacks.
Base Stack
Our base stack will contain all our common resources. These will then be exported using CloudFormation Outputs. You will note we have preferred to declare many of the resources directly using CloudFormation syntax – this is because the serverless way of generating them makes them hard to share around between stacks.
IAM Role
While you can create an IAM role per stack (where it makes sense) or even per Lambda, a shared role is easiest and only generates one resource.
We can use the generated serverless role and add our own statements with iamRoleStatements e.g.: provider:
name: aws
...
iamRoleStatements:
# The following two statements will need to be added if you are not declaring any http event handlers
# in your base stack
-
Effect: "Allow"
Action: [ logs:CreateLogStream ]
Resource:
Fn::Join:
- ""
- ["arn:aws:logs:", {Ref: "AWS::Region"}, {Ref: "AWS::AccountId"}, ":log-group:/aws/lambda/postsapi-*:*"]
-
Effect: Allow
Action: [ logs:PutLogEvents ]
Resource:
Fn::Join:
- ""
- ["arn:aws:logs:", {Ref: "AWS::Region"}, {Ref: "AWS::AccountId"}, ":log-group:/aws/lambda/postsapi-*:*:*"]
# other lambdas
-
Effect: Allow
Action: ["sqs:SendMessage", "sqs:SendMessageBatch"]
Resource:
Fn::GetAtt: MySQSQueue.Arn
-
Effect: Allow
Action: ["sns:Publish"]
Resource:
Fn::GetAtt: MySNSTopic.ArnThen, we export the ARN of the generated IamRoleLambdaExecution resource in the Outputs section (see below).
Note: The IAM Role is not automatically generated unless you specify at least one Lambda, such as the authorizer. In that case, you can declare the IAM role directly e.g.:
resources:
Resources:
IAMRoleLambdaExecution:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service: [ "lambda.amazonaws.com" ]
Action: [ "sts:AssumeRole" ]
Path: "/"
RoleName: { "Fn::Join": [ "-", [ "postsapi", "dev", "ap-southeast-2", "lambdaRole" ] ] }
Policies:
- PolicyName: "${opt:stage}-postsapi-lambda"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action: [ logs:CreateLogStream ]
Resource:
Fn::Join:
- ""
- ["arn:aws:logs:", {Ref: "AWS::Region"}, {Ref: "AWS::AccountId"}, ":log-group:/aws/lambda/postsapi-*:*"]
-
Effect: Allow
Action: [ logs:PutLogEvents ]
Resource:
Fn::Join:
- ""
- ["arn:aws:logs:", {Ref: "AWS::Region"}, {Ref: "AWS::AccountId"}, ":log-group:/aws/lambda/postsapi-*:*:*"]Authorizer
If you are implementing an authorizer, you should declare its Lambda e.g.:
functions:
...
generalAuthorizer:
handler: authorizer.handlerand then create an API Gateway Authorizer resource and associated lambda permission (so API Gateway may invoke it) using CloudFormation syntax:
resources:
Resources:
...
# Authorizer
# ==========
ApiGatewayAuthorizer:
Type: AWS::ApiGateway::Authorizer
Properties:
AuthorizerResultTtlInSeconds: 60
AuthorizerUri:
Fn::Join:
- ''
-
- 'arn:aws:apigateway:'
- Ref: "AWS::Region"
- ':lambda:path/2015-03-31/functions/'
- Fn::GetAtt: "GeneralAuthorizerLambdaFunction.Arn"
- "/invocations"
IdentitySource: method.request.header.Authorization
IdentityValidationExpression: "Bearer .+"
Name: api-${opt:stage}-authorizer
RestApiId: { Ref: ApiGatewayRestApi }
Type: TOKEN
ApiGatewayAuthorizerPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Fn::GetAtt: GeneralAuthorizerLambdaFunction.Arn
Action: lambda:InvokeFunction
Principal:
Fn::Join: ["",["apigateway.", { Ref: "AWS::URLSuffix"}]]Note that you need to:
- substitute the name of the function you declared with its generated CloudFormation name. These take the form
{FunctionName}LambdaFunction(where the first letter is capitalized). In our case, the function was calledgeneralAuthorizer, so the CloudFormation name will beGeneralAuthorizerLambdaFunction - set the other authorizer parameters, such as its
Type,IdentitySource, etc (see the documentation)
API Gateway
Unless you declare a function with a http event, serverless will no longer generate a RestApi CloudFormation resource. For this reason, you have two options:
- Create a dummy resource with a HTTP event (effectively creating a dead API on your gateway). This will generate a
AWS::ApiGateway::RestApiresource with the logical nameApiGatewayRestApias well as the associated IAM Role. - Declare your own. This is straightforward enough:
resources:
Resources:
...
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
Name: postsapi
Description: Posts API GatewayIn the child stacks, we can tell serverless to use our shared API Gateway and its root resource instead of creating a new one:
provider:
name: aws
...
apiGateway:
# we'll define the exports for these later
restApiId:
Fn::ImportValue: postsapi-${opt:stage}-RestApiId
restApiRootResourceId:
Fn::ImportValue: postsapi-${opt:stage}-RootResourceIdCreating and exporting the API Paths
Thankfully this is not too difficult:
- In your base stack, declare the path part resources in the
Resourcessection that will be shared between the stacks. In our case, this is the/usersand/mepath parts:
# Base Stack
resources:
Resources:
...
ApiGatewayResourceUsers:
Type: AWS::ApiGateway::Resource
Properties:
RestApiId: { Ref: "ApiGatewayRestApi" }
ParentId: { Fn::GetAtt: "ApiGatewayRestApi.RootResourceId" }
PathPart: users
ApiGatewayResourceUsersMe:
Type: AWS::ApiGateway::Resource
Properties:
RestApiId: { Ref: "ApiGatewayRestApi" }
ParentId: { Ref: "ApiGatewayResourceUsers" }
PathPart: "me"- Export them as outputs (we’ll see how to do this in a later section)
- Tell serverless to use these APIs in our child stacks:
# Child Stack
provider:
name: aws
...
apiGateway:
...
restApiResources:
/users:
Fn::ImportValue: postsapi-${opt:stage}-ApiGatewayResourceUsers
/users/me:
Fn::ImportValue: postsapi-${opt:stage}-ApiGatewayResourceUsersMeExporting our shared resources
The shared resources can then be exported from the CloudFormation Outputs section. I chose to put the stage name (using the ${opt:stage} property) in the export names to avoid conflicts if the same stacks are deployed twice in the same account and region (e.g. for full deployment of development branches).
resources:
...
Outputs:
# RestApi resource ID (e.g. ei829oe)
RestApiId:
Value:
Ref: ApiGatewayRestApi
Export:
Name: postsapi-${opt:stage}-RestApiId
# RestApi Root Resource (the implicit '/' path)
RootResourceId:
Value:
Fn::GetAtt: ApiGatewayRestApi.RootResourceId
Export:
Name: postsapi-${opt:stage}-RootResourceId
# The IAM Role for Lambda execution
IamRoleLambdaExecution:
Value:
Fn::GetAtt: IamRoleLambdaExecution.Arn
Export:
Name: postsapi-${opt:stage}-IamRoleLambdaExecution
# The Authorizer (only if you are using an API Gateway authorizer)
ApiGatewayAuthorizerId:
Value:
Ref: ApiGatewayAuthorizer
Export:
Name: postsapi-${opt:stage}-ApiGatewayAuthorizerId
# Path Resources
ApiGatewayResourceUsers:
Value:
Ref: ApiGatewayResourceUsers
Export:
Name: postsapi-${opt:stage}-ApiGatewayResourceUsers
ApiGatewayResourceUsersMe:
Value:
Ref: ApiGatewayResourceUsersMe
Export:
Name: postsapi-${opt:stage}-ApiGatewayResourceUsersMeChild Stacks
Once we have a solid base stack, we can start defining our child stacks. The main parts are referencing the API Gateway, Root and Path resources and IAM Role in the provider section, and then referencing the Authorizer in each lambda HTTP event.
I’ve given an abbreviated example below of both the global values and some functions.
# /api/users/serverless.yaml
name: postapi-users
provider:
name: aws
runtime: nodejs8.10
memorySize: 1024MB
timeout: 10
role:
Fn::ImportValue: postsapi-${opt:stage}-IamRoleLambdaExecution
apiGateway:
restApiId:
Fn::ImportValue: postsapi-${opt:stage}-RestApiId
restApiRootResourceId:
Fn::ImportValue: postsapi-${opt:stage}-RootResourceId
restApiResources:
users:
Fn::ImportValue: postsapi-${opt:stage}-ApiGatewayResourceUsers
users/me:
Fn::ImportValue: postsapi-${opt:stage}-ApiGatewayResourceUsersMe
functions:
usersGet:
handler: usersGet.handler
events:
- http:
method: GET
# You still provide the full paths, as serverless will figure
# out from the `provider.apiGateway.restApiResources` section
# whether to generate or reference the parent.
path: /users
integration: lambda-proxy
authorizer:
type: CUSTOM
authorizerId:
Fn::ImportValue: postsapi-${opt:stage}-ApiGatewayAuthorizerId
usersMe:
handler: usersMeGet.handler
events:
- http:
method: GET
# You still provide the full paths, as serverless will figure
# out from the `provider.apiGateway.restApiResources` section
# whether to generate or reference the parent.
path: /users/me
integration: lambda-proxy
authorizer:
type: CUSTOM
authorizerId:
Fn::ImportValue: postsapi-${opt:stage}-ApiGatewayAuthorizerIdYou can simplify the above a bit further by declaring the authorizer section as a custom variable and referencing it e.g.:
custom:
authorizer:
type: CUSTOM
authorizerId:
Fn::ImportValue: postsapi-${opt:stage}-ApiGatewayAuthorizerId
# in the lambda event
events:
- http:
method: GET
...
authorizer: ${self:custom.authorizer}Example
All of the above code, including working functions using DynamoDB, can be found in GorillaStack’s repository on GitHub.
Instructions for deploying it can be found in the project README.md file.
1 Nested stacks work by deploying a parent stack that contains parameters passed to your child stacks. The main difficulty in deploying nested stacks is when something goes wrong – CloudFormation will roll back all the changes in a particular change set, which is quite time consuming during development testing (and especially if you have a lot of stacks before the error that have changes).
