|7 min read

re:Invent 2018: Lambda Layers, Transit Gateway, and the Services That Matter

A practitioner's take on the AWS re:Invent 2018 announcements that will actually change how we build

cloud

I just returned from AWS re:Invent 2018 in Las Vegas. Five days, 50,000 attendees, and a firehose of service announcements that will take months to fully digest. AWS announced over 100 new services and features, which is both impressive and exhausting.

Not all announcements are created equal. Some are genuinely transformative. Some are incremental improvements. Some are solutions looking for problems. After a week of keynotes, breakout sessions, and hallway conversations, here are the announcements that I believe will have the most impact on how we architect systems at a major entertainment company.

AWS Transit Gateway

This is the announcement I am most excited about, and it got surprisingly little attention relative to the sexier Lambda and machine learning announcements.

Transit Gateway is a regional network hub that lets you connect multiple VPCs, VPN connections, and AWS Direct Connect gateways through a single managed service. Before Transit Gateway, connecting N VPCs required N*(N-1)/2 peering connections, each managed individually. At 10 VPCs, that is 45 peering connections. At 50 VPCs, it is 1,225.

We operate roughly 40 VPCs across our AWS accounts. Our peering mesh was a maintenance nightmare. Adding a new VPC required creating peering connections to every other VPC that needed connectivity, updating route tables on both sides of each peering, and hoping that nobody had a CIDR conflict. The process typically took two days and involved multiple teams.

Transit Gateway reduces this to a hub-and-spoke model. Each VPC attaches to the Transit Gateway. The Transit Gateway manages routing between them. Adding a new VPC means creating one attachment and updating one route table.

        VPC-A ─────┐
        VPC-B ─────┤
        VPC-C ─────┼──── Transit Gateway ──── Direct Connect
        VPC-D ─────┤                     ──── VPN
        VPC-E ─────┘

The networking team is already planning the migration. I expect we will have Transit Gateway deployed in production within the first quarter of 2019.

Key details that matter for production use:

  • Bandwidth: Up to 50 Gbps per VPC attachment. Sufficient for our workloads.
  • Route tables: Transit Gateway supports multiple route tables, enabling network segmentation. Production VPCs can route to each other but not to sandbox VPCs.
  • Cross-region: Transit Gateway Peering is coming (announced but not yet available), which will extend this model across regions.
  • Pricing: $0.05 per GB of data processed plus hourly attachment charges. For our traffic volume, the cost is lower than the operational overhead of managing peering connections.

Lambda Layers

Lambda Layers solve one of the most persistent pain points in serverless development: dependency management.

Previously, every Lambda function was a self-contained deployment package. If ten functions used the same Python library (say, the AWS SDK, a database driver, and a shared utility module), that library was duplicated ten times. Updating it meant redeploying all ten functions.

Lambda Layers let you package shared code and data into reusable components that multiple functions can reference. A layer is just a ZIP archive that gets extracted into the /opt directory of the Lambda execution environment. Functions can use up to five layers.

We immediately see three use cases:

Shared libraries. Our Python Lambda functions share a common set of dependencies: boto3 (pinned to a specific version), our internal SDK, a logging library, and an Oracle database driver (cx_Oracle with the Oracle Instant Client). Packaging these as a layer means our function deployment packages shrink from 50MB to under 1MB, and dependency updates are a single layer deployment.

Custom runtimes. Lambda Layers enable custom runtimes, which means Lambda can now run any language that can run on Amazon Linux. AWS announced support for Ruby and C++ at launch, but the custom runtime API means you can bring your own. We have a team evaluating Rust for a performance-sensitive Lambda function.

Shared configuration. Layers can contain data files, not just code. We are considering using a layer for shared configuration data (feature flags, environment-specific settings) that multiple functions need.

# Function code (small, focused)
import shared_utils  # from layer
import cx_Oracle     # from layer

def handler(event, context):
    conn = shared_utils.get_oracle_connection()
    # ... business logic

The deployment model changes significantly. Instead of deploying monolithic function packages, we deploy thin function code that references shared layers. This is closer to how traditional applications manage dependencies, and it is a welcome improvement.

Lambda ALB Integration

Lambda functions can now be registered as targets behind an Application Load Balancer. This is significant because it provides an alternative to API Gateway for HTTP-triggered Lambda functions.

API Gateway is powerful but expensive at scale ($3.50 per million requests plus data transfer) and complex to configure. ALB pricing is based on fixed hourly charges plus LCU (Load Balancer Capacity Unit) consumption, which is often cheaper for high-throughput workloads.

For our content API, which handles roughly 100 million requests per day, the cost difference between API Gateway and ALB is substantial. We are evaluating a migration for endpoints that do not need API Gateway's features (request validation, usage plans, API keys).

AWS Step Functions Service Integrations

Step Functions can now directly integrate with DynamoDB, SQS, SNS, ECS, Fargate, Glue, SageMaker, and other services without requiring a Lambda function as an intermediary.

Previously, orchestrating a workflow that wrote to DynamoDB and then sent an SNS notification required two Lambda functions: one to call the DynamoDB API and one to call the SNS API. These "glue" Lambda functions added latency, cost, and maintenance burden for trivial operations.

With direct service integrations, a Step Functions state can call DynamoDB directly:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::dynamodb:putItem",
  "Parameters": {
    "TableName": "workflow-state",
    "Item": {
      "id": {"S.$": "$.workflowId"},
      "status": {"S": "completed"},
      "timestamp": {"S.$": "$$.State.EnteredTime"}
    }
  },
  "Next": "SendNotification"
}

This eliminates an entire category of Lambda functions whose sole purpose was proxying calls to other AWS services. Our Step Functions workflows will become simpler and cheaper.

Amazon DynamoDB On-Demand

DynamoDB On-Demand pricing removes the need to provision read and write capacity units. You pay per request, and DynamoDB scales automatically to handle your traffic.

This is a significant operational simplification for workloads with unpredictable traffic patterns. We have several DynamoDB tables that spike from 100 requests per second to 10,000 during content release events. Managing provisioned capacity for these tables required auto-scaling configurations, alarms, and over-provisioning margins.

On-Demand pricing is more expensive per request than provisioned capacity at steady state. But for bursty workloads, the elimination of capacity planning and throttling risk more than compensates. We will likely use a hybrid approach: provisioned capacity for predictable tables, on-demand for bursty ones.

AWS CloudMap (Service Discovery)

CloudMap provides managed service discovery for cloud resources. Services register their instances (IP addresses, URLs, ARNs) with CloudMap, and consumers discover them via DNS or the CloudMap API.

We have been using a combination of Route 53 private hosted zones and ECS Service Discovery for this purpose, and CloudMap formalizes and extends that pattern. The ability to register non-ECS resources (EC2 instances, on-premises services, Lambda functions) in the same namespace is the key improvement.

What I Am Skipping

I am deliberately not covering several announcements that got significant attention but that I do not expect to use in the near term:

  • AWS Outposts (on-premises AWS hardware): interesting concept but early and expensive.
  • Amazon Quantum Ledger Database: niche use case, and our audit logging needs are served by CloudTrail and DynamoDB.
  • Amazon Managed Blockchain: solution looking for a problem in our domain.
  • Firecracker: fascinating technology (the microVM that powers Lambda and Fargate), but an infrastructure layer we will never interact with directly.

The Meta-Observation

The theme of re:Invent 2018 is AWS filling gaps. Transit Gateway fills the networking gap. Lambda Layers fill the dependency management gap. Step Functions integrations fill the orchestration gap. DynamoDB On-Demand fills the capacity planning gap.

These are not revolutionary new categories. They are solutions to real operational pain points that practitioners have been dealing with for years. And honestly, that is more valuable than revolutionary new categories. I would rather have a Transit Gateway that simplifies my daily networking work than a quantum computing service I will not use for a decade.

The announcements that will change how I work day to day: Transit Gateway, Lambda Layers, and DynamoDB On-Demand. Everything else is interesting but not immediately actionable. I will revisit in six months when these services have had time to mature and we have production experience with them.

Share: