AWS re:Invent 2013: Announcements That Matter

AWS re:Invent is happening this week in Las Vegas, and I am following it remotely with the kind of obsessive attention that my colleagues find slightly concerning. I have been refreshing the AWS blog, watching livestreams at odd hours, and reading every summary I can find.

This is the second re:Invent (the first was last year), and the announcements are significant. Let me break down the ones I think actually matter, as opposed to the ones that are just incremental improvements.

CloudTrail: Finally, an Audit Log

AWS announced CloudTrail, a service that logs all API calls made to your AWS account. Every time someone launches an instance, modifies a security group, creates an S3 bucket, or does anything else through the AWS API, CloudTrail records it.

This might sound boring, but it is enormously important. One of the biggest challenges with cloud infrastructure is knowing who did what and when. When a security group gets modified and suddenly your database is exposed to the internet, you need to know who made that change. When instances get terminated unexpectedly, you need to trace back the API call that did it.

Before CloudTrail, you had to rely on AWS being transparent about what happened in your account, or you had to build your own logging infrastructure on top of the API. Now it is built in.

For anyone managing production workloads on AWS, this is the announcement that matters most. Security and compliance teams have been asking for this since day one.

Amazon Kinesis: Real-Time Data Streams

Kinesis is AWS's answer to Apache Kafka. It is a managed service for real-time data streaming. You push data into a Kinesis stream, and any number of consumers can read from it in real time.

The use cases are compelling. Real-time analytics on clickstream data. Processing IoT sensor data as it arrives. Building real-time dashboards. Replicating database changes to downstream systems.

What makes Kinesis interesting is not the technology itself (streaming data is a well-understood problem) but the fact that AWS is offering it as a managed service. Running your own Kafka cluster is operationally demanding. Kinesis eliminates that overhead.

I am watching this one because data streaming is becoming increasingly important. Applications are moving from batch processing (run a job every night to process yesterday's data) to stream processing (process data as it arrives). Kinesis makes that transition much easier for teams running on AWS.

WorkSpaces: Desktop as a Service

AWS announced WorkSpaces, a managed virtual desktop service. You provision a virtual Windows desktop in the cloud, and users connect to it from any device with a client application.

This is interesting for a few reasons. First, virtual desktop infrastructure (VDI) is one of those technologies that enterprises want but struggle to operate. Running your own VDI environment with VMware Horizon or Citrix XenDesktop is complex and expensive. A managed service could change the economics significantly.

Second, this signals that AWS is serious about the enterprise market. Virtual desktops are an enterprise technology through and through. Home users do not need them. Startups do not need them. Large companies with compliance requirements, remote workers, and BYOD policies need them.

AWS is clearly going after enterprise IT budgets, and WorkSpaces is a direct play for that market.

RDS for PostgreSQL

AWS added PostgreSQL support to RDS (Relational Database Service). RDS already supported MySQL, Oracle, and SQL Server. Adding PostgreSQL fills an important gap.

PostgreSQL has been gaining popularity steadily, especially among teams that want a powerful, open-source relational database without MySQL's licensing complications (which got more complex after Oracle acquired Sun). RDS for PostgreSQL means you can run PostgreSQL with the same managed convenience as RDS MySQL: automated backups, point-in-time recovery, read replicas, and multi-AZ deployments.

We use PostgreSQL in several environments, and the operational overhead of managing it is significant. Backups, replication, failover, patching. Having AWS handle all of that is genuinely appealing.

The Bigger Picture

What I find most interesting about this year's re:Invent is not any individual announcement. It is the overall direction.

AWS is moving up the stack. In the early days, AWS was about infrastructure: virtual machines, storage, networking. Basic building blocks. Now they are offering managed services for increasingly complex workloads. Managed databases, managed streaming, managed desktops, managed caching, managed search.

Each managed service represents an operational burden that teams no longer have to carry. You no longer need a DBA to run a database. You no longer need a streaming expert to process real-time data. You no longer need a VDI team to provide virtual desktops.

This has profound implications for the IT industry. The skills that are valuable are shifting. Deep expertise in running specific technologies (database administration, for example) becomes less important when AWS will do it for you. What becomes more important is understanding architectures, designing systems that leverage managed services effectively, and knowing how to operate in a cloud-native way.

Following From a Distance

I should acknowledge the irony of following a cloud computing conference from a place where reliable internet access is not universal and cloud adoption in enterprises is still in early stages. Most of the companies I work with are still running traditional data centers. VMware virtualization is considered cutting-edge.

But that is changing. I see more interest in cloud services every month. Clients are asking about AWS. They are asking about hybrid architectures. They are starting to understand that running their own data centers might not be the most efficient use of their resources.

Cloud adoption here will accelerate, and when it does, it will happen fast. The talent is here. The engineering capability is here. What we need is for the business leadership to get comfortable with the idea of putting their workloads on someone else's infrastructure.

Conferences like re:Invent help build that comfort. Even following from a distance, seeing the pace of innovation and the growing ecosystem of managed services makes the cloud feel less like a risky experiment and more like an inevitable future.

What I Am Going to Try

Based on this year's announcements, I have a few things I want to experiment with.

First, I want to set up CloudTrail in our AWS development account and start building a habit of audit logging from day one. Security is always harder to retrofit than to build in from the start.

Second, I want to explore RDS PostgreSQL as a potential replacement for some of our self-managed database instances. If the performance and compatibility are good, the operational savings could be significant.

Third, I want to spend more time understanding the AWS pricing model. One of the challenges with cloud adoption is that costs can be unpredictable if you do not understand how billing works. I want to build that knowledge so I can have informed conversations when clients ask about cloud economics.

re:Invent is over, but the work of understanding and applying what was announced is just beginning. Back to the documentation.