Security Automation: Days to Hours Provisioning

Three months into my Lead Systems Engineer role, I finally have something concrete to show for all the late nights and weekend work: we cut our server provisioning time from five days to under four hours.

Let me explain why this matters and how we did it.

The Old Way

When I started, provisioning a new server looked like this. A request came in from a client or an internal team. Someone from our team would create a virtual machine in VMware vSphere, manually selecting the compute resources, storage, and network. Then they would mount an ISO and walk through the RHEL installer, clicking through screens and entering configuration values. After the OS was installed, they would manually configure networking, create user accounts, set up SSH keys, harden the security configuration, install monitoring agents, configure log forwarding, and run through a checklist of about forty other tasks.

Each of these steps required human attention. Each step was an opportunity for error. And each step depended on the engineer remembering to do it and doing it correctly.

The whole process took three to five business days, depending on how many other requests were in the queue and how many interruptions the engineer dealt with along the way.

Five days to stand up a Linux server. In 2013. That is not acceptable.

Where the Time Actually Went

Before I could fix the process, I needed to understand where the time was going. So I tracked a few provisioning requests in detail, noting every step and how long it took.

The breakdown surprised me. The VMware provisioning itself was quick, maybe thirty minutes. The OS installation took about an hour with the manual installer. But the post-installation configuration took the bulk of the time. And much of that was not active work; it was waiting. Waiting for someone to provide the network configuration. Waiting for firewall rules to be approved. Waiting for the client to confirm their requirements. Waiting for another team to set up DNS entries.

The actual hands-on work was maybe four hours total. The five-day timeline was mostly queuing and coordination overhead.

That told me two things. First, I could automate the hands-on work and cut it to under an hour. Second, I needed to fix the coordination process to eliminate the waiting.

The Automation Stack

Here is what we built, piece by piece.

VMware Templates: Instead of installing the OS from scratch every time, we created golden VMware templates. A template is basically a pre-installed VM image with the base OS configured to our standards. Deploying from a template takes about five minutes instead of an hour. We created templates for RHEL 5 and RHEL 6, each with our standard security hardening already applied.

Kickstart for Customization: For cases where we need a custom installation (different partitioning scheme, different package selection), we use Kickstart files. A Kickstart file is a text file that answers every question the RHEL installer would normally ask. Disk layout, package selection, network configuration, root password, timezone, everything. You point the installer at the Kickstart file and walk away. It does the entire installation unattended.

I wrote a base Kickstart file and created a system for generating customized versions based on client requirements. Change a few parameters, regenerate the file, and the installer handles the rest.

Post-Install Scripts: After the OS is installed, a shell script runs automatically to handle all the post-installation tasks. Creating standard user accounts. Configuring SSH to disable root login and password authentication. Setting up iptables firewall rules. Installing and configuring the Nagios monitoring agent. Setting up log forwarding to our central syslog server. Applying CIS security benchmarks. Registering the server with our inventory system.

One script, about three hundred lines of bash, replaces forty manual checklist items. And unlike a human, the script never forgets a step.

PXE Boot: For physical servers (we still have a few), we set up PXE boot so machines can boot from the network and pull their Kickstart configuration automatically. No CDs, no USB drives, no walking to the data center with installation media.

The Security Angle

One thing I insisted on from the start is that automation must make security better, not worse. This was a hard sell initially because some people associate automation with cutting corners.

The opposite is true. When you automate security hardening, every single server gets the same hardening. No exceptions, no "I forgot to disable root SSH login on that one server," no configuration drift over time.

Our post-install script applies about sixty security configurations based on the CIS (Center for Internet Security) benchmark for RHEL. Things like disabling unnecessary services, setting proper file permissions, configuring audit logging, enforcing password complexity, disabling USB storage, and restricting cron access.

Before automation, compliance with these benchmarks was spotty. Engineers would apply most of the settings but miss a few, especially under time pressure. Now, every server that comes out of our pipeline is compliant from the moment it is deployed.

The Process Fix

Automating the technical work was the easy part. The harder part was fixing the coordination overhead.

I created a standard provisioning request form that captures all the information we need upfront: hostname, IP address, DNS entries, resource requirements, application team contact, firewall rules. No more back-and-forth emails asking for missing information.

I also worked with the network and DNS teams to establish a pre-allocated pool of IP addresses and DNS entries for each client. Instead of requesting a new IP for every server, we draw from the pool. That eliminated the two-day wait for network team approvals.

The Results

The new process looks like this. A provisioning request comes in with all required information. An engineer deploys from a VMware template, customizes the hostname and network settings, and runs the post-install script. Total hands-on time: about thirty minutes. Total elapsed time including validation and handoff: under four hours.

From five days to four hours.

But the numbers only tell part of the story. The real benefit is consistency. Every server we deploy now is configured identically. Same security settings, same monitoring, same log forwarding, same baseline packages. When something goes wrong, we know exactly what the configuration should look like because it is defined in our scripts and templates.

What I Learned

This project taught me something important about automation: the value is not just in speed. Speed is the obvious benefit and the easiest to sell to management. But the real value is in consistency, repeatability, and reduced error rates.

A human doing the same forty-step process for the hundredth time will make mistakes. Not because they are careless, but because humans are not designed for repetitive precision. A script does it right every time.

I also learned that automation is a cultural change, not just a technical one. Some team members were resistant at first. They worried that automation would make their jobs obsolete. I had to show them that automation eliminates the boring, repetitive work and frees them up for more interesting challenges like troubleshooting complex issues, designing new architectures, and learning new technologies.

The server provisioning automation is just the beginning. I have a list of other processes that are begging to be automated. Patch management, user account provisioning, backup verification, compliance reporting. Each one is a five-day process that could be a four-hour process.

We are just getting started.