We Had the Automation. Nobody Was Using It.

What was needed

We were paying a six-figure annual license for a commercial infrastructure automation platform. The product could do much of what we needed, and arguably more.

Nobody used it.

It had been poorly implemented and never adopted. We kept paying for it while the automation people actually used lived on laptops and shared tools servers. Those scripts worked, but someone else couldn’t safely change them.

I had the same problem. My infrastructure health checks had replaced manual offshore work. They ran successfully on Windows Task Scheduler on a tools server. But I had no version history, saved output files, or record of what had run. I wouldn’t have been comfortable telling someone else to depend on that setup.

We’d made several attempts to get systems administrators working with shared source control and repeatable deployments across the enterprise. As far as I know, this was the first attempt in this organization that reached production and stayed there.

What I built

What bothered me was that we already had a capable tool. People weren’t using it. I needed to address the reasons engineers hesitate to move their work: losing control of it, handing ownership to someone else, or having to rewrite scripts that already work.

I separated the requirements from the platform I was building.

The requirements were five minimum controls for automation touching production: versioned source, a repeatable way to run it, controlled credentials, useful output or evidence, and a named owner. Any tool meeting those five points was fine.

My platform was one way to meet them. GitLab held source, history, and runbooks. Jenkins scheduled and coordinated jobs, accepted parameters, required approvals, and retained logs and output files. Purpose-built Docker agents provided isolated Windows and Linux runtimes so workflows didn’t interfere with each other. Jobs used service accounts rather than personal accounts. Credentials came from a store, were limited to the folder and job, and were purged after the run.

I wanted to say clearly that teams didn’t have to move everything onto my tool. That was close to what had gone wrong before. The requirements mattered more than my choice of software.

Engineers kept ownership of their logic, its correctness, and its failure handling. The platform handled the runtime, credential injection, saved results, version tracking, access control, and audit logs. They could keep their scripts and run them somewhere another person could repeat the work.

For a working script, I used this sequence: put it in source control, run it manually through Jenkins, add parameters, add pre-checks, save the results, add an approval where the risk called for one, and document the owner. Each addition was useful without requiring a rewrite.

I wanted the team to be able to run and own its workflow without routinely depending on whoever wrote it. A job merely existing in Jenkins didn’t tell me that had happened.

What was running at the time

At the time of this account, three of my own workflows ran on a schedule against production infrastructure:

Those workflows had 2,415 production runs, with 209 of 213 succeeding in the retained window. Every pipeline, 13 of 13, pulled its secrets from the credential store. None had inline secrets in its pipeline file. Agent images explicitly pinned their contents. Terraform build workflows required approval. The directory sync took a full database backup before each run and stopped if the backup failed.

Across the controller, 3,902 builds had run, including 1,285 in four other groups’ folders. One group accounted for 893 builds across thirteen jobs, two on schedules. Those counts measured work, not users. At the time, I and roughly five other systems administrators, engineers, and architects used the platform directly.

It was in production, but use was still early. I was communicating what it could do, helping engineers find repeatable work, and helping them move that work onto it. The target was fifteen to twenty people building and operating workflows, plus another twenty running work someone else had built. The organization had several hundred people. Those were targets, not achieved user counts.

What it replaced

A six-figure annual license renewal, replaced at no license cost.

The commercial platform could do more than mine. It was a mature product with features I hadn’t built and might never build.

But we weren’t using it. My replacement had about 3,900 real executions and no license cost, though we still paid for infrastructure and had to operate it. I cared about which jobs people actually ran and whether the next person could change them safely.

What I still needed to fix

The review also found that my own platform fell short on two of its five requirements. Here’s what was missing at the time.

No pipeline checked system state after making a change. Zero of thirteen. Several had thorough pre-checks for credentials, syntax, and module availability, and all archived output. None independently confirmed the result afterward. That mattered particularly on the build pipelines; read-only health checks carried less risk from that gap.

None of the ten repositories documented rollback. One documented taking a backup before a change. It still didn’t explain how to undo the change.

The SAN health check ran outside the platform. It had produced 1,269 hourly runs of output on disk, but its Jenkins job had only one build in its history. Something was running it hourly. I hadn’t established what, and that execution had no Jenkins build record, platform output retention, or failure alerting.

Two build pipelines had never run. Two more had last run nine months earlier. They had code and approval gates, but that wasn’t evidence of work people were currently using.

One repository’s documentation contradicted its code in six places, including a claimed safety default that the shipped script overrode. The code was right and the document was wrong. Nothing failed to draw attention to the bad instructions.

What I learned

1. We had paid for the license, setup, and support of a platform nobody adopted. Before replacing it, I needed to understand why people hadn’t used it. More features wouldn’t answer that.

2. I wrote down the minimum controls separately from my tool. Other teams could meet them without giving me their work or using my platform.

3. A working script was worth keeping. Asking someone to rebuild it made starting harder. I could add source control and safer execution around what they already had.

4. One successful run wasn’t enough. I wanted someone else to be able to run it next quarter and see what it did.

5. Jenkins log rotation hid older builds. The UI showed 92 builds for a job that had run 1,256 times. nextBuildNumber - 1 survived rotation. I was undercounting my own platform by a factor of thirteen until I checked that.

6. I had to compare the documentation with the code. A wrong instruction can sit there for years when the code keeps working and nobody has a reason to question it.