Automating infrastructure health checks off a manual, offshore process

Three times a day, an offshore support team logged into every storage array, SAN switch, and vCenter by hand. They read the metrics off each system, assembled them into dashboards, and emailed the result out. Then four hours later they did it again.

It worked, in the sense that the emails arrived. But a human reading a console three times a day sees what a human sees: the number in front of them, right now, with no memory of what it was yesterday. Nobody was comparing this morning's error counters against last night's. Nobody could, because the previous reading existed only in an email that had already scrolled away.

My manager gave me the directive: replace it.

The first version was PowerShell on Windows Task Scheduler. It connected to each system, pulled the metrics the offshore team had been reading manually, applied thresholds, and emailed a formatted report — three times a day, on the same cadence the people had been working, because that cadence was what the operation expected. The schedule is a fossil of the manual process, and it still is.

That version worked, and it exposed a second problem that shaped everything after it: a script on a tools server is not a system. It ran, but there was no record of which version had run, no artifacts, no history, and no way for anyone but me to change it safely. Solving that became a platform in its own right — described separately — and the health checks were the first workloads promoted onto it. They now run as Jenkins pipelines out of GitLab, in purpose-built container runtimes, pulling every credential from a credential store, archiving their own evidence on every run.

What went into them is mostly judgment about what actually indicates trouble. Eighty explicit thresholds, authored per platform and written into configuration rather than buried in code:

Buffer credits at zero as immediate-critical, below five as a warning, sustained over 300 seconds. CRC errors above ten per million frames. SFP temperature at 70 °C warning and 85 °C critical, with a flag for any change over 10 °C in an hour. Transceiver receive power above 0 dBm or below −15 dBm. Replication lag warning at one hour, critical at four. SnapMirror lag past twelve hours. Datastores and aggregates at 80 and 90 percent. Aggregate expansion blocked outright if the change exceeds one percent or would leave the aggregate above 95 percent full.

The SAN alerting works on deltas rather than absolutes, which is the piece that answers the original problem directly. A switch port with 4,000 lifetime CRC errors may have been that way for three years and mean nothing. The same port gaining 100 errors since the last run is a failing optic right now. So the engine keeps its own state across runs: link failures rising at all is critical, CRC errors climbing on three of the last four runs is critical even when the count is small, and no delta rule will fire until it has at least two runs of retained history to compare. That is the capability a person reading a console three times a day could never have had.

Retained-window success: 209 of 213.

The reports go to the storage and virtualization engineers who act on them, to the offshore team, to my manager, and up through a director and a senior director. The same run, in the same format, serves the person who has to go replace an optic and the person who wants to know whether the estate is getting worse.

What it catches

Capacity, routinely. That is the thing this system does most often and most usefully, and it works because it was built as a loop rather than as a report.

Volumes, aggregates, and datastores are measured against thresholds on every run. Ninety days of capacity history is retained, so the check is not only asking how full is this but how fast is it filling — a volume crossing 80 percent gets flagged, and any volume projected to reach full in under thirty days gets flagged whether or not it has crossed a threshold yet. Growth is the signal; the current number is just where the growth happens to be today.

Then it escalates where someone will actually see it. Anything over 80 percent renders yellow in the report and anything over 90 renders red — and each red increments a counter that writes itself into the subject line of the email. That detail is the whole design. The failure mode of a daily report is not that it's wrong, it's that after ninety uneventful mornings nobody opens it. So the severity has to leave the report and reach the inbox on its own.

For the volumes that cross 90 percent, the check emails a service-management intake mailbox directly, which opens a ticket without a person deciding to open one.

And the remediation path is guarded rather than automatic. The expansion tooling refuses to act if the change would exceed one percent of the aggregate, or if the result would leave that aggregate above 95 percent full — so the system that tells you a volume is filling will not let you fix it by quietly creating a worse problem one layer down.

Detect, forecast, escalate visibly, open the ticket, and refuse an unsafe fix. Capacity problems get handled before they become incidents because every step of that runs three times a day whether anyone is thinking about it or not.

What they watch

SAN switches	16 across two fabrics
vCenters	14 across two sites
Storage arrays	6 current, 8 at peak
NetApp clusters	2 current-generation, 2 legacy
Virtual machines, one discovery run	4,156 found, 1,924 counted as servers
Storage under watch	775 TB committed, 1.3 PB provisioned
Client organizations served	~159 banks and credit unions

The manual process it took over was roughly two full-time offshore positions.

That is my estimate and not a measured baseline. Nobody timed the old process before it went away, and I am not going to manufacture a number now and present it as measured — the honest version is that the work of two people, three times a day, every day, stopped being done by people.

What actually changed is harder to put a figure on and matters more. The manual process produced a snapshot; this produces a series. Eighty thresholds get evaluated on every run whether or not anyone is looking, error counters get compared against their own history rather than against a person's recollection, and the result is archived as evidence rather than sent as an email that scrolls away.

A monitoring system has one failure mode worse than all the others: reporting health it cannot actually see. A monitor that crashes gets fixed the same morning. A monitor that returns green while blind runs for a year and makes the environment look safer than it is every hour it runs.

I have found four of them in my own systems, and they are worth listing because the alternative is a page claiming I write monitors that never lie.

The storage check treated a missing answer as a good answer — a component with no reported health state defaulted to OK, and a parse failure rendered as All OK. Fixed in 2025 to report Unknown with a log line, alongside a written remediation summary classifying every masked path by risk.

The virtualization health score was always 100 percent. Not usually — always. When I rebuilt it, the first honest run reported 32 active alarms, 74 snapshots past their age limit, 23 hosts in maintenance mode, and 7 of 14 vCenters unreachable from the monitoring host. Half the environment had been invisible behind a green number.

The monitoring-account check confirms a login it never verified. If the query throws, the exception is caught, logged as a warning, and the failure counter stays at zero — which renders as the account is currently logging in correctly. A check that could not run and a check that passed produce identical, affirmative output.

Seven clusters reported zero percent CPU and memory with hosts connected, at DEBUG level where nobody reads, and the zeros fed the health score as healthy values.

Two of those four are fixed in the pipeline-executed copy and still present in a second, manually-run copy that has drifted from it. That divergence is itself the finding: two copies of one tool, one corrected, one not, and nothing in place that would have told me.