A Green Report Has to Mean Something.

Three times a day, an offshore support team logged into every storage array, SAN switch, and vCenter by hand. They read the metrics off each system, assembled them into dashboards, and emailed the result out. Then four hours later they did it again.

The emails arrived, but each was a snapshot. Nobody was comparing this morning’s error counters with last night’s. The previous reading was in an older email, so looking for a developing problem meant finding and comparing those messages.

My manager asked me to replace the manual process.

I started with PowerShell on Windows Task Scheduler. It connected to each system, collected the metrics the team had been reading, applied thresholds, and emailed a formatted report. I kept the three-times-a-day schedule because that was what people expected. The automated schedule came directly from the manual one.

It worked, but I had no record of which version ran, no archived output, and no safe way for anyone else to change it. Fixing that became a separate platform project. The health checks were its first workloads. I moved them to Jenkins pipelines using GitLab, purpose-built containers, stored credentials, and archived results. The SAN exception is recorded below.

I wrote eighty explicit thresholds for the different platforms and put them in configuration. I wanted the rules for identifying trouble to be visible without digging through the code:

Buffer credits at zero as immediate-critical, below five as a warning, sustained over 300 seconds. CRC errors above ten per million frames. SFP temperature at 70 °C warning and 85 °C critical, with a flag for any change over 10 °C in an hour. Transceiver receive power above 0 dBm or below −15 dBm. Replication lag warning at one hour, critical at four. SnapMirror lag past twelve hours. Datastores and aggregates at 80 and 90 percent. Aggregate expansion blocked outright if the change exceeds one percent or would leave the aggregate above 95 percent full.

The SAN checks compared counters with their previous values. A port with 4,000 lifetime CRC errors might have had them for three years. A port gaining 100 since the last run needed investigation for an optic failing now. I retained state between runs: any rise in link failures was critical, and CRC errors increasing on three of the last four runs were critical even if the count was small. Delta rules waited for at least two saved readings. The old process hadn’t made those comparisons.

In the retained window, 209 of 213 runs succeeded.

The reports went to storage and virtualization engineers, the offshore team, my manager, a director, and a senior director. An engineer could use them to investigate an optic. A leader could use the same report to see whether conditions across the systems were getting worse.

What it catches

Capacity was what the checks caught most often. I wanted a capacity finding to lead to action instead of sitting in an email.

Each run checked volumes, aggregates, and datastores against their thresholds. I kept ninety days of capacity history to see how fast they were filling. A volume crossing 80 percent was flagged. So was a volume projected to fill in under thirty days, even if it hadn’t crossed a threshold yet.

Above 80 percent showed yellow; above 90 showed red. Each red finding added to a count in the email subject. I couldn’t assume someone would keep opening a report after ninety uneventful mornings. The subject needed to tell them there was something to look at.

For the volumes that cross 90 percent, the check emails a service-management intake mailbox directly, which opens a ticket without a person deciding to open one.

Expansion was a separate step with its own checks. The tooling refused a change larger than one percent of the aggregate, or one that would leave it above 95 percent full. Fixing a full volume shouldn’t leave the surrounding aggregate in worse trouble.

The checks ran three times a day, forecast capacity trouble, put it in front of people, and opened tickets at the critical threshold. The separate expansion checks blocked unsafe fixes. Together, those steps helped get capacity problems handled before they became incidents.

What they watch

SAN switches	16, across two fabrics
vCenters	14
Storage arrays	6 current, 8 at peak
NetApp clusters	2 current-generation, 2 legacy
Virtual machines, one discovery run	4,156 found, 1,924 counted as servers
Storage under watch	775 TB committed, 1.3 PB provisioned

I estimated the manual work replaced at roughly two full-time offshore positions.

Nobody timed the old process before it went away. This is my estimate of the repeated work people stopped doing three times a day, not a measured baseline or a claim that two jobs were eliminated.

The saved history was another change. Eighty thresholds were evaluated every run, error counters were compared with earlier readings, and the results were archived. Engineers had more to work from than a series of emails.

What worried me most was a check saying things were healthy when it couldn’t see them. A failed monitor is obvious. A green report can leave people believing systems are fine when they haven’t been checked.

I found four examples in my own systems:

The storage check treated missing data as healthy. A component with no reported health state defaulted to OK, and a parse failure displayed All OK. I fixed that in 2025 to report Unknown and write a log entry. I also documented the paths that hid failures and classified their risk.

The virtualization health score always reported 100 percent. After I rebuilt it, the first honest run showed 32 active alarms, 74 snapshots past their age limit, 23 hosts in maintenance mode, and 7 of 14 vCenters unreachable from the monitoring host. Half the environment had been missing behind that green score.

The monitoring-account check reported a successful login without verifying it. When the query threw an exception, the code logged a warning but left the failure counter at zero. The report then said the account is currently logging in correctly. A failed query and a passed check gave the reader the same answer.

Seven clusters reported zero percent CPU and memory even though hosts were connected. The log entries were at DEBUG level, and the health score treated those zeros as healthy readings.

At the time of the review, two of those four problems were fixed in the pipeline-executed copy but still present in a manually run copy. The copies had drifted, and I had nothing in place to alert me to that.