A check that could not fail
My development CA went down and I didn't notice, because the thing I was using to check it was answering. The device answering was an air conditioner.
My development CA went down and I did not notice, because the thing I was using to check it was answering.
The device answering was an air conditioner. It cost me a day. Almost none of that day was spent on the actual fault.
Setup
I build Signum, an e-signature platform: PDFs in, PAdES-BASELINE-LT signatures out — long-term-validatable, with an RFC 3161 timestamp and embedded revocation data. The PKI is EJBCA CE, a Root CA over a Sub CA over per-user signing certificates, and the signing itself runs through the EU DSS library.
The CA runs on a machine on my LAN. This is a development environment. In production the revocation endpoints are public URLs on a name I control; on my desk they are an IP address on a flat home network. That trade is deliberate and it is not what this article is about.
What it is about is that the same flat network contains consumer devices, and one of them took the CA’s address.
The failure
First full browser test of the signing flow. Signing failed, and the failure was reported in the vocabulary of certificate status — the system could not establish whether a certificate was valid, and said so.
So I looked at certificates. Signing certificate valid, Sub CA valid, CRL fresh, DSS trust chain configured correctly (Root CA in trustedCertSources, Sub CA in adjunctCertSources — get that backwards and DSS silently skips revocation checks on the Sub CA, which is its own article). EJBCA healthy; it signed a new CRL on demand.
All fine. And the CA host answered pings.
That last check is the defect. Not a symptom of it — the defect.
Twenty minutes of work, spread over a day
A host that answers ICMP but refuses every port it is supposed to serve is not a host with a service problem. It is a different host. Once I actually looked:
$ ping 192.168.1.10
64 bytes from 192.168.1.10: icmp_seq=0 ttl=64 time=2.1 ms
$ curl http://192.168.1.10:8280/crl/SubCA.crl
curl: (7) Failed to connect to 192.168.1.10 port 8280: Connection refused
$ arp -n 192.168.1.10
# MAC does not belong to the CA machineA port scan returned exactly one open port: 6668/tcp, the Tuya local control protocol. A smart air conditioner, recently installed, had been leased the CA’s address by DHCP and won the ARP race.
This should have taken twenty minutes. ping succeeded, curl was refused — the contradiction is right there in the first two commands, and arp is the obvious third. It took a day because I trusted the ping and let the error message pick my search space for me.
Both halves of that are worth separating, because only one of them is a mistake I could have avoided by being smarter, and the other is a thing you fix with tooling.
Why the address was takeable at all
It was a plain misconfiguration and the router did nothing wrong.
The CA held 192.168.1.10 as a statically configured secondary address on the machine, never registered anywhere. 192.168.1.10 sat inside the router’s DHCP pool. So the router had an address it believed to be free, a new device asked for one, and it handed over an address that was already load-bearing.
A statically assigned address inside a live DHCP scope, with no corresponding reservation, is a collision waiting for its trigger. The AC was just the first device to show up.
Why it broke signing, and not something more obvious
This is the part that generalises past my network.
Every certificate the CA issues has its revocation endpoints written into it — CRL distribution point, OCSP responder, AIA — and those URLs are signed in and cannot be edited afterwards. To produce a long-term-validatable signature, the library must fetch that revocation data at signing time and embed it in the PDF.
My configuration is deliberately fail-closed:
certificateVerifier.setAlertOnMissingRevocationData(new ExceptionOnStatusAlert());That is the setting I want. A PAdES-LT signature that quietly ships without embedded revocation data is claiming a validation property it does not have, and I would rather the signing operation fail loudly than emit a document that overstates itself.
So the fault propagated as: wrong device on the address → revocation data unreachable → signing aborts → and by the time it reached me it was phrased in the vocabulary of certificate status.
The message was not wrong. It was reporting, accurately, that the system could not establish the status of a certificate. But cannot establish status and certificate has a problem arrive looking almost identical, and I read it as the second one for several hours longer than I should have.
The fix that mattered
Not the network config. The check.
The CA host exported no metrics at all, so its availability was something I confirmed by hand, with a tool that could not tell me what I actually needed to know. ping answers a question about an address. Every question I had was about a service.
I added blackbox_exporter probes against the CRL distribution point, the OCSP responder, the AIA certificate download and the admin UI. The design detail that carries the whole thing is that the probes assert the response, not the reachability — because that is what separates the two failure modes:
| Failure mode | probe_success |
probe_http_status_code |
|---|---|---|
| CA host down | 0 | 0 |
| Different device answering on its address | 0 | 404 |
The second row is this incident. It took a day by hand and it is now a distinct signal I can alert on, because a wrong device answering produces a response — just not one that means anything. A reachability check cannot see that difference. A check that asserts what it expects to get back can.
What actually held
The obvious fix was excluding .10 from the DHCP pool. It did not hold — with the CA machine powered off for an evening, the AC took the address straight back, same single open port, every CA port closed.
The durable version is a reservation keyed to the CA’s MAC rather than an exclusion, and that is the config-level answer. But the reason the incident is over is not the router setting. It is that the failure is now observable in seconds and specifically identifiable as wrong device, so if the address is ever contested again I find out immediately instead of by way of a confusing signing failure two layers up the stack.
Config that prevents a failure mode is worth having. Detection that survives the config being wrong is worth more.
The thing I would carry to any system
A check that cannot fail is worse than no check at all.
No check leaves you uncertain, and uncertainty makes you go and look. A check that always passes gives you a fact — and that fact walks into your next hypothesis and quietly deletes an entire region of the search space. I did not spend a day because the problem was hard. I spent a day because I had a green light pointed at the one thing that was broken.
Worth auditing your own monitoring for that shape: probes that confirm an address rather than a service, health endpoints that return 200 without touching a dependency, smoke tests that assert a process is running rather than working. They pass identically whether the system is healthy or not, which means the only signal they carry is that they are still running.