Compliant and breached
What the public record supports behind the eleven months: what happens when someone goes back and measures the gap between "assessed compliant" and "compliant now", and what a regulator found when it examined a certificate held at the time of a breach.
In June we argued that “compliant at the time” is a breach pattern, not a defence. That was an argument from structure: an audit is a sample taken on a day, and nothing in the process is built to notice what changes afterwards. Structure is persuasive but it is not proof. This is what the public record supports, and where it stops, in three parts.
The first is measurement, from the one place where an assessor has gone back, checked, and published what it found. The second is a regulator, stating the thesis in a published finding. The third is the well-known cases, which turn out to be less about drift than about scope and sampling, and are the more useful for it. None of the three is a post-incident report that dates the moment an in-scope control lapsed. That record is thin for a reason this post comes back to: between assessments, almost nobody is measuring.
Where someone actually measures the gap
Most certification regimes issue a document and then wait a year. In the payment card industry, one large assessor, Verizon, tests its clients between their annual validations against the Payment Card Industry Data Security Standard and has published the aggregate results for over a decade. That makes it the clearest public dataset on what happens to a compliant state after the assessment ends.
Verizon’s forensic team has been direct about what they see when they arrive after a breach. Their 2015 report put it plainly: of all the breaches its forensics team had investigated over the preceding ten years, “not a single organization was PCI DSS compliant at the time of the breach”. The 2017 report repeated the finding across every breach investigated, and put a number on one of the gaps: 91.9% of organisations assessed after a breach were not compliant with Requirement 10, the logging and monitoring requirement. The requirement that exists to record what is happening on the systems was one that nine in ten of them were not meeting when it mattered. The report does not say when each of them stopped meeting it, or whether they ever had, and that is the limit of what a post-breach assessment can tell you about drift: it records the state at the end, not the date it changed.
The interim figures show how quickly the state moves. In 2019, only 27.9% of organisations demonstrated full compliance at their interim validation, down from 36.7% the year before. In 2020 it recovered to 43.4%. Both numbers belong in the sentence, because the trend reversed and citing the low year alone would be selecting the evidence. Read together they say something more useful than either does alone: even in the better year, fewer than half of the organisations assessed were fully compliant when tested between their annual validations.
PCI is not ISO 27001, and I am not going to pretend the figures transfer. The argument is the reverse. PCI is where the decay is most visible because PCI is where it is most often measured. An ISO certificate is subject to an annual surveillance audit, and a SOC 2 report covers a stated period that organisations commonly renew each year, but there is no comparable public dataset recording how many of the organisations examined were still in the certified state when the auditor came back. The absence of a number for those regimes is not evidence that the gap is smaller. It is evidence that nobody is publishing the count.
A regulator states the thesis
In 2019 the Privacy Commissioner of Canada published its finding into the 2017 Equifax breach. Among the questions it examined was how Equifax Canada had satisfied itself about the security of its parent company, which held the data.
The answer, at paragraph 90, is that Equifax Canada’s monitoring “was limited to the receipt of tri-annual certificates of compliance with the ISO 27001 information security standard”. Two paragraphs later the finding records that the company also had access to external penetration tests and internal audits “which plainly demonstrated the existence of practices which were not ISO 27001 compliant”.
That is the whole argument of this post, made by an investigator rather than by us. The certificate was real. Equifax’s own annual report for 2017 records that customers and contracts required the company to hold ISO 27001, and that “due to the 2017 cybersecurity incident, certain of our ISO certifications have been suspended”. The finding does not fault the certificate or the body that issued it. It describes the certifier as “a firm with substantial relevant expertise” and notes that the audits were conducted annually. What the Commissioner says is narrower and harder to answer: receiving a certificate was treated as monitoring, and it was not monitoring. The evidence that the practices did not match the certificate was in the organisation’s own hands, in documents it already had, and the certificate did not change.
The certificate never covered the thing that broke
“Certified and breached” is a list, and lists invite the wrong conclusion. The more useful question for each case is what the certificate’s scope actually said, and whether the failure happened inside it.
Okta’s ISO 27001 certificate, issued in July 2021 and running to October 2024, scopes the certified management system to the one “supporting Okta’s cloud-based IDaaS platform along with auxiliary Okta product offerings”, at a single San Francisco location whose listed functions include Support. Its January 2022 intrusion ran through a workstation owned and managed by a subprocessor, used by one of the contract engineers that subprocessor supplied to Okta’s own customer support organisation, working in Okta’s own support tools. Was that workstation inside the certified boundary? The certificate cannot say. It names a location and a list of functions. It does not say which suppliers perform those functions, or from where, and the statement of applicability it refers to is not public. Its October 2023 intrusion ran through the support case management system, which sits inside a function the certificate does name, so that one, as far as the certificate lets a reader tell, was inside. Of the two, then, one was inside the stated boundary and one cannot be placed on either side of it from the document a customer is given. Neither is a finding against the certificate, and the certificate says so itself: its printed conditions state that the audit is “not designed to detect or prevent criminal activity” and that certification “should not be construed as a guarantee or assurance that an organization is unsusceptible to information security breaches”. That sentence is printed on the certificate, in the conditions below the scope. It is not printed on the logo.
The two payment card cases show two further mechanisms. The first is time. Target’s Chief Financial Officer told the United States Senate that “as recently as September 2013, our systems were certified as compliant” with the PCI standard. The Senate Commerce Committee’s analysis of the breach records the attack as running through November and December 2013, with the point-of-sale malware installed from mid-November. The second is the sample. Heartland Payment Systems had been assessed compliant repeatedly, including during the period in which the vulnerability that was eventually exploited had been present, and none of those assessments detected it. Its Chief Executive was careful, afterwards, not to blame the standard. He described compliance as a minimum.
So there are three ways to be compliant and breached, and all of them are structural. The state can lapse between assessments, which is drift: the eleven months. The state can hold and the failure can sit outside the boundary the assessment was asked to look at, or somewhere the boundary as written does not place, which is scope. Or the assessment can look inside the boundary, on the right day, and not land on the thing that is wrong, which is the sample. None of the three requires anyone to have lied.
What this is not
It is not a list of organisations that got it wrong. Every one named here held a current certificate or assessment when it was breached. Whether each also met its wider obligations was a question for its regulators, and in Equifax’s case the Commissioner found that it had not. That is a separate finding from the one this post is about, and it does not change it. The auditors and assessors who issued the certificates were examining what the method asks them to examine. If the argument were that these were bad actors, it would be a weak argument, because the pattern would then be about them and not about the method.
The argument is that the method produces a document whose truth is highest on the day it is issued, bounded by a scope the reader rarely reads, and unchanged by anything that happens afterwards. That is true of good organisations and good auditors, which is why it is worth saying.
What stays true after the audit
None of this is an argument against certification. An organisation with a certificate has been examined by someone independent against a standard, and that is worth more than the alternative. It is an argument against reading the certificate as a statement about now, or as a statement about everything.
Frank does not issue certificates and does not replace the assessment. What it publishes for each scope is a coverage assertion: the posture, the named scope it applies to, the date it stops being defensible, and the evidence underneath it. The scope is on the page rather than in an appendix, so the Okta question, whether the system that failed was inside the boundary, is answerable before anything fails. The expiry is on the page, so the Target question, whether the state that was certified in September still held in November, is a date rather than a reconstruction. And the basis is the organisation’s own records, which is where the Canadian Commissioner found the evidence that the practices did not match the certificate.
We will not claim that any of this prevents a breach. It does not, and a product that said so would be manufacturing exactly the confidence this post is about. What it does is narrower. It makes “compliant at the time” a fact you can check on any given day, with its scope and its date attached, rather than a sentence you first hear in a post-incident review.
You can see what your own evidence supports today, and where its boundary sits. Start by seeing where you stand. It will not certify you. It will tell you what your records currently hold up, what they never covered, and the date each claim stops being true.