Methodology

State of compliance

This research is based on the analysis of quantitative data gathered by QSAs from multiple Qualified Security Assessor Company (QSAC) organizations across the world. The dataset for this edition is based on information from five sources, four of them external to Verizon.⁶⁶ These findings are presented globally, with additional comparisons between geographic regions (Americas, EMEA and APAC).

Dataset

Producing a PCI DSS assessment report may involve numerous assessments. In several cases, an assessment report is the product of assessments conducted globally or across a specific region. Individual PCI DSS compliance reports consist of between one and, in some cases, up to 120 or more assessments per report, covering multiple in-scope locations. 

Assessments 

PCI DSS version: PCI DSS v3.2.1 consists of 12 PCI DSS Key Requirements, 79 base requirements, 252 control requirements and 440 test procedures.

In 2020, the compliance status of a total of 77,504 PCI DSS controls validated against PCI DSS v3.2.1 was assessed and compared against 68,992 controls from PCI DSS v3.2.1 assessed in 2019.

Reports: The 2019–2020 comparative analysis is based on an aggregate of 328 PCI DSS compliance validation reports and a combined total of 146,496 controls. 

PCI DSS Report on Compliance (ROC) dataset.

The PSR analysis process

Our overall PSR data collection and analysis process remains intact and unchanged from previous years. All assessment data included in this report was individually reviewed and converted to create a common, anonymous aggregate dataset. The collection method and conversion are the same between contributors. In general, three steps were used to accomplish the dataset:

1. Contributor identification and collection of eligible PCI DSS v3.2.1 assessment reports
2. Full anonymization and conversion of the reports by the contributors into normalized data. All contributors received instruction to omit any information that might identify organizations or individuals involved
3. Secure submission of the anonymized data to the Verizon PSR data science team for aggregated analysis

Data eligibility

For a potential entry (Interim Report on Compliance) to be eligible for the PCI DSS compliance validation corpus, several requirements must be met. The entry must be data from a confirmed PCI DSS validation assessment conducted by a QSA who completed an ROC for an interim validation assessment. In addition to meeting the baseline definition of a draft or Interim Report on Compliance (IROC), the entry is assessed for quality. We then create a subset of compliance report data that passes our quality filter.

In addition to having the level of details necessary to pass the quality filter, the assessment reports must be within the time frame of analysis. For the 2020 dataset, this includes PCI DSS assessments conducted between January 1 and December 31, 2020.

What percentage of total PCI DSS compliance validation assessments that are conducted worldwide each year is covered in the survey? We do not know. We only have access to the data for the validation assessments that were conducted by Verizon and contributing QSACs.

"Anything can be measured. If a thing can be observed in any way at all, it lends itself to some type of measurement method. No matter how ‘fuzzy’ the measurement is, it’s still a measurement if it tells you more than you knew before.”⁶⁷

—Douglas W. Hubbard

Noncommittal disclaimer

We would like to reiterate that we make no claim that the findings of this report are representative of all PCI DSS compliance assessments for all of organizations at all times. Even though the combined records from all our contributors more closely reflect reality than any of them in isolation, this dataset is still a sample. Although we believe many of the findings presented in this report are appropriate for generalization (and our confidence in this grows as we gather more data and compare it to that of other security organizations), bias undoubtedly exists.

The findings are based on aggregated demographic information. While aggregations are made up of individual organizations, individual organizations are not made up of aggregations. It’s not a two-way street. There are limitations to the extent these aggregations can be useful in making decisions. Therefore, when reading the findings of this report, you should not make assumptions about their applicability to individual organizations. Some findings and conclusions require additional context and data to add more value on the individual level.

"Anything that gives us new knowledge gives us an opportunity to be more rational.”⁶⁸

—Herbert A. Simon

"One accurate measurement is worth a thousand expert opinions. Without data, you’re just another person with an opinion.”⁶⁹

—Rear Admiral Grace M. Hopper and W. Edwards Deming, Chicago Analytics Group

⁶⁶ See page 163 of this report for list of PCI DSS data contributors.

⁶⁷ Douglas W. Hubbard, “How to Measure Anything,” Third ed., Wiley, 2014.

⁶⁸ Herbert A. Simon. https://www.brainyquote.com/lists/authors/top-10-herbert-a-simon-quotes.

⁶⁹ Rear Admiral Grace M. Hopper and W. Edwards Deming, Chicago Analytics Group, Mar 30, 2016, http://chicagoanalyticsgroup.com/blog/archives/03-2016.