M
Ten minutes into our SOC 2 audit, the auditor asked where our test data came from, and I told her it was anonymized production data, in the tone of a man who has just said a correct thing. She wrote it down, then asked how it had been anonymized, and somewhere in the pause that followed I worked out that I did not really know.
What we did in practice was swap the name and email columns for generated strings, hash the account numbers, and ship everything else exactly as it came out of the nightly dump. I had been calling that anonymization for about eighteen months. It took another fortnight and a lot of reading before I understood that the correct word was a different one, and that the gap between the two words was most of the argument I had just lost.
That conversation happens long before anyone opens a seed script, and having it on your own schedule is considerably more comfortable than having it across a table.
Whether a dataset still counts as personal data turns on which of the two words applies to it, which is why the distinction survives being dismissed as hair-splitting.
The UK Information Commissioner's Office draws the line in its anonymisation guidance. Pseudonymisation swaps whatever directly identifies somebody for something else, a reference number standing in for a name, and the person behind the record stays reachable by anyone holding the other half. Anonymisation means nobody can get back to them at all, which is why only the second of the two takes a dataset out of the regime.
Twelve years ago the Article 29 Working Party had already put it more bluntly in Opinion 05/2014, the doctrine the ICO's guidance still rests on, describing pseudonymisation as something that merely reduces the linkability of a dataset to the original identity, and calling it a useful security measure rather than a method of anonymisation.
Apply that to a nightly dump with the names taken out and the timestamps, amounts and postcodes left in, and anybody holding a second dataset has enough to start matching rows against it. The ICO tests this with what it calls a motivated intruder, a reasonably competent person who wants to re-identify somebody and has public records and a search engine to work with, and it asks two things about the result. Can you single out an individual? Can you link a record to something held elsewhere?
Our staging rows answered yes to both, and had done since the first dump. We had built pseudonymized personal data, given it a name that implied we had done something more, and carried every obligation attaching to it in production straight into an environment we thought of as scratch space.
Card data has a requirement written for exactly this situation, and it is unusually blunt about it.
PCI DSS v4.0.1, Requirement 6.5.5: "Live PANs are not used in pre-production environments, except where those environments are included in the CDE and protected in accordance with all applicable PCI DSS requirements."
The exception clause is where the whole thing lives. Either the environment holds no live primary account numbers at all, or it belongs to your cardholder data environment and inherits every control that comes with that, down to the segmentation and the quarterly scanning, and no third arrangement is on offer. A box somebody spun up on a Thursday to reproduce a customer's bug does not get a lighter version of the standard on account of what it is called.
Version 4.0 widened the wording too. Requirement 6.4.3 in the previous revision talked about testing and development environments, and the summary of changes records the move to the phrase pre-production, which quietly swept in the preview environment CI spins up per pull request and the demo instance the sales team keeps warm. Neither of those is development by any honest reading, and both are covered now.
Most teams I have talked to found out where they stood through an inventory rather than an audit. Somebody finally sat down and listed what was actually in each environment, and at least one of the answers surprised the room.
SOC 2 says less about this than PCI does, and teams tend to read the silence as permission.
The control is CC8.1, and the two letters in front of it carry the weight. Common Criteria applies to every SOC 2 report ever issued, because Security is the one mandatory trust services category, so declining to scope in Confidentiality gets nobody out of it. Its point of focus reaches confidential information handled during design, development, testing and change.
Nothing in the criteria spells out masking, and the difference between what the standard says and what auditors do with it is worth being careful about. In practice they ask where test data comes from, and an answer involving production leads to a second round of questions about who can reach it and how long it stays there. Practitioners report being asked for screenshots of the dataset itself, or of whatever tool produces it. That is auditor practice as SOC 2 advisory firms consistently describe it rather than a sentence anybody can quote at you from the AICPA criteria, and walking in knowing which is which is worth something.
Either way you arrive where PCI put you. A team treating its staging database as nobody's business because only Security got scoped has misread which criteria are the common ones.
Regulatory argument only carries so far, so here are two occasions when the thing it warns about actually happened.
In May 2016 the UK baby-goods retailer Kiddicare told its customers that around 794,000 records had been taken, and those records had not come from the live store. They came from a test website standing since the previous November and populated with real names, addresses and phone numbers. The company learned about it because customers started receiving phishing texts from a domain dressed up as a Kiddicare subsidiary, and it reported itself to the ICO. Years later New Zealand's Privacy Commissioner wrote the case up under a heading that saves everyone the analysis, testing systems with real data leads to breach.
Australia's Optus lost a great deal more, and lost it differently. Somewhere around ten million current and former customers, roughly a third of the country, had their names, dates of birth and contact details taken in September 2022, and for about 2.1 million of them the haul included passport or licence numbers. Independent technical write-ups put the cause at an API endpoint with sequential customer identifiers and no authentication in front of it, reachable from the internet, on what several of those accounts describe as a test network. Optus disputed that description and called the incident a sophisticated attack. I am not in a position to settle which account is right, so take the scale as established and the cause as contested.
Neither case turned on anything sophisticated. In both of them the security budget was pointed at the front door while real records sat in a room nobody had thought to defend, because everybody involved believed that room was either empty or full of something that did not count.
This is the point where an article usually reaches for a regulator's fine, and I have to report that I could not find one. Going through ICO monetary penalty notices and CNIL sanctions looking for a penalty whose stated cause was real data in a test environment, every action I read traced back to a production system or to general security failures. That reads to me as a fact about which door tends to get kicked first rather than as evidence the exposure is cheap.
Instead of copying production and stripping out what is dangerous, you can build the dataset from the shape of the database.
A schema already carries most of what a test environment needs, down to which columns are required and which rows have to point at rows that exist. Generate against that description and the result behaves like your database at whatever volume you ask for, without a customer ever having stood behind a single row of it. Neosync does this, and so does Seedfast, which I work on.
The compliance consequence is the part worth noticing, because there is nothing to pseudonymize when nothing was ever real, and the motivated intruder has nobody to single out.
It also changes the shape of the bad day when a lower environment leaks, and they do leak. An exposed staging database full of generated rows costs an incident report and some embarrassment, where the same database full of real ones costs a letter to every customer in it.
Everything above should be read with one limit attached, and I would rather state it than let it be inferred.
Taking real records out of lower environments closes one exposure and leaves the rest of the work exactly where it was. It produces no SOC 2 report, and it has no opinion whatsoever about who can reach your systems or how you documented your scope.
There are also cases where copying production is the right call, and I would not argue against them. A fraud model has to be validated against the distributions it will actually meet, and some bugs only ever appear in one customer's data. What that workflow should look like is a copy with a short life and a record of who touched it, which somebody then actually deletes. Deliberate copies are rarely the ones that hurt, because somebody is watching them; the damage tends to come from a copy made for a good reason on a Thursday afternoon and still sitting there in November.
None of that takes long, and the reason it rarely happens is that a staging database has no owner in most companies, so nobody schedules the afternoon it would take to go and look at something that has not yet caused a problem.
What we did in practice was swap the name and email columns for generated strings, hash the account numbers, and ship everything else exactly as it came out of the nightly dump. I had been calling that anonymization for about eighteen months. It took another fortnight and a lot of reading before I understood that the correct word was a different one, and that the gap between the two words was most of the argument I had just lost.
That conversation happens long before anyone opens a seed script, and having it on your own schedule is considerably more comfortable than having it across a table.
"Anonymized" and "pseudonymized" are different words for a reason
Whether a dataset still counts as personal data turns on which of the two words applies to it, which is why the distinction survives being dismissed as hair-splitting.
The UK Information Commissioner's Office draws the line in its anonymisation guidance. Pseudonymisation swaps whatever directly identifies somebody for something else, a reference number standing in for a name, and the person behind the record stays reachable by anyone holding the other half. Anonymisation means nobody can get back to them at all, which is why only the second of the two takes a dataset out of the regime.
Twelve years ago the Article 29 Working Party had already put it more bluntly in Opinion 05/2014, the doctrine the ICO's guidance still rests on, describing pseudonymisation as something that merely reduces the linkability of a dataset to the original identity, and calling it a useful security measure rather than a method of anonymisation.
Apply that to a nightly dump with the names taken out and the timestamps, amounts and postcodes left in, and anybody holding a second dataset has enough to start matching rows against it. The ICO tests this with what it calls a motivated intruder, a reasonably competent person who wants to re-identify somebody and has public records and a search engine to work with, and it asks two things about the result. Can you single out an individual? Can you link a record to something held elsewhere?
Our staging rows answered yes to both, and had done since the first dump. We had built pseudonymized personal data, given it a name that implied we had done something more, and carried every obligation attaching to it in production straight into an environment we thought of as scratch space.
PCI DSS has no staging exemption
Card data has a requirement written for exactly this situation, and it is unusually blunt about it.
PCI DSS v4.0.1, Requirement 6.5.5: "Live PANs are not used in pre-production environments, except where those environments are included in the CDE and protected in accordance with all applicable PCI DSS requirements."
The exception clause is where the whole thing lives. Either the environment holds no live primary account numbers at all, or it belongs to your cardholder data environment and inherits every control that comes with that, down to the segmentation and the quarterly scanning, and no third arrangement is on offer. A box somebody spun up on a Thursday to reproduce a customer's bug does not get a lighter version of the standard on account of what it is called.
Version 4.0 widened the wording too. Requirement 6.4.3 in the previous revision talked about testing and development environments, and the summary of changes records the move to the phrase pre-production, which quietly swept in the preview environment CI spins up per pull request and the demo instance the sales team keeps warm. Neither of those is development by any honest reading, and both are covered now.
Most teams I have talked to found out where they stood through an inventory rather than an audit. Somebody finally sat down and listed what was actually in each environment, and at least one of the answers surprised the room.
SOC 2 asks the same thing from another angle
SOC 2 says less about this than PCI does, and teams tend to read the silence as permission.
The control is CC8.1, and the two letters in front of it carry the weight. Common Criteria applies to every SOC 2 report ever issued, because Security is the one mandatory trust services category, so declining to scope in Confidentiality gets nobody out of it. Its point of focus reaches confidential information handled during design, development, testing and change.
Nothing in the criteria spells out masking, and the difference between what the standard says and what auditors do with it is worth being careful about. In practice they ask where test data comes from, and an answer involving production leads to a second round of questions about who can reach it and how long it stays there. Practitioners report being asked for screenshots of the dataset itself, or of whatever tool produces it. That is auditor practice as SOC 2 advisory firms consistently describe it rather than a sentence anybody can quote at you from the AICPA criteria, and walking in knowing which is which is worth something.
Either way you arrive where PCI put you. A team treating its staging database as nobody's business because only Security got scoped has misread which criteria are the common ones.
Two breaches that started in the room nobody was watching
Regulatory argument only carries so far, so here are two occasions when the thing it warns about actually happened.
In May 2016 the UK baby-goods retailer Kiddicare told its customers that around 794,000 records had been taken, and those records had not come from the live store. They came from a test website standing since the previous November and populated with real names, addresses and phone numbers. The company learned about it because customers started receiving phishing texts from a domain dressed up as a Kiddicare subsidiary, and it reported itself to the ICO. Years later New Zealand's Privacy Commissioner wrote the case up under a heading that saves everyone the analysis, testing systems with real data leads to breach.
Australia's Optus lost a great deal more, and lost it differently. Somewhere around ten million current and former customers, roughly a third of the country, had their names, dates of birth and contact details taken in September 2022, and for about 2.1 million of them the haul included passport or licence numbers. Independent technical write-ups put the cause at an API endpoint with sequential customer identifiers and no authentication in front of it, reachable from the internet, on what several of those accounts describe as a test network. Optus disputed that description and called the incident a sophisticated attack. I am not in a position to settle which account is right, so take the scale as established and the cause as contested.
Neither case turned on anything sophisticated. In both of them the security budget was pointed at the front door while real records sat in a room nobody had thought to defend, because everybody involved believed that room was either empty or full of something that did not count.
This is the point where an article usually reaches for a regulator's fine, and I have to report that I could not find one. Going through ICO monetary penalty notices and CNIL sanctions looking for a penalty whose stated cause was real data in a test environment, every action I read traced back to a production system or to general security failures. That reads to me as a fact about which door tends to get kicked first rather than as evidence the exposure is cheap.
The other way round
Instead of copying production and stripping out what is dangerous, you can build the dataset from the shape of the database.
A schema already carries most of what a test environment needs, down to which columns are required and which rows have to point at rows that exist. Generate against that description and the result behaves like your database at whatever volume you ask for, without a customer ever having stood behind a single row of it. Neosync does this, and so does Seedfast, which I work on.
The compliance consequence is the part worth noticing, because there is nothing to pseudonymize when nothing was ever real, and the motivated intruder has nobody to single out.
It also changes the shape of the bad day when a lower environment leaks, and they do leak. An exposed staging database full of generated rows costs an incident report and some embarrassment, where the same database full of real ones costs a letter to every customer in it.
Where this stops helping
Everything above should be read with one limit attached, and I would rather state it than let it be inferred.
Taking real records out of lower environments closes one exposure and leaves the rest of the work exactly where it was. It produces no SOC 2 report, and it has no opinion whatsoever about who can reach your systems or how you documented your scope.
There are also cases where copying production is the right call, and I would not argue against them. A fraud model has to be validated against the distributions it will actually meet, and some bugs only ever appear in one customer's data. What that workflow should look like is a copy with a short life and a record of who touched it, which somebody then actually deletes. Deliberate copies are rarely the ones that hurt, because somebody is watching them; the damage tends to come from a copy made for a good reason on a Thursday afternoon and still sitting there in November.
Five things worth checking while nobody is watching
- Does any pre-production environment ever hold a live PAN? PCI DSS offers two arrangements here, and the status quo is usually neither of them.
- Check whether what you call anonymized data is actually pseudonymized. A record that can be singled out, or matched against something held elsewhere, is personal data wearing a different label.
- Did your SOC 2 scope stop at Security? CC8.1 sits in the Common Criteria, so it still reaches how data gets handled during development and testing.
- Go and look inside staging. The policy describing what should be in there is not evidence of what is in there.
- Put the access list for your lower environments next to the production one. In most companies, it is the longer of the two.
None of that takes long, and the reason it rarely happens is that a staging database has no owner in most companies, so nobody schedules the afternoon it would take to go and look at something that has not yet caused a problem.