Data Are Made, Not Found
by danah boyd
Association of Public Data Users
July 29, 2020

How do we know what we know? This question has tickled the brains of philosophers for centuries and birthed scientific inquiry as we know it. I am an empiricist at heart. I want data; I want evidence. I want to believe in claims that can be grounded. I began my career in computer science because I loved the structured way of thinking. The machine will tell you if your code compiles. Watching a system run as intended is satisfying. Structure. Precision. I love it all.

I began my career modeling and visualizing the traces of data humans left behind. I wanted to understand the structure of social networks, long before there was Facebook. So I analyzed email dumps and Usenet records to build models of people's relationships with one another. Two things became quickly apparent: 1) data traces are extremely biased; 2) data traces reveal more than people realize. I spent the next decade working to understand how people conceptualize, experience, and navigate privacy. I've spent the last decade grappling with the limitations of data. 

In my talk today, I'm going to marry those two inquiries with an eye towards the question that is currently keeping me up at night: What makes data legitimate? All of these issues collide on a topic that is pertinent to many of you: the 2020 US census. But before we go there, let's first set a stage. An impressionist painting if you will. A bit of a dance. 

Part One: Agnotology. 

Epistemology is the study of how we know what we know. When we as scientists disagree over methodology, we often talk past one another. Methodological debates function as a proxy for broader epistemic ones. What may appear to be an externality from one vantage point is the explanation from another. I often return to this quote by Erving Goffman. Screen grab this for posterity. 

Yet, even as we as scientists argue over methodology, we also know that there are limits to even our favorite methodological tools. There are times and places where we see what we want to see, where we read into data even though somewhere deep down, we know that's problematic. 

When I last taught an intro to data science course, I asked my students on the first day to load the NYPD "stop and frisk" data set into their environment and tell me the average age of someone who was stopped, purportedly to show that they could work with the data. One by one, they yelled out 27. I asked them if it was right and they blinked at me in confusion. I asked them to explain what they found and they started projecting their own experiences onto the data. They thought young people were stopped; maybe a lot of older homeless people skew the data. I then asked them to run a distribution and they gasped as they realized that most common ages for being stopped were 0 and 99. The data were flawed and they never stopped to check; they just projected their assumptions. These were students on the first day, but all who work with data encounter this situation at one point or another. We think we understand, but we didn't ask the right question. 

Data can be flawed for any number of reasons. But what scares me the most is when it's strategically flawed to achieve a particular agenda. Given that epistemology is the study of knowledge, Robert Proctor and Iain Boal coined the term "agnotology" in 1995 in order to encourage the study of ignorance. In Proctor's book co-edited with Londa Schiebinger, they dove into this concept, arguing that there are many types of ignorance. There's ignorance that comes from not yet knowing, which is the domain of science. There's ignorance that comes from forgotten knowledge, such as that which is produced through the erasures of people through colonial oppression. But, there's a third type of ignorance: that which is strategically produced.  Ignorance that is manufactured for political, financial, or ideological reasons. We live in a society shaped by agnotology.

The Agnotology book includes a chapter by David Michaels on how the tobacco industry funded studies to seed doubt into the emerging consensus surrounding lung cancer. A chapter from Naomi Oreskes and Erik Conway would form the foundation of their popular book/film "Merchants of Doubt," where they lay out evidence showing how our inability to reach consensus on climate change stems from strategically manufactured ignorance. Right now, we are living through another case study of this with COVID-19. It's not simply that we don't have enough tests to get a good baseline number; it's that there are politically interested parties invested in ensuring that we don't know. We're seeing death data be manipulated. We're seeing states selectively choosing what data to report. We are seeing data used to manufacture ignorance for political and economic reasons. This is why agnotology, or the study of ignorance, is so importance. 

Part Two: Data Voids

In 1998, while still graduate students at Stanford, Sergey Brin and Larry Page wrote a paper describing a prototype search engine called Google. In describing their design goals, they explain the need to improve search quality, highlighting how existing search engines surface up too many “junk results.” They saw PageRank, their algorithm, as the solution. They believed that they could get true and meaningful signal from the data out in the wild. Of course, once Google started operating in the wild, the search engine optimization (SEO) industry simply evolved to game the new ranking structures. Like Altavista and Infoseek before, Google had to respond to these efforts and evolve its approach accordingly. The game of cat-and-mouse continues to this day, even though companies like Google have erected more and more barriers. The motivation for most SEO attacks is obvious - there is a lot of money to be made when your site appears at the top of a popular search query. 

In 2015 and 2016, I spent a lot of time embedded inside darker corners of the internet trying to understand the dynamics of conspiracy theorists and hate groups. One thing caught my attention: they were all interested in manipulating tech company algorithms to help amplify their toxic messages. Michael Golebiewski and I became interested in one type of attack that focused on a type of search engine vulnerability that we call "data voids." 

Search engines require a lot of data to be useful. When people search for common terms, there are millions of results that search engines rank to provide users with results. SEO on those pages is hard. It's a lot easier to SEO results that people rarely search. Or to SEO terms associated with breaking news stories. In our work on "data voids," we describe many ways that this vulnerability is exploited, but let me describe one way in which this works. 

Shortly after the Sandy Hook school shooting in 2012, a group of conspiracy theorists started hypothesizing that people who appeared on TV as witnesses or family members of dead children were actually actors working for the Obama Administration to build a case to challenge the 2nd Amendment. They started describing them as "crisis actors." This term had very little web content associated with it at the time, so conspiracy theorists went about building a web of content. They produced images showing that the same actors appeared after every shooting, worked on identifying people and associating them with the deep state. They built a harassment network that targeted parents of deceased kids. One father died by suicide because he couldn't take the harassment. Other parents sued one of the major conspiracy theorists who regularly propagated this message. 

In a different world of media, this conspiracy theory would've been esoteric. But these media manipulators built a web of content online with significant SEO. And then, after each subsequent mass shooting, they targeted news media with this term, trying to get them to pick up this term. In 2018, they were successful. After the Parkland school shooting, Anderson Cooper asked David Hogg if he was a crisis actor. This sent thousands of people searching for the term and walking straight into a web of conspiracy. 

While we have little quantitative evidence about how effective these campaigns are at increasing the adoption of conspiracy theories, we do have case studies of real harm. The massacre at Emanuel African Methodist Episcopal Church in Charleston was committed by a teenager whose manifesto detailed how his attempt to understand the Trayvon Martin shooting by going to Wikipedia led him to a strategically staged term that, when he searched for it, led him into white nationalist content. He discussed this as an awakening, as having his eyes opened.

This linguistic tactic of coining a term and then building an ecosystem around it is not unique to the web. This is precisely what Republican operative Frank Luntz popularized in the 1990s when he built drumbeats out of terms to get them into the media. You know many of his terms. Climate change. Partial-birth abortion. Death Tax. You also know many of the conspiracy theorist's terms. Deep state. White genocide. Caravan. Incel. Globalist. 

This form of media manipulation works by understanding the algorithm and human psychology and then staging the data in a particular way to create a condition where people feel as though they stumble upon information. This is how many of the COVID-19 conspiracies have spread so rapidly.

Part Three: Data Out of Context

Survey methodologists have spent decades trying to find ways to get people to answer sensitive questions. They've developed countless techniques. One of my favorite - the random response technique - emerged in the 1960s. This technique involves a combination of plausible deniability and statistical accounting. 

Imagine being a researcher in the 1960s wanting to get a sense of how common abortions are. You couldn't expect respondents to be honest and it was impossible to weight respondent bias effectively. Instead, you hand someone a spinner. Or a coin. The respondents are told if they get a heads, tell the truth; if it's tails, lie. They flip the coin or spin the spinner, but they don't show the result of the random number generator to the interviewer. Instead, they use that as an instruction to decide whether or not to tell the truth. This way, the interviewer couldn't know for certain if the response was true or false, but they would still know statistically. And with enough sample, they could get a probabilistic answer. 

This method is beautiful because it ensures that the study gets statistically meaningful data without violating the privacy of the informants. And it does so in order to get better data. The method is designed to take into account both human behavior and human psychology. 

As social media grew in popularity, social scientists started drooling over Facebook and Twitter data. Finally, they could study humans in the wild. Or so they thought. 

Angela Xiao Wu and Harsh Taneja are disturbed by the fetishization of Facebook digital trace data. Examining the way that computational social scientists are using this data and the claims being made, they have concluded that the thirst for large scale data has blinded many scholars to what this data does and doesn't say. They argue that the data says more about Facebook than it says about the people using Facebook. 

Social scientists often treat Facebook data as behavioral data, as strong indicators of attitudes, preferences, tastes, etc. Yet, there's nothing natural about the Facebook environment. When people use Facebook, they are performing for their imagined audience. They are constructing impressions for both Facebook the corporation and their viewers. They want to give off good impressions. Facebook does not contain administrative or behavioral records; it contains performative records. 

Let's make this more concrete. When I was interviewing teenagers, I regularly encountered outright lies on their social media profiles. Some were humorous. Apparently, the population of Christmas Island is huuuuuge and it's far funnier to be 69 than to be 15. Some were intended for status reasons, ways of performing being cool through references to drugs and sex. Others were about privacy. I met young addicts whose profiles focused on worshipping Jesus. And I met teens who talked about all sorts of drugs that they never tried. Their profiles conveyed more about the norms and cultural context in which they are embedded than their personal practices, habits, and interests. 

Growing up in the world of computer science, I naively thought that privacy was about controlling access to information. But what I learned from talking with teenagers for over a decade is that privacy is about controlling a social situation. It's about what Erving Goffman calls impression management. Teens care deeply about privacy. Adults mistake their status performances as being a rejection of privacy. But teens perform to see and be seen, to garner attention and status. But they have numerous expectations about how what they share will be used. In a world where everyone has tools of amplification at their fingertips, teenagers don't expect that they will be able to control the flow of information. Instead, they focus on ensuring that the information that will flow will create a certain impression. 

Teenagers invoke the concept of privacy in ways that are confounding to a computer scientist. For example, they would talk about how Facebook violated their privacy by re-amplifying earlier posts. Those posts were previously visible in a technical sense, but when Facebook took them out of context, it was experienced as a privacy breach. 

This is why computational social science is so tricky. The data are methodologically flawed because they weren't produced with researchers' uses in mind. And when the data are used as such, people experience and articulate a violation of privacy. As one teenager explained to me, "Just because I posted it online doesn't mean you have the right to use it." 

Part Four: The Census and its Privacy Problems

So let's talk about the census. I don't need to tell you why the census is important, but we do need to talk about the disconnect between data users and the public whose data we all need to do our work. We live in a country where many people do not trust governmental data collectors. This is not new. We have laws that require census participation because some people would opt-out. Getting people to participate is hard work and our friends at the Census Bureau have devoted their lives to developing new techniques to increase participation and ensure the best data possible. And still, they face an uphill battle, especially given a fraught political climate in which getting the best data possible is not considered universally desirable. 

The Census Bureau is accustomed to encountering difficulties in doing its job. There have been snowstorms and hurricanes. Boats filled with census records have sank; buildings full of records have caught fire. Census lore is filled with hardship -- and a mantra focused on the importance of carrying on. But between the pandemic and an Administration hellbent on scoring political points by undermining the census, it's not an easy time to be a civil servant. 

Meanwhile, a significant segment of the public is scared to interact with the government. They're scared about how the data might be used or abused. And they have good reason to be concerned. People are losing their jobs because of their Facebook data, which is why we're seeing a new wave of what Mikaela Pitcan identified as digital respectability politics, where Black and brown people take to Facebook to perform in a certain fashion so that they can be seen as respectable in a white-dominated society. People have grown accustomed to not telling the whole truth if they're afraid that the data can be used against them. So now imagine how you might feel about filling out the census if you're afraid your landlord might evict you for having too many people at the house. How might you feel about including your mother who overstayed her visa to help you if you're afraid of the government? The biggest barrier to a complete count isn't missing households; it's missing members of households. 

The Census Bureau is bound by Title 13 of the US Code. Part of this law includes strong protections around user data, making it illegal for the Census Bureau to disclose information that could identify someone. This exists so that the Bureau can encourage people to participate, even when those people are scared to share. Unlike polls and surveys, the Census Bureau must find everyone as part of the census. 

Well, a peculiar thing happened on the way to the 2020 census: the desire for increased access to data and the importance of protecting the confidentiality of census data collided. It was a crash course, inevitable but nonetheless gruesome. 

Mathematicians and computer scientists have been working to break encoded information since World War II. One of the battlegrounds in this arena has been to discern individual records from statistical tabulations. You can think of this as a large Sudoku puzzle. How can you figure out what characteristics uniquely represent each individual in a dataset to produce the statistical tabulations that you have?  For small datasets, this reconstruction puzzle was solved decades ago, which is why we play Sudoku for fun. But the larger the dataset, the more complex the puzzle, and the more combinations you need to try. Unfortunately for confidentiality, computational power has grown exponentially in recent decades, while mathematical advancements have resulted in more efficient algorithms. Today, reconstructing a dataset like the census when you have enough data points is computationally trivial. 

Reconstructed data doesn't provide the names of individuals because that data isn't in the original dataset. But it's still identifiable. This is why the Census Bureau only ever produced a Public Use Microdata Sample. Title 13 prevented them from publishing microdata about small area geographies or where people could be uniquely identified from their neighbors, even if the data didn't contain the names. 

Of course, linkage attacks are far scarier for those who care about individual privacy. Thanks to the tremendous increase in public and private sector data, linking reconstructed records to other records is becoming easier and easier. In other words, you don't have to be someone's neighbor to be able to find someone in a dataset. You could be an insurance loan officer seeking to get people off your rolls, a city official looking to crack down on kids not living where they've registered for school, or a violent offender seeking to track down your ex-wife. This is no longer an abstract far-off future. This is contemporary actuarial science, predictive policing, and public sector algorithmic work. It's what domestic violence and death threats look like in a world of data traces. 

So the Census Bureau is at a crossroads. What should they do? They have a responsibility to protect everyone's data in order to protect the data of those who are most vulnerable. That's not just a values-driven goal or a legal commitment (although it is both); it's also mission-critical to produce data that we can use. They need all respondents to be honest. And even when someone is too scared to share, they need to feel confident that the data is protected before they pull on administrative records, such as those collected by the IRS.

The Census Bureau's approach to disclosure avoidance has historically involved a lot of table suppression. They simply don't publish data. But, over the decades, there has been pressure to publish more data and at lower levels of geography. When I started this research, I had no idea that nation-wide block-level data wasn't available until 1990!  The decision to increase the nuance of race data in 2000 is fantastically important for society, but it also makes people a lot more identifiable. Many of the methodological and technical moves that the Census Bureau has made in the last decade make the data of higher quality, but also more readily identifiable. 

Higher quality data means more risk of reidentification. Greater computational power and more data points means reconstruction is much easier. One way to manage privacy in this would be large-scale data suppression, but that's unlikely to please anyone. 

Another approach might be to really invest in swapping. This involves taking households and swapping them across blocks to areas where they might be less identifiable. This is a technique the Bureau has previously used, although we have never known how much. What we do know is that the swapping protections from 2010 did not prevent reconstruction of the entire dataset. Census Bureau researchers were able to re-reidentify at least 50 million people using the easiest to access, least sketchy commercial data out there. This can be replicated by external actors too. Don't get me started on what you can do if you use the underground market of sketchy commercial data. Like the stolen Equifax and Target data. We don't know how much swapping would be necessary to actually protect the data, but what we do know is that every swap that is made biases the data by making blocks more homogeneous. 

And this is how the Census Bureau ended up beginning its explorations into differential privacy back in 2008. For those who are mathematically minded, differential privacy is statistical beauty. It produces a type of noise that is evenly distributed, governable through a variable, and statistically easy to accommodate. In other words, you can know the range of noise you're dealing with and adjust your analyses accordingly.

For those who want a picture in their head, imagine a piece of Pointillist art. The beauty of impressionist paintings is that you can stand back and see the full picture. Up close, you can see each pixel in gory detail. What noise injection controlled by differential privacy means is that you would alter each pixel by a tiny amount such that the portrait stays the same. But that each blue dot over there would become a slightly different blue color. You couldn't tell at the macro-level unless the range of edits was great, but at the microlevel, you would protect the true color of every pixel. 

Differential privacy is a scientific marvel. Unfortunately, the Census Bureau faces constraints that prevent them from producing data that is simply protected by noise controlled by differential privacy. They must produce data in a specific format. And that format means that there are to be no negative numbers or partial people in any cell across the data. This may seem like a reasonable requirement, but imagine if there are only 5 people in the state of Colorado who are Black and Asian and AIAN and NHPI. That combination of race data is rare, but the Census Bureau's format produces a count of how many people matching that combination exists in every block. If you wanted to protect the confidentiality of those people, you need to spread them out across other blocks. If you don't want to end up with 500 of them, the noise injection work gives some blocks a negative number of them. Those 5 people exist statistically, but finding them on a given block is impossible. 

The Census Bureau doesn't feel like it can produce data filled with negative numbers. So the disclosure avoidance system does another pass, a pass known as post-processing. It contorts the data back into the accepted format. Post-processing starts by making the data non-negative and consistent; this produces fractional results, which is then managed through a rounding algorithm. Post-processing introduces a lot of new noise into the data. And like swapping, this noise is biased. It disproportionately affects small area geographies and smaller populations. If you're hearing complaints about data quality due to differential privacy, odds are you're hearing about post-processing. It's equivalent to creating glitches in the middle of your Pointillist painting. 

Part Five: The Political Contestation

Differential privacy came up Monday and it will be raised again tomorrow. Data users are reasonably concerned about what this technique means for the quality of census data. What's most troubling is: how will we truly know if the data are fit for use? 

To the best that I can tell, the history of the census over the last 100 years is filled with people accepting the data as fit for use because they trusted the professionals who conducted the operation. Even as data users and stakeholders started recognizing that there were serious undercounts and pushed the Census Bureau to do better, they accepted the data that came in and then spent the next couple of years trying to figure out how to do better next time. Programs like the Population Estimates Challenge were designed not in an adversarial way, but to invite data geeks to come in and try to help improve the counts in local areas. Participating in this program had real material implications - agreed upon additions resulted in an increase in federal funding. 

But when it comes to the 2020 census, trust is not how I would describe the relationship between the Census Bureau and the ecosystem of stakeholders. We could probably have a rich discussion of how we got to this point, but here we are. And it's bad. The real question for me is: what will it take for the 2020 data to be deemed legitimate? And will it? 

Whenever I struggle to make sense of a contemporary puzzle, I turn to the past. So I want to tell you a story about the 1920 census, a story that my collaborator Dan Bouk pieced together. 

Many people here probably know that Congress failed to re-apportion after the 1920 census data were delivered. Margo Anderson did a fantastic job tracking down pieces of this. With the incisive eyes of a historian, it's clear that most Congressional antagonists had political, racist, and nativist motivations. Throughout the 1910s, people migrated from rural parts of the country towards more urban parts. Immigrants came into the country and moved primarily to cities. But these are not the only stories that they told on the floor of the House. They talked about operational issues. The census shouldn't have taken place on January 1. It shouldn't have taken that long. Etc. But one thing caught Dan's eye. The major hook that was used to justify why apportionment shouldn't happen was an algorithm. 

Up until 1920, it was common to increase the House of Representatives every decade when the new data came in, but there was pressure after 1910 to cap the size of the House. This would make reapportionment a zero-sum game. For a state to gain a representative, another state would need to lose a representative. In response to this, Walter Willcox started arguing that Congress should a priori determine the algorithm that Congress used to re-apportion itself. Up until then, the ultimate decision of how to apportion seats - and the methods used - fell to negotiations in Congress. Willcox thought this was too arbitrary. The last few seats were a political fight, shaped less by formula than by argument - and often being resolved by increasing the size of the House. To prevent this, Willcox put forward a method that he thought should be the standard going forward.

Joseph Hill was the chief statistician at the Census Bureau. He didn't agree with Willcox's method but he did agree that choosing an algorithm in advance was a good idea and he was happy to defer to Willcox's method if that's what Congress should choose. Congress was pretty much set to formalize Willcox's method when, on the even of Congress's debate, a Harvard mathematician named Edward Huntington published a letter to the editor of the New York Times challenging Willcox and his method. He argued that if Congress did not accept his method (which was a slight improvement on Hill's method by the way), they would be committing a grave injustice. This letter and subsequent conversations with Huntington sent Congress into a tizzy. 

As you may remember from our earlier discussion on agnotology, nothing is more powerful than the seeding of doubt. As we would later work out, there is no "right" answer to creating an apportionment algorithm. Each approach has biases. Huntington and Hill's method was biased towards small states; Willcox's method was biased towards large states. But Congress didn't try to work out this technical detail. Instead, they decided to use the lack of consensus from the experts as an excuse to avoid re-apportioning Congress. It benefited those who were going to lose seats using either method to just keep postponing. And so they did. And, in the process, they helped delegitimize the data as well. 

It wasn't until 1929, when Arthur Vandenberg from Michigan pleaded with Huntington to stop this fight that it became possible to move forward with the Reapportionment Act of 1929. Vanderberg was frustrated because his state had been punished over that decade. As he wrote to Huntington, “I am sure you will agree with me that the paramount need is to get re-apportionment—–regardless of the method used…You can help incalculably in getting this enabling legislation. The greatest possible help would be your agreement with me upon this amended form which undertakes to leave the pending legislation entirely above and beyond any argument over methods.”  

Right now, we're in a political fight about methods. And that methods fight is happening at the same time that political actors are working to delegitimize the Census, gaslight the public into believing that the Constitution aims to exclude people, and scare people into not participating in the first place. This is a classic story of agnotology and data manipulation. And if we're not careful, our scholarly debates and disagreements about disclosure avoidance methods are going to be used as a hook in a bigger political battle. 

Personally, I love differential privacy. I think that statistically managed noise injection provides people with well-deserved privacy protections that allow the Census Bureau to keep its Title 13 commitments and ensures that this data is not used against people in the coming decade. That said, as a scientist, I loathe biased noise injection. This is what make me uncomfortable with swapping and deeply concerned about post-processing. I think that post-processing introduces unacceptable biases, as so many data users have shown. I know the Bureau is working to contain these problems so I'm holding my breath. But I'm worried. With these publication and formatting constraints, some bias is inevitable.  That said, I've been willing to stomach post-processing because most data users have unequivocally said that they need the data in that format, full stop. No negative numbers. No partial people. A true microdata file. That's a trade-off: format for bias. Many folks are upset by the trade-off conversation that we're having, feeling like confidentiality vs. utility is a false trade-off. I share this frustration, which is why I'm frustrated by the format vs. utility trade-off. But I also feel like we're collectively living in a rock-and-hard place scenario. I don't know how we get out this. 

There's no returning to a world where reconstruction and reidentification aren't possible. I'd love to live in a world where vulnerable members of our society wouldn't need to be afraid if their data were made public, but I can't play ostrich on that one. The need for confidentiality is crystal clear to me. What's not clear is what is an acceptable approach through this. From a pure technical place, I understand that noise injection managed by differential privacy without post-processing is the best way to get statistically usable data. But this isn't a technical debate, just as the arguments around the best apportionment algorithms weren't simply a technical debate. Data quality is one factor. Another factor is data format. Another factor is data users' impressions. 

We live in a world where people are becoming increasingly skilled at manipulating data and perverting data to enable ignorance. In a world where more and more data is publicly available, people are developing new techniques to achieve and undermine privacy. We also live in a world where partisan interests are dismantling the legitimacy of scientific work. This presents a myriad of methodological and epistemic challenges for us, but above all it presents an existential one. How do we ensure that the collection, production, analysis, and dissemination of public data remains legitimate? How do we ensure that the public data effort isn't dismantled? Because above all else, I don't want to see the delegitimization of federal statistics. And that's the precipice that I see up ahead.

Thank you