The word “information” figures into many of the phrases we use to explain the present and the role of computing technologies in that present, such as information technology and information age. But what is information, and what makes it useful? In order to be ethically responsible users and developers of technological systems that deal with information, we need to understand what we’re working with, and how circulating, creating, and using technological systems can help or hurt individuals and societies.

We live in an age when computers and technology have vastly increased our ability to collect, store, process, and communicate information. These changes have, in turn, increased the amount of information that we encounter on a daily basis, both actively, when we are looking for it, and passively, when it is put in front of us unasked.

The massive surge in available information has been interpreted by many in the tech world as evidence that human beings are getting smarter, or that they have the potential to become smarter. But is information enough to make us smarter? Access to information is not the same as the ability to use that information effectively. It can be difficult to know how to interpret or make suitable use of a particular item of information. And sometimes the sheer volume of available information can make it hard to find the relevant item. For example, in the months preceding the September 11, 2001, terrorist attacks, many security reports issued by American intelligence agencies reported the likelihood of terrorists using airplanes as weapons (National Commission on Terrorist Attacks upon the United States 2004), but these bits of information were lost in the deluge of other reports. More information can be as much a problem as a solution.

Computers are new, but most of the effects they have had on human culture are not. There have been earlier technologies such as writing that have similarly expanded our capacity to store information, some of them as radical and socially transformative as the introduction of computers. When movable type was invented in Germany in the mid-fifteenth century—enabling inexpensive, high-quality printing on a scale previously unknown in Europe—the entire cultural landscape of that continent was transformed as the number of available books and the number of copies in circulation increased exponentially (Briggs and Burke 2002, 12–13; Sohn 1959, 103). The influx of available information was not well received by those who already had access to that information; early responses about the printing press include complaints about the “confused and irritating multitude of books” and a flood of “so many books that we do not even have time to read the titles” (Gardner 2012).

History teaches us again and again that access to ideas can be revolutionary. But the phrase most American school kids are raised on isn’t “information is power;” it’s “knowledge is power.” Knowledge is a more complicated construct than information. How is information related to knowledge? Can the presence of new ways of processing, storing, and communicating data change or expand our capacity for knowledge? Will they help us become wise, better able to make sense of life and its challenges?

In the historical example above of the printing press, many people worried that new forms of information storage meant that people would know less: that access to external information storage would harm humans’ capacity for information storage and understanding. Contemporary research suggests that these societies did indeed experience a shift in how memory worked. Poets and performers in contemporary oral cultures, such as in Papua New Guinea and Northern Australia, have demonstrated mental retentive abilities far beyond those in literate cultures (Bakker 2008; Minchin 2008; Rubin 1997), and scholars of both ancient literature and cognitive psychology now take it for granted that the poets and storytellers who preserved ancient epics such as the Iliad and the Ramayana were able to retain and reproduce those stories in a manner comparable to the poets of contemporary oral cultures.

In addition to worrying about new technology’s quantitative impact on memory, ancient Greek thinkers were also concerned about its effects on the quality of human learning. In the Phaedrus, Socrates argues (by way of a “retold,” likely fabricated Egyptian myth) that writing actually damages a person’s ability to learn, enumerating several ways in which humans will become less adept at processing information, and therefore less wise, once they come to rely on this new external storage technology:

It will introduce forgetfulness into the soul of those who learn it: they will not practice using their memory because they will put their trust in writing, which is external and depends on signs that belong to others … the appearance of wisdom, not … its reality. [Writing] will enable them to hear many things without being properly taught, and they will imagine that they have come to know much while for the most part they know nothing. And they will be difficult to get along with, since they will appear to be wise instead of really being so. (Plato 1995, 275a–275b)

It is not hard to imagine these same accusations being leveled at the internet. Were Socrates’s concerns legitimate? Does information-sharing technology have the paradoxical effect of creating citizens who are shallowly informed, or even hostile toward those with greater expertise? Or have humans been able to use their expanded ability to engage with cultural knowledge to think more broadly and deeply about the world and to question old assumptions in productive ways? As we will find when we go further in this chapter, developments in both directions can, and do, occur as a result of progress in technology, just as they have occurred in response to prior shifts in the technology of information or knowledge.

Just as recent advances in information and communication technologies resemble the advances brought about by the introduction of writing and the printing press, we are confronting many of the same risks and concerns raised by those earlier technological advances. As we will see in this chapter, many of the important ethical issues in technological systems that deal with information require us to reckon seriously with epistemology, that is, the study of how we know the things that we know. As people who work in information technology, it is crucial that we understand what we are working with and how it impacts the world, so that we can use it to enhance the quality of human life and society rather than damage it.

How is information related to knowledge? How are each of these things related to data? And how do agents—human or computational—put these resources to work in trying to identify the best course of action, ethically speaking?

Data, information, and knowledge are all potential ingredients in ethical decision making. Any time an agent has to make a decision, they cannot consider their options or evaluate what is at stake in their response without making recourse to some of the data, information, and knowledge available to them. But in the age of the internet, with vast reserves of content easily and rapidly accessible, how does one decide which information to assimilate, which knowledge bases to consult, and how much is enough?

In an information- and knowledge-rich culture, every decision, large or small, is tied up in larger, systemic-level choices about how to manage the knowledge and information that is available. It is a truism that computers and modern technological systems can vastly expand our capacity to manage, organize, and even synthesize information. Whether or not they can expand our capacity to manage knowledge, however, is a separate question, and a much more difficult one. How we answer it will rely, in part, on how we define information and knowledge and understand the relationship between them.

Therefore, when we discuss ethical issues in managing knowledge, we need to clarify what we mean when we use terms like “knowledge,” “information,” and “data.” These terms might seem familiar, but it is not always clear how, for instance, knowledge differs from information, or what counts as data.

In the last section, we looked at one key way in which humans manage knowledge differently than computers do: we saw that the human mind uses the things it already knows and believes as the basis for integrating new things. In this section, we examine another crucial but underrecognized difference between humans and computers: human beings’ capacity for storing knowledge and using the knowledge that they have stored.

In recent decades, advances in digital technology have drastically increased the volume, velocity, and variety of data. These aspects, called the “three Vs” of big data, mean that we have a larger total volume or amount of data that is generated at an ever-accelerating pace or velocity, and that takes more varied forms including video, email, spoken word, and numerous other formats (Laney 2001). In 2021, it was estimated that humans generated about 500 million tweets, 294 billion emails, and 720,000 hours of YouTube video per day. Attempting to measure all the books, tweets, newspapers, and videos that have ever been created and stored is a difficult task, but it is estimated that about 59 zettabytes, or 59 trillion gigabytes, of data has been created, captured, copied, and consumed, and this will grow to 175 ZB by 2025 (Gantz and Reinsel 2011; Vopson 2021).

There is a widespread belief—held by both the developers of information technology systems and those that use those systems in areas as diverse as marketing, medicine, and manufacturing—that having access to ever larger information sources is, by itself, a kind of magic bullet for problem-solving. This belief rests on the fundamental supposition that deploying a higher volume of information will reliably yield the answers to important problems: all we need to do is collect more data. Indeed, some public intellectuals have gone so far as to argue that advances in technology have made, or will continue to make, human beings fundamentally better or smarter than they were in the past (Pinker 2018). The underlying reasoning of these arguments is that because technology has dramatically enhanced our access to data, information, and knowledge, it will likewise enhance our ability to reason in an effective, just, and timely way. But it is important to remember that these new vast troves of data are not available to everyone equally: not all individuals have access to the data or the permissions necessary to access it; the hardware necessary to attain, visualize or process the data; the knowledge that such data actually exists and where to find it; or the technical know-how to do any or all these things. And having access to data that is stored elsewhere is not the same thing as knowing it, much less understanding it.

Most of us can recall plenty of times that digital information has made our lives easier. Perhaps you have visited a healthcare website like WebMD and looked up some symptoms you were experiencing: for example, What are these strange bumps on the back of my tongue?! Within moments, you probably had a plausible explanation for those symptoms and could make a judgment about the best course of action—whether to get medical attention, treat the symptoms yourself, or simply wait for the problem to go away, if it was a problem at all. Oh, those are lingual tonsils, and everyone has them.

When using such a website, you do not (and cannot) process all the information available to you. The curators of the website have distilled the massive body of medical information to the most important details, and in the course of browsing you narrow down the information even further and consume only the content that you recognize as relevant to your individual needs. But between “clearly relevant” and “irrelevant” lies the troublesome gray area of “possibly relevant,” where the need for selective attention, and at least moderately informed judgment to guide it, can lead to mistakes. When we are predisposed to a particular diagnosis or happen to come across it early in our search, it is easy to convince ourselves that we have one disease when we actually have a different one—or none at all. For instance, “Morgellons disease,” typically the presence of sores on the skin that appear to be made of fibers like cotton or other textiles, was named and made famous by an online community, and it is controversial whether it is a real skin infection or simply a misinterpretation of a common itch reinforced by the accounts of other sufferers on the internet (Vila-Rodriguez and Macewan 2008).

If keeping track of all the available knowledge and information seems impossible, we can take some solace in the fact that knowledge is a social project, shared among many (as we discuss further in chapter 5, section 5.3). There are experts—in the above example, medical doctors—who devote their careers to helping us navigate complicated subject areas. But even doctors have their own narrower specialties, and amassed information can get unwieldy even within those specialties. A case study in 2010, using echocardiography as an example, estimated that reading every existing paper on the topic would take over a decade, during which nearly another decade’s worth of new reading would accumulate—and even that calculation assumed that the reader’s daily time commitment to reading was equivalent to a full-time job (Fraser and Dunstan 2010). The days when a lone genius like da Vinci could become a leading expert on a wide variety of fields are over; as the deluge of information becomes deeper, the nature of expertise becomes narrower.

For those of us who work in the design and development of technology, it is reflexive to ask whether this technological problem has a technological solution—whether the world’s digital information would be more digestible if only we had a finer-tuned human-machine interface to deliver it to us. We can see this mentality at work even among the giants of the field; Google co-founder Sergey Brin once remarked in an interview (Levy 2004), “Certainly if you had all the world’s information directly attached to your brain … you’d be better off.” The Google founders described their mission as making “the world’s information … universally accessible and useful.” Their products have certainly done a lot to make information accessible. But have they made it more useful? What does usefulness of information even mean?

It is important to note that usefulness of information (or data or knowledge) is not an absolute quality, but a relative one: it depends on the particulars of a situation. As discussed in section 3.1, wisdom can help guide our assessment of the usefulness of a piece of information at any given moment. Another key factor, however, is the sheer volume of information. It may be possible to build a system with some arbitrarily high amount of storage, but stored data and information gain force in the world only under the direction of human projects; therefore, systems should be built in a way that acknowledges human limitations.

The usefulness question may seem secondary to a person who considers information to be an intrinsic good, valuable and important for its own sake. But even such a person has to prioritize other goods, such as bodily health, if only to keep her mind in optimal information-gathering condition. And even an optimally maintained mind has a limited capacity to retain and employ information. Hitting that limit is called information overload.

The concept of “information overload” encompasses both descriptive and normative content. Descriptively, it suggests that we have an excess of information, which implies that there is some limit that we can in some way exceed. Normatively, it suggests that this excess is in some sense “bad” or undesirable. Much recent scholarship presumes that information overload is a problem that impedes a person’s ability to reason clearly and make good decisions. The stakes of this impediment range from making bad purchasing choices to undermining citizens’ ability to participate effectively in a democratic society (Himma 2005).

In the age of mass telecommunications, information overload gets far more attention than it did in the 1970s, when the term first began appearing. But the underlying notion can be traced back to the mid-1940s, when an engineer and science administrator named Vannevar Bush published an article discussing how the library system could not keep up with the constant accumulation of information as a result of ever-increasing research specialization. According to Bush, the quantity and inaccessibility of new research had become an obstacle to researchers, because the amount of effort required to track down and corral new material left them with far too little time and energy for understanding and analyzing the problems in their field.

To cope with the increasing quantity of information on offer in the mid-twentieth century, Bush proposed a technological solution: a device called a Memex, which foreshadowed hypertext and which inspired the development of many computational tools. The technological solutions Bush identified—including increasing accessibility, creating search functions, and allowing for associative indexing—have been realized, to a degree that could not have been imagined 70 years ago. But, as Levy (2005, 283) has pointed out, these developments “seem not to have solved the problem Bush identified but to have exacerbated it.… More of the record is broadly available to us than ever before, but there is less time to make use of it—and specifically to make use of it with any depth of reflection.”

In the early twenty-first century, we likewise face an unprecedented deluge of information: more easily accessed and navigated than that faced by researchers of the mid-twentieth century, but no less overwhelming. The visionary technological solutions to that era’s problems have transformed the information landscape, but they have not been able to solve the underlying problem of human limitations.

Sections 3.1 and 3.2 discuss some of the practical constraints on managing knowledge that often go underacknowledged in working with large volumes of data. In this section we examine some of the opportunities as well as some of the mistakes that can and do happen when we either lose sight of or ignore these practical constraints. Specifically, we explore the concept of bias in automated decision-making systems such as medical diagnosis, bail and sentencing, finance and lending, and product recommendation. We highlight the situations in which technology has helped and the situations in which technology has caused harm as these systems take on a larger role in our lives.

Bias is a complex concept with many definitions. In common usage it is tightly bound with a concept of fairness and typically means an inclination for or against some people, ideas, or things that results in treating them unfairly. As a statistical concept, bias is a systematic tendency for the true value and the estimated value for a population to differ. A common bias in statistics is selection bias, in which we do not choose a sample randomly and hence our estimates of a population are inaccurate. For instance, if we want to estimate the mean height of everyone in the world but select only male individuals from one country, our selection is biased and hence our estimate of the overall mean will be biased (Everitt and Skrondal 2010). Finally, in cognitive psychology, a cognitive bias is typically a heuristic or shortcut that one may use to make decisions, possibly irrationally. These can be both positive (e.g., the optimism bias may incline you to believe that things will work out) or negative (e.g., the base rate bias that leads us to focus on salient, specific instances instead of more general trends) (Kahneman 2011). So, although the word “bias” may mean many things, in this section we focus specifically on its association with using past data to make future judgments or predictions in ways that may seem unfair.

In many contexts we rely on information technology systems that use a mix of data, algorithms, artificial intelligence (AI), and/or machine learning to make decisions and manage complex processes. These decisions can be as mundane as adjusting the timing of traffic lights and as fraught as deciding bail prices and sentencing for criminal cases. In many cases where we rely on these data-driven decision-making systems, it can be tempting to portray these systems as improvements upon the flawed or limited abilities of humans. One famous example is the “hungry judges” case: a study of parole boards in 2011 found that parole was granted nearly 65% of the time at the start of a session, barely above 0% at the end of a session before a meal break, and again nearly 65% after a break (Danziger, Levav, and Avnaim-Pesso 2011). This is an example of the status quo cognitive bias in which we may make default or status quo decisions because they require less cognitive effort. This finding also triggers our common definition of bias because it seems inherently unfair that whether or not you are given parole may depend on where you happen to fall on the docket. Given that even a simple meal break may affect the decision-making process, one might argue that we should make decisions using only “raw data.” We have seen the rise of many similar systems that have allowed professionals of various kinds—mortgage lenders, recruitment agencies, and even criminal court judges—to outsource the ethical responsibility of making crucial decisions that affect people’s lives (O’Neil 2016).

It is important to remember, however, that “raw data” is not some universal, predefined truth. In the introduction to “Raw Data” Is an Oxymoron, Gitelman and Jackson remind us that “data need to be imagined as data to exist and function as such, and the imagination of data entails an interpretative base” (2013, 4). If we lose sight of the fact that someone selected which data to record, how to count things, and what not to count, we can end up hiding behind a mask of “objective data.” This phenomenon is not new. Consider Frederick L. Hoffman who in the 1890s began a quest to understand “objectively,” through data, whether or not formerly enslaved Black people in the United States were better off in a condition of slavery or freedom. Hoffmann claimed, “being of foreign birth, a German, I was fortunately free from personal bias which might have made an impartial treatment of the subject difficult” (Hoffman 1896, v). Hoffman proceeded on one of the first data collection studies across areas such as criminology, where he found that Blacks had higher arrest and incarceration rates than whites; health, where he found that Blacks had higher mortality rates and were more likely to suffer from various diseases including tuberculosis and syphilis than whites; and economics, where he found that Blacks produced fewer crops than whites. Through this data collection and a comparison of the numbers in the 1890s to those during slavery 30 years earlier, Hoffman reached the conclusion that Black Americans were better off enslaved and that, left to freedom, Black Americans were on a path toward “gradual extinction” (Hoffman 1896, 95; Kendi 2016, 2020). This data collection exhibited a high degree of selection bias and ignored the practical reality of the Jim Crow era in the US South, where Blacks were actively excluded from economic opportunity and health care and were actively pursued and arrested for minor crimes such as spitting in public (Chafe et al. 2011).

As we saw in section 3.1, it is not possible to create systems that are completely independent of the thought processes of the humans who build them. These thought processes are inherently encoded in the knowledge and data that the system leverages. Data does not simply produce knowledge; it is also the product of knowledge, and specifically of human knowledge, because when computers “know,” they do so on terms created by human beings. Data that record past human decision making will preserve the value judgments that influenced those decisions. The value judgments are still present, even if their “neutral” appearance as data disguises them. As Barocas and Selbst point out, “data mining can inherit the prejudices of prior decision-makers or reflect the widespread biases that persist in society at large. Often, the ‘patterns’ it discovers are simply pre-existing societal patterns of inequality and exclusion” (2016, 671).

These concerns are not merely theoretical but have been borne out by several situations in which social decision-making algorithms have reproduced subjective bias in harmful ways (Crawford 2016; Staab et al. 2016). These include the following:

• Predictive policing: Several major city police departments are using analytics to anticipate where crimes are more likely to happen and are sending more police to neighborhoods where a high crime rate has been reported. This practice can lead to a higher rate of reported crime in these same neighborhoods because there is much closer scrutiny by the police than in other neighborhoods (Staab et al. 2016). Police have also used similar programs to decide whom to hold in custody or to charge, again leading to a feedback loop based on historical patterns of overpolicing (Angwin et al. 2016).
• “Weblining”: This term is an allusion to “redlining,” the practice of offering services such as home loans or insurance on a selective basis, making them unavailable to the residents of neighborhoods that are predominantly poor or are ethnic minorities. By using a person’s home address or a constellation of other data as a proxy for other characteristics, businesses were able to discriminate against members of racial minority groups while staying within the bounds of the law that forbade them from making decisions based on race (Staab et al. 2016).
• Targeted advertising: Retailers that collect purchasing information from its customers can and do use this information to predict individual shoppers’ choices and habits, and advertise accordingly (Staab et al. 2016). This targeting of advertising can take other forms as well: in 2016 it was shown that Google was displaying ads for high-paying jobs to women much less frequently than to men on the basis of historical characteristics of who would click on the ad and be successfully hired (Crawford 2016).
• Sentencing software: Judges in parole cases are using algorithms to determine the likelihood that a given individual, who is being considered for parole, will re-offend if released. In the case of one widely used program, it was shown that black defendants are twice as likely to be flagged incorrectly as high-risk, whereas white defendants were twice as likely to be incorrectly flagged as low-risk (Crawford 2016).
• Hiring and admissions: Algorithms now almost completely control where job ads appear on hiring websites, social media, and most digital platforms. The placement of these ads can reflect historic bias based on perceived race, gender, economic, or educational status (ACLU 2019). Many companies then use algorithms to sort resumés and applications, leading to further reinforcement of biases (Bogen and Rieke 2018; Schumann et al. 2020). These algorithms are also in use for admissions for higher education, in which they have been shown to replicate a preference for white or male applicants (Burke 2020).

Numerous other examples of these issues abound in popular literature, including O’Neil’s (2016) accounts of biased decision making in areas as diverse as law, loans, and hiring. These issues are also the focus of Eubanks (2018), who draws specific attention to how services for the poor are biased on the basis of what data is collected and what is not. D’Ignazio & Klein (2020) and Perez (2019) look at the gender biases that are coded into data, and Noble (2018) focuses on racial bias.

In many of these cases, big data analytics reproduce existing patterns of bias and discrimination—even when such discrimination is entirely unintentional—because the historical effects of these long-standing biases are a part of the data set and reflect systemic issues across society. The history of discriminatory treatment has meant that women, minorities, people with disabilities, and people with a low economic status have not had the same opportunities to achieve and succeed, and that comparative lack of success is recorded in the data. It is also possible to see how the history of discrimination might “follow” a homebuyer or an inmate into an algorithmic assessment of that person’s future, even if their race or gender is not being explicitly considered. Machine learning techniques and massive data sets can easily (even unintentionally) be leveraged to make predictions that correspond to historical patterns for a given person’s race or gender, even if race or gender is not being used directly as an analytic category. In this way, data ends up being used (deliberately or otherwise) to circumvent societal rules and laws that exist explicitly to prevent decision makers from using these features for making decisions.

All is not lost, and many times the use of automated decision-making systems can lead to positive outcomes and even address societal issues and bias. Abebe et al. (2020) point out three possible roles for computing and large data systems for promoting positive social change: using these systems for diagnosis, for rebuttal, and as a synecdoche. In many of the previous examples we can see computing as a synecdoche, that is, computing as a part that stands in for a larger whole, bringing new attention to long-standing issues. As Eubanks (2018) points out, the problem of how systems of power reinforce and punish poverty is not a new one, but the conversation has gained new meaning and new attention when framed as a technological problem, hopefully bringing attention, resources, and change to how we support those in need. There are also positive outcomes from large, automated decision-making systems: crisis and drug intervention systems have helped to identify, and to match with treatment, people suffering from drug and depression issues, cutting rates of suicide and drug overdoses (Calzon 2020). Within finance there has been a massive decrease in fraud and credit card theft through the use of machine learning systems that are biased by users’ individual behavioral data, because what we are trying to detect is deviations from an individual’s typical behavior (Calzon 2020). In more mundane areas, the ability of Netflix to suggest a great show that you haven’t seen and for Amazon to highlight products that interest you requires leveraging massive data sets and information about what interests people like you; that is, predicting your behavior from a biased sample increases the applicability and accuracy of these systems (Netflix 2021). Finally, in the space of large-scale public health, using data on cancer and health outcomes has helped establish new interventions for cancer treatment and screening and led to new EPA protocols and guidelines (Shaw and Younes 2021). All of this is driven by collecting and analyzing large troves of data and using these data to make predictions, sometimes with full knowledge that a sample may be biased by location or some other factor.

Abebe et al. (2020) describe two other ways in which information technology systems can help produce positive social change: using computing as a diagnostic and using computing as a rebuttal. Many of the systems that we have described in this section, once formalized as an information technology system, can then be tested for bias. In this way one can see the role of these computing systems as a way to perform diagnosis or “debug” decisions that we have made in the past, and to highlight fairness issues within particular sociotechnical systems that may have long been a part of actual practice, we will return to this discussion of sociotechnical systems in chapter 5. Finally, computing can be used as a rebuttal to the deployment of information technology systems or to long-standing beliefs that are not valid. Abebe et al. (2020) point out that collecting and marshaling the huge volumes of data that modern society produces and records can be leveraged to argue against the use of that data. Researchers were able to convince the US Immigration and Customs Enforcement Agency (ICE) not to deploy a system to evaluate whether or not potential immigrants were going to be a “positively contributing member of society” by showing that no system could perform this analysis on the basis of available data. Likewise, researchers who work on matching mechanisms for resettling refugees have shown that integrating more information about local communities, including whether or not there are clusters of people speaking the same language, can improve resettlement outcomes.

One contemporary discussion in the area of automated decision making, which perhaps offers some hope for mitigating issues in this domain, is the question of whether biases are introduced by biased data or biased developers. On the biased data side of this theory, the idea is that many applications are developed using historical data and that these data reflect and could possibly perpetuate any bias present in a former time as represented by the data. On the biased developers side of the theory, the idea is that the programmers developing the algorithms are not representative of the populations that will be impacted by the system, and this lack of representation may lead to bias in the algorithms that are created. Cowgill et al. (2020) attempted to answer this question using an intervention study with over 400 AI workers. They used multiple interventions to test whether programmers with different characteristics and different levels of implicit bias created biased decisions in the final software. Cowgill et al. found no evidence that minority or low-implicit-bias workers generate better, less biased predictions. Conversely, better data leads to better predictions, and as Cowgill et al. found in their study, so does a simple intervention that reminds workers, “As you write your algorithm, please be mindful that your training data set may originate in a biased social system. Adjusting your algorithm to account for discrimination in hiring, self-sorting of applicants, or other sources of such bias could improve your accuracy on the test set. You will be evaluated only on the accuracy of your predictions on the test set.” Cowgill et al. found that this warning serves as a reminder about the main point of the first part of this chapter: that data is not value-neutral and that we must carefully reflect on what the data is, where it comes from, and why it was collected, before we leverage that data (or decide not to) in the design and development of new systems.

As we have seen, large automated decision-making systems can both counter and perpetuate bias. Imagine a system based on the decisions of our hungry judges. This system, if not interrogated thoroughly, may perpetuate and reinforce the existing bias. This is an even more pernicious risk in the context of automated decision-making systems because data is quite often thought of as value neutral, and the answers generated by these algorithms have the appearance of objectivity. The systems therefore make it possible for these bias-dependent assessments to be characterized as impartial, lending strength to the biases that the algorithms were designed to avoid and making it more difficult for those unjustly denied to appeal the decisions (e.g., “this is just what the system said to do”). This appeal to data as value-neutral is called “ethics washing,” a cover-up or facade to hide unethical behavior (Bietti 2020). It is important to note that collecting more data, or making sure that we understand what “fair” and “unbiased” decisions are in the particular domain, is not enough. As Corbett-Davies and Goel (2018) argue, “one cannot generally address [problems of bias] by requiring that algorithms satisfy popular mathematical formalizations of fairness.” Hence, as we discuss further in chapter 5, we must take great care when working on systems that make decisions in society, as it is all too easy for these to perpetuate a history of constricted opportunities into limited future opportunities for historically marginalized groups.

These social and ethical issues posed by automated decision-making algorithms have drawn the attention of both major professional organizations in computing, the Association for Computing Machinery and the Institute of Electrical and Electronics Engineers, which we discuss further in chapter 6. As of 2020, both these organizations have established working groups for discussing topics of fairness, accountability, and transparency in algorithms, and for developing guides and best practices for working professionals in computing technology. Many companies, governments, and academic institutions have also released various statements about AI principles and practices, and as of 2020 there were over 25 of these types of statements. Researchers at the Berkman Klein Center at Harvard University have analyzed the trends in AI ethics statements and possible solutions (Fjeld et al. 2020), finding that highlighting the need to mitigate bias and promote transparency is of tantamount importance for the deployment of AI and information technology systems. Finally, significant legislation in the European Union, called the General Data Protection Regulation (GDPR), specifically requires the right to an explanation of any decision made by an algorithm in order to increase transparency and battle bias (Goodman and Flaxman 2016). These efforts and guidelines underscore the importance of approaching the design and development of automated decision-making technology with a systemic understanding of the nonneutral nature of data, information, and knowledge and provide jumping-off points for how to integrate best practices about data into your professional life.

As we saw above, on the one hand it is appealing, and potentially useful, to have a wealth of information available to all of us all the time. But on the other hand, past a certain point more information causes as many problems as it helps solve.

For all of recorded human history (and likely even before that), human beings have stored knowledge and information in places where they do not have instant access. They have shared knowledge among themselves and come up with social systems for organizing and tracking who is in charge of knowing what and with regulatory systems to make sure that knowledge is shared (or withheld) in ways that somebody or other has deemed appropriate (Hirst et al. 2018). It’s useful to examine those prior approaches to managing knowledge—how they have worked and what their limits are—as a way of thinking about new technologies, those that exist and those still in development, that help human beings store, retrieve, and analyze the things they have learned.

So far, we have discussed the issue of how and where information and knowledge are stored and used relative to the individual. But knowledge and wisdom have always been the product and possession of communities, as much as of individuals. In this section, we consider how knowledge has historically been stored, synthesized, and shared by communities, and how the advent of modern technological systems that process vast amounts of data builds on existing patterns as well as offering new possibilities. We further discuss how modern information and communication technologies have helped to create new communities of knowledge and to mediate existing ones.

The paradigm with which we opened this chapter, and that is widely assumed within STEM fields, characterizes data and information as objective. By extension, it assumes that the knowledge and even wisdom rooted therein will be broadly recognizable as correct in some way because it is synthesized from data and information that is essentially objective.

As we have seen in this chapter, however, that account of data, information, knowledge, and wisdom falls short in several ways. It fails to account for the way in which cultures of knowledge shape the kind of information that seems intelligible or instructive and the sorts of data that seem useful or available. These cultures of knowledge are durable: for better and for worse, a given knowledge culture is far more likely to find a way to assimilate information that might otherwise challenge the collective, shared knowledge and wisdom than it is to discard, or even adjust, its existing frameworks for making sense of the world. Moreover, the human mind can absorb a finite amount of data, information, and knowledge before its ability to manage those things begins to break down. Even before the advent of the information age, persons and cultures were forced to develop strategies to manage their interaction with the vast quantities of data, information and knowledge that confronted them.

The challenges that necessitated these strategies of managing knowledge have not been solved in the information age; they have, on the contrary, intensified. In order to understand how data, information, and knowledge circulate within the world, and to participate productively in this world, it is essential to understand the strategies—and the strategic weaknesses—of human individuals and cultures in coping with the overwhelming onslaught of all the things there are to know.

Once we reformulate our understanding of DIKW, we can gain a clearer picture of how these things circulate in the world and how technological systems shape that circulation and the lives of its users. As this revised paradigm makes apparent, DIKW are not just freestanding bits that circulate in the world, independent of the human beings who know them or resist them or are guided by them. Rather, the things a person knows inevitably become a part of who that person is. This is also true on a broad-strokes historical level, with regard to the traditions of knowledge that we inherit: as the philosopher Hans-Georg Gadamer notes, “we do not conceive of what tradition says as something other, something alien. It is always a part of us, a model or exemplar” (1992, 283).

Our choices about what kinds of knowledge to be guided by, what kind of wisdom to aim for, what kind of data to look for, and what kind of information to accept as real are ultimately choices about what kind of individuals and citizens we will be.

This interrelationship between knowledge and the self raises some key ethical questions that are ever more urgent in the information age. If we as individuals and citizens are to an extent constituted by our knowledge or wisdom or ability to collect information, does the existence of modern information and communication technologies render us, ourselves, less valuable? If the DIKW that constitutes our experience is more widely available to everyone, what makes us unique? To what extent does using modern information and communication technologies make us more autonomous and more free to choose our course—and to what extent does it limit our freedom?

Even asking these questions tips the deck in one direction: it implies that we as individuals can choose to what degree we interact with technological systems and let them shape our decisions. This isn’t necessarily true. Even if we can make some choices about how we ourselves engage with the things that we can learn from or know, we can’t control the choices of others in our community, and we can rarely choose what is known about us. To an ever-growing degree, individual people are being commodified as components of data sets, which are available to marketers and politicians alike. This reality poses a particular ethical challenge for technologists: how can one build systems that serve the people who use them and enhance their lives, instead of converting them into products for consumption by the powerful few?

The next chapter digs more deeply into these questions of privacy, and of what makes a person who they are and helps them to flourish. As will become clear, these two subjects—which may on their face appear to be unconnected—are in fact intimately related.

1.  Pick a decision-making process that incorporates technology, data, information, and knowledge in modern society, such as college admissions, targeted advertising, or bail and sentencing. Describe what needs to be collected or processed, such as what data, processed how, incorporating what background knowledge or judgments, at each stage of the DIKW paradigm to arrive at these decisions. What risks are involved with formalizing such a system in this way? How could errors at one level of the hierarchy impact the functioning at other levels?

2.  Find an example of a data storage methodology that predates computing. What were the goals of this methodology? What had to be left out because it could not be recorded in this way? Find an example of a modern, digital age, technology that addresses this shortcoming. How have we, as a society, come to use this new technology? Has it helped or hurt our decision-making abilities?

3.  When you are preparing to make even a small purchase decision—for example, what to eat for lunch—what data goes into your decision making? Do you consider price, nutrition, where it comes from, aesthetic qualities, reviews from friends, reviews from the internet, or other sources? Make a list of all the sources of data and/or information that you could consult when making this decision. Do you always use all these when making a decision? Why or why not?

4.  Dive deeper into one of the situations in which bias encoded in an algorithm has had impacts in our society—predictive policing; weblining and the digital divide; targeted advertising; bail and sentencing software; or hiring and admissions—and find a recent example in popular media. Take this article and analyze it through one of the ethical frameworks we discussed in chapter 2. For example, if we are looking at hiring software through a utilitarian lens, what is the objective of using the software? What values, such as time or money, are being traded off? Or if we look at sentencing software through a virtue ethics lens, what are the virtues, such as fairness and generosity, that the system design strives to emulate?

5.  Think about attempting to verify something as complex as a medical research paper (e.g., more chocolate makes you less likely to have a heart attack) or a data-based claim in local media (e.g., crime is on the rise). Pick one of these and attempt to find the data necessary to verify the claim. Were you able to? Given the data that you have access to, are you able to verify the claims given the knowledge and information-processing abilities that you currently possess? Why or why not?

6.  Consider the discussion in section 3.6.3 about the idea that social media and mass platforming will bring together communities and usher in a “bond of perpetual friendship between the kindred nations.” In what ways have things gotten easier or better through the use of mass platforms such as YouTube, Facebook, TikTok, and other social media? In what ways has this changed how we communicate with each other and where we get our information? How has this changed how we come to know things?

7.  In this chapter we have claimed that knowledge is a shared enterprise between ourselves and society. Justify this claim by breaking down the ways in which we decide that a particular piece of information is legitimate—for example, that a particular historical event really happened, that vaccines prevent illness, or another topic of your choice.