This article will look at how bias arises in algorithmic systems. We will restrict ourselves to the case of algorithmic personalization and how exactly bias emerges from such algorithms and the use of them.

1. What Is A Personalization Algorithm?

When I talk about algorithms here I will follow Hill's definition of an algorithm:"An algorithm is a finite, abstract, effective, compound control struc- ture, imperatively given, accomplishing a given purpose under given provisions"1. There are many formal definitions of algorithms that make little sense to discuss here, Hill also had mostly algorithms in mind that solve straightforward mathematical problems. As Mittelstadt et al. 2016 rightly note, the inclusion of provisions and a purpose in Hill’s definition means that algorithms are not just mathematical constructs but that they need to be "imple- mented and executed to take action and have effects"2. This in turn then means that we will not just discuss algorithms as mathematical constructs but also the particular implementations of such constructs which are part of a technology. Following Mittelstadt et al. 2016 we will discuss bias in algorithms as algorithms being "mathematical con- structs, implementations (technologies, programs) and configurations (applications)"3


A personalization algorithm then, is an algorithm that has the given purpose to personalize it’s results to a certain user with certain creator-given provisions, those are usually unknown to the user. Consider for example Google and its search engine. Google’s search algorithm does not only provide results to a certain user query (classical information retrieval) it also personalizes those results: The algorithm’s purpose is to personalize the results. It works under the provision that the dataset be the same for each user with the same query but the sorting (the order of the results to a certain search) is done by giving weight to the data accumulated and deemed important about each user. So each user will be presented with different results based on their user profiles, even if they entered the same search term.

To put this bluntly: A personalization algorithm is an algorithm that gives special weight to the data about a user when providing results to the user.

To summarize the issues and definitions discussed here:

Working Definition: A personalization algorithm is a finite, abstract, effective, compound control structure, imperatively given, providing results that give special weight to the data about a user with the aim to make the results as relevant to the user as possible4.

The Phases In An Algorithm’s Creation

Before we can turn to our main subject of bias in personalization algorithms, we need first to discuss how they are created since we want to understand how bias is developed.

Two main phases in an algorithm’s production cycle can be differentiated. Those phases are not so much differentiated by time as they are differentiated by the different actions associated with each phase. There is a design phase and an use phase. The design phase is the phase were the algorithm is created, the main part of development, testing and designing all happen in this phase. The core action associated with this phase is the formalization of the problem to be solved. In other words: The design phase translates social structures and actions into formal, machine-readable text.

The use phase presupposes a functioning algorithm. It doesn’t have to be well functioning but it should produce results. This phase can be partitioned into the act of personalization, that is the algorithmic (mathematical) computation that leads to the results on a user’s screen, and the context of use, the user’s reaction and interaction with the results presented on her screen. The core action of the personalization is the algorithmic computation, the core action in the context of use is the user’s reaction and interaction with the results of the personalization.

Those two acts belong to the same core phase, the use phase, because they rely on there being a functioning algorithm. They are also both phenomena of the same action, the use of the algorithm. One uses the algorithm itself, the other one uses the results of the algorithm. They can also be distinguished by time. The context of use at t1 always presupposes the act of personalization at t0, otherwise there can be no context of use because there is nothing to be used.

Search Company Inc.

To illustrate this, consider the following example: Search Company Inc. want to create a web search engine, so they create an algorithm that produces the results for their search engine. Like most modern search engines, the developers and designers decide upon two criteria by which the results of the search are to be weighted: the number of visits a certain page has and the number of links on other sites that link to this particular page (so called backlinks)5. The algorithm is created and put to use.

So we switch from the design to the use phase. At this juncture the algorithm does not yet personalize its results, so we turn to the context of use. The company soon realizes that other companies that want their websites to be ranked higher in the search results started to gain the system, they did so by paying other websites to put links to their website on theirs. This forces Search Company Inc. to readjust its algorithm and incorporate other criteria than backlinks to weigh the results and we turn back to the design phase. This feedback loop between design and use phase happens constantly6. Only the context of use provides the company with meaningful information on how to ameliorate its search algorithm further.

After a while, the company decides to personalize its results7. This means that this algorithm now also has a personalization sub-phase. The company starts collecting data about its users and creates profiles of them. Since Search Company Inc. has a very large user base, they decide to automate the user profiling, which builds the basis for the personalization. That means that they develop a feedback loop that accumulates data about a user’s interaction with the results of the personalization. This interaction data is then compared to the rest of the user’s profile and the algorithm will now automatically change the personalization based on the data it has gathered. The engine now not only considers data related to the search term, it also considers data related to users initiating the search (their user profiles).

2. Bias In Personalization Algorithms

After introducing proper categories for two understandings of bias relevant to our case, this section will look at the ethical issues raised in each of the two phases of an algorithm’s creation. Those issues can lead to bias but don’t have to, they can be of ethical importance without giving way to bias. I will only sketch out most of those issues since they are not the main point of my argument. Raising these issues however, will give us the adequate grounding for the subsequent argument. This discussion should provide us with a clear picture of the issues raised in the literature and how those can be related to bias.

Individual Bias & System Bias

There are two broad categories of bias which need to be separated properly. For one, bias can refer to attitudes held by social agents without proper evidence and information to support those attitudes8. This kind of bias, let’s call it individual bias, is held by individual agents and is directed towards someone or something (usually groups of people). Such individually held bias can either be explicit or implicit. Philosophers so far have mostly focused on implicit bias and how such attitudes are related to concepts such as moral responsibility or how such attitudes could be changed9.

But there is also another category of bias, system bias, which is not held by individual agents but refers to the outcome of a system. System bias is not held by a system, e.g. the system does not hold such and such biases (considering that the system is not a social agent), the outcomes of a system are biased towards such and such in a systematic way. Such bias can either be part of a social system or can be a more strict, mathematical bias. System bias refers to systematical distortion of results in a system, no matter the composition of the system.

The important distinction of these two forms of bias is that individual bias always needs to be held by an agent, system bias on the other hand, can not be held, it is produced. In other words: A biased agent is biased whereas a biased algorithm systematically produces biased results.

Algorithms can hold both kinds of bias. Technically, an algorithm is a system, therefore it should only produce system bias but an algorithm is created and used by agents. It does therefore not only create bias as a system but it can also incorporate the biases of the agents responsible for the creation of the algorithm. Furthermore, the outcome of a personalization algorithm is used by other agents, they act upon the information they receive, so it might be possible that a personalization algorithm not only incorporates individual bias and produces system bias but that it also creates and reinforces individual biases of the agents using the algorithm.

This categorization of biases will allow us to better understand how bias develops within a personalization algorithm: The developers of Search Company Inc. had some individual biases (implicit and explicit) and somehow those biases landed in their search engine (via formalization). The search will now systematically reflect those individual biases of its creators in its results, they unintendedly created system bias10.

Friedman & Nissenbaum name three forms of bias that can be inherent to computer systems. These are

  • technical bias: Bias that arises through technical limitations,

  • preexisting bias: Bias that is introduced into the system by its creators through their preexisting bias, either implicit or explicit and

  • emergent bias: Bias arising in the context of use, bias that happens through the use/misuse of the algorithmically provided results11.

Those three forms are of special importance because they are directly related to algorithms and they provide a sound basis to rest our work upon. Technical bias is classical system bias because it is inherent to the system and will result in systematic distortions. It is not directly related to any individual bias, it certainly can not be reduced to such bias. Preexisting bias is introduced into the algorithm as individual bias and is then turned into system bias because the system will produce biased results. Emergent bias on the other hand is created as system bias and is turned into individual bias by the users, they develop biases or reinforce biases through the outcome of the system.

Those differentiations and different kinds of biases will help us later on when we will be pinpointing biases to their exact time of creation. Let us now turn to the ethical issues that each phase of an algorithm can pose.

Design Phase

As previously detailed, the design phase entails all processes related to the formalization of the problem to be solved. There are two big issues I see in the design phase: one is the formalization of human needs into an algorithm and the other is the automation of this personalization process. The creators of Search Company Inc’s search engine had to formalize human needs into their algorithm when they created their engine. Their system needs to react to the intentions of its users, while those intentions are only represented through a search query. Complex social behavior (deriving intentions, attitudes and needs expressed in words) was turned into formal language (the computer’s instruction on how to react to Sara’s query). Because Search Company Inc. have so many users, they needed also to automate the formalization process.

The bigger issue of the two is that highly complex social structures and coherences have to be translated and formalized into a mathematical construct. I see two troublesome points in this formalization, for one, the resulting model should come very close to social and human realities, which is a very difficult task. The second is that the formalization also serves the very different needs of two distinct groups, that of the user of the personalization algorithm and that of the creator (business) behind the algorithm12. Let’s call the first problem The Problem Of Approximation and the second The Problem Of Misaligned Needs

Friedman & Nissenbaum deemed the problem of approximation only a part of technical bias, the idea being that the technology is putting constraints on the translation process from human needs to a formalized model trying to fulfill those needs13. Their view does the formalization process not justice. With this view one would have to argue that most or all bias resulting from a computer system is technical, as long as there is no complete copy of "real" life into an algorithm, there will always be just technical bias, all bias resulting from the system would be ascribable to some technological shortcoming. The problems in the search algorithm of Search Company Inc. would then be only part of technological bias, the individual biases would only be introduced into this system because of technology not because the process of formalization in itself is flawed. As long as there is technology that is different from the world (technology that requires a translation process) will the problem of approximation not be reducible to just technological bias.

Formalization does not only lead to technical bias, it also introduces preexisting bias into a system14. Creating a formalization inevitably introduces individual biases and turns them into System Bias. The preexisting attitudes of the designers (be those explicit or implicit) reflect for one their perception of the world but those attitudes will also be reflected in their model of the world. Because those attitudes are in turn formalized and put into the algorithm, it becomes System Bias. The whole algorithm will now reflect the formalized attitudes of its creators. Formalization is not just a problem of putting the world into a mathematical model (As Friedman & Nissenbaum have it) but also a problem whether a particular world-view is properly formalized because it can and will reflect human biases and fallacies.

The problem of misaligned needs amplifies an algorithm’s inclination towards bias. The needs of the user and of the creator/business running the algorithm are misaligned, the user wants information, Search Company Inc. want ad clicks. Such a misalignment of needs tends to skew the scale by which the formalization and its success is measured. It can be assumed that the business needs will carry more weight and that those therefore be favored over the need of information. It is of course just that, a misaligned scale that can lead to bias but doesn’t necessarily have to.

The second issue is that of the automation of the formalization. By that I meant that personalization happens automatically, if it were to be done manually there would be thousands (sometimes billions) of implementations of the main personalization algorithm to be maintained and adjusted to each user. Think back to Search Company Inc. They have hundreds of thousands of users, it is just not possible to have one personalization algorithm for each of those users. They have a clear need for automated personalization. This automation could lead to bias if it happens unsupervised. If there is no (automatic) control mechanism to control what the system is actually doing, it could very well produce biased results. Zarsky holds the opinion that transparency will help to "overcome such concerns" but fails to put forward compelling arguments that would prove that15.

The Act Of Personalization

The act of personalization is a sub-phase that is only present in personalization algorithms. It requires the algorithm not just to achieve some generic task or to create some generic output, said output needs to be personalized and adjusted to the user at the exact juncture of use. There are three main issues in this sub-phase I want to discuss. The first and most obvious one is that of user profiling, the second is an issue of opacity, the black box mentality of most personalization, and the third is about the gatekeeper function and the autonomy of the user.

One building stone for personalization in an algorithm is the use of user profiles. There are three main phases which can be distinguished in user profiling. First, there needs to be collected data. An algorithm usually collects as much data about a user as possible. Second, the collected data needs to be sorted and weighted to create actual profiles of the users. Third, the created profile is used by the algorithm to personalize its output16. Such profiling can be done either explicitly or implicitly17. Explicit profiling allows the user to have some say over the data collected and to actively influence the second phase, how her data is turned into an actual profile of her. Implicit collection requires no interaction from the user whatsoever.

User profiles are necessary for personalization algorithms. If there is no information on which upon they can weigh their results, no personalization is possible. They are, however, another possible source of bias. What’s troublesome about most user profiling is that it happens implicitly. Even services that require the user to make explicit choices about their profile at the beginning will later make many implicit choices and adjustments to the user profile without requiring further interaction from the user. Apple Music for example lets users choose their preferred genres and artists the first time they use the service. Those personal preferences are then automatically adjusted and changed to the listening habits of the user without further interaction by the user.

This is the first point I want to stress out here, the problematic nature of implicit user profiling. This point is of course also part of the black box mentality and lacking transparency of most personalization. An implicitly created user profile neglects the user’s choice to actively and deliberately drive what is presented to her. The second point I’d like to stress here is that the transition from the collected data to the actual user profile can be potentially troublesome. There are just a whole lot of possible variations on a user’s motivation to click on a certain link. Maybe Fiona opened the first result of her search because it sounded interesting, maybe the click was accidental or her computer was used by someone else. It is almost impossible to attribute clicks to certain needs and desires without a bigger picture and more information. I am not saying that such a task is impossible, the point is that an improperly configured personalization algorithm may turn the collected data into a completely wrong profile. And a wrong profile can lead to biased results.

The second issue that needs discussion is the black box mentality or lack of transparency of most personalization algorithms. The predictions a personalization algorithm makes based on user profiles happen in a "black-box".18 The act of personalization as a whole usually is a black box to the users. Both transitions, from data to a user profile and from the profile to the predictions that make out the results, are usually opaque. The user does neither know what data about her is collected, what information is derived from that data (a user’s interests, attitudes etc.) and how the data is used to personalize her results.

Again, the opacity of this sub-phase is in itself not yet a problem or a direct source of bias. But it can be.

As might be clear of now, the differentiation of the issues I am trying to make here is very artificial. Usually those problems and issues do arise together and influence each other. I am just trying to differentiate them, so that we have a more clear understanding of each single issue and its impact.

The black-box issue goes hand in hand with the third issue, the gatekeeper function. Traditional media served some sort of a gatekeeper function, journalists and editors decided what was newsworthy and how the news of a given day were to be sorted. This function has now mostly been taken over by personalization algorithms19.

This shift in the gatekeeper function from human editor to personalization algorithm is in itself not yet that problematic. But it does illustrate how such a shift can give lead to bias when it is combined with complete opacity. The problem is not that there is a gatekeeper or even that that gatekeeper is not human, the problem is that the gatekeeping happens behind closed doors. There is no oversight, no instance of control. This problem is not equal to the problem of algorithmic filtering, there is no such thing as an objective machine the same as there were no objective newspaper editors. The real issue is the personalization of the gatekeeping function. To keep the figure of speech up, where there once was one huge gate for everyone to go through, there are now many gates, one for each user.

A good example of the potential danger of such a situation is detailed by Eli Pariser about the 2000 presidential election and the microtargeting of voters through personalized ads(but every subsequent election would serve as a great example)20. Both candidates were showing personalized ads to users. Most users were shown wildly differing ads. Potential misconduct or even a proper ground for discussion is not given in a world where each user is presented with different (mis-)information, such personalization makes it "increasingly difficult to have a public argument".21

The Context Of Use

The context of use is the sub-phase where the results of a personalization algorithm are shown, put into information and are acted upon by (human) agents. There are two core issues we need to discuss for this sub-phase. Issues of filtering and feedback loops.

Pariser framed the issue of filtering as an issue of transparency. He argued that algorithm-creators should be "making their filtering systems more transparent to the public".22 By filtering he means a function of personalization algorithms that is present in many social networks: If a social network users follows someone else they are not shown every post of this other user they follow, the posts are filtered and some are shown, some aren’t. Such filtering usually happens in complete darkness, the user has no way of understanding why some posts are shown to her while others aren’t. If Search Company Inc. have profiled you in a certain way, they will use filters to show you certain things or to hide certain things from you. So if your user profile has you classified as an ultra-left political activist, it might filter out websites from the opposing side of the political spectrum, for the simple reason that such sites were not often interacted with by other people from your group. Those sites are now gone. If something does not show up in your search results, it might just as well never have existed.

This issue is closely related to that of the gatekeeping function of traditional media but our focus has shifted. We turn from the information-creator/provider to the user of the information and what she does with the provided information. Sunstein called this phenomenon of filtered realities, to which filters in personalization algorithms can lead, "cybercascades".23 Such cybercascades are closely tied to preexisting bias. Existing views are reinforced and confirmed. If users are presented with the same over and over again, if there is no cultural diversity, preexisting bias will harden because there are no diverging views present and new bias will be created, simply because of the homogeneity of the group, leading in turn to more preexisting bias.

The second issue that needs discussion is that of pernicious feedback loops or the omission of feedback loops altogether24. Feedback loops can either be inherent to the system (the system verifies itself and constantly adapts its output to the historical data of use with previous output of the system) or it can happen in the context of use. This issue is closely related to the previous one. Think back to our political activist. She will never ever get to see sites from the political right again. All she will know about them is what is written about them on websites that have a similar political viewpoint to hers. This means that there will be system bias, the system systematically hides results from her but it is also possible that new individual bias will form, she is caught in a cybercascade that only reinforces preexisting notions and new information is well packaged, in accordance to her viewpoints.

A system can either lack important feedback loops or it can create feedback loops like in the example given above, that lead to undesired outcomes because its users start to act differently because of the output of the system. If the outcomes of personalization are not mapped and compared to the expected interaction of the user, the system will be in blind flight. The algorithm will produce and keep on producing shots into the dark. Feedback loops can also develop without any intention by the creator. If Frank is treated by an algorithm as such and such he will soon start to act as if he were such and such and finally actually become such and such. Simply by the power of cognitive biases such as self-fulfilling prophecies and confirmation bias.

Now that we have discussed the ethical issues in the different phases of a personalization algorithm, we can turn our attention to our main argument. The next section will argue for a fourth kind of bias, Aggregation Bias. Aggregation bias only arises when some or all of the ethical issues discussed here are aggregated.

3. Aggregation Bias

I will argue that there is at least a fourth significant kind of bias that can be inherent to an algorithm: Aggregation Bias. Such bias arises through the aggregation of many small factors, design-decisions and uses cases. Considering that our focus lies on personalization algorithms, I perceive aggregation bias as especially problematic in personalization algorithms because a personalization algorithm has to get it right every single time, even a small failure rate will lead to a sub-par experience for some users, if not to systematic mistreatment of certain user groups. However, there is no inherent quality to this kind of bias that would make it only present in personalization algorithms.

To better understand the aggregated kind of Bias I will first introduce the theory of distributed morality by Luciano Floridi.

Floridi’s Theory Of Distributed Morality

Floridi has developed a theory of distributed morality that can be summarized as follows: A multi-agent-system (MAS) can produce morally relevant actions through the aggregation of morally-neutral and morally-negligible actions by its agents25. The agents in this system do not have to be human, they can also be artifical26. Floridi’s idea is that there is some kind of moral threshold in a MAS. Only if the aggregated actions of the system’s agents cross this considerably high threshold will the system produce morally relevant results. That means that most systems will have some morally neutral, some morally evil, some morally good actions within but that those systems will solemnly cross the threshold of becoming morally relevant. Sadly, all the examples brought forward by Floridi are highly unconvincing. I will not try and defend Floridi’s theory explicitly, my arguments will only rest upon the two basic ideas of his theory. The following argument should make it clear though, that the aggregation of actions in a system can lead to morally relevant outcomes.

The premise of this theory is very simple but not easy to digest. It presupposes that individual morality can be aggregated to some sort of system morality. That there is causal aggregation in a system which can lead to troublesome outcomes seems clear but why should aggregated actions in a system be morally relevant? The problem that many seem to have with this point is that there is a general lack of responsibility in a system and our understandings of morality are usually closely tied to the idea of holding someone morally responsible for her actions.

The two core ideas of his theory, that the agents needn’t be human and that the aggregation of morally-neutral and negligible actions can lead to morally relevant actions, are very fruitful in explaining a lot of the issues arising from algorithmic systems, granted that they are convincing. These are of course not new ideas, the collection of individual actions that can lead to morally relevant actions has been widely discussed under the term of "collective responsibility" . It is those two ideas that I will rest the remaining argument upon. First, that actions, no matter their moral direction, can be aggregated in such a way that the outcome is morally relevant and second, that such actions needn’t come from human agents27.

The Aggregation Of Issues

raised many ethical issues and problems of personalization algorithms. However, those issues are mostly benign if they are looked at in isolation. Can a black box generate bias? Yes, most certainly. Is a black box even more prone to create bias in combination with automation? Of course. Do misaligned needs between user and creator amplify bias? Probably.

My argument is simple, aggregation bias, the fourth major form of bias in personalization algorithms, is created through the aggregation of the issues I discussed earlier28. That is not to say that some of these issues might not give rise to bias all on their own, it is to say that such single-issue bias does not carry the same characteristics aggregation bias does. Only the aggregation of many issues gives rise to aggregation bias.

Several of the issues we discussed are already covered by the three existing forms of bias. However, those formulations of bias do not clearly capture what happens in algorithmic systems. Let’s think back to the two different understandings of bias we discussed earlier: Individual bias vs. system bias. There I said that preexisting bias is individual turned system bias, emergent bias is system turned individual bias and technical bias is system bias.

Aggregation bias on the other hand brings all forms together in that it is both system and individual bias turned individual and system bias. Before I explain that any further, let us think back of the seven issues discussed earlier:

  • Formalization (Problems Of Approximation & Misaligned Needs)

  • Automation Of Formalization

  • User Profiling

  • Gatekeeper Function

  • Black Box/ Lack Of Transparency

  • Filtering

  • Feedback Loops

The problem of approximation is covered by technical bias and preexisting bias, the problem of misaligned needs can not be reduced to any of the three forms of bias, neither can the automation of the formalization be reduced or be traced back to any of those forms. User profiling can in large parts be attributed to preexisting bias and can also be a result of poor formalization but some part of it remains free-floating. A lack of transparency does itself not contribute to any kind of bias, it is just a mere fact. The gatekeeper function is in itself also not part of any kind of bias, it is just a function a personalization algorithm might serve. The same has to be said about filtering and feedback loops29.

This leveas us with: The problem of misaligned needs, the automation of the formalization, some part of user profiling, lack of transparency, gatekeeper function, filtering and some part of feedback loops. Those are the issues we discussed that are not easily ascribed to the other three forms of bias.

Can The Aggregation Of Issues Lead To Bias?

I will now argue two things: One, there is bias in algorithmic systems that is neither preexisting, technical or emergent and two, such bias can be traced back to the combination of the issues named above. Only a non-reducible combination of those issues can be called aggregation bias. The non-reducible characteristic of such a combination is the diffusion of responsibility. Aggregation bias is not attributable to anyone in particular. This subsection will make the argument for the existence of aggregation bias and the next will make the argument for the diffusion of responsibility in such bias.

If preexisting bias is inherent to a system, it is there, it does not change in its severity whether such an algorithm be completely open or opaque. Aggregation bias on the other hand does change in severity if one of it’s issues is not present. Aggregation bias would be mitigated if it were in a completely transparent algorithm for example.

Before I make those arguments let me make one thing clear: By saying that aggregation bias is both individual and system bias turned system and individual bias, I do not mean that every form of bias can be traced back to aggregation bias. Aggregation bias is the characteristic form of bias that incorporates parts of those two forms of bias in both phases, the phase of use and the phase of design.

Let me illustrate this. Search Company Inc. decide to use the power of their personalization algorithm to provide other services, one of those services predicts the recidivism rate of inmates. Judges and advisory boards that hold parole hearings are the targeted audience for those services. Each inmate up for a parole hearing will get an algorithmically generated recidivism score. The idea being that the higher the calculated risk (chance of recidivism) the more unlikely should it be that inmates are released early on in their sentences on parole and the more likely are those inmates to be considered a threat to society30. After this service has been in use for some time, an independent study finds out that the resulting recidivism scores of inmates were biased. Inmates belonging to any kind of minority were awarded systematically higher recidivism scores than inmates with similar profiles belonging to the ethnic majority.

Now let’s suppose that the developers were very careful, they did not formalize any preexisting biases they might have had into the algorithm. Let’s further suppose that there is no technological constraint whatsoever for this service, everything the software intends to do can be fully realized with the existing technology and suppose also that the users were not misusing the software.

This means that we have a personalization algorithm that does show bias that is neither technical, preexisting nor emergent. We have a plausible example for a system that shows a fourth kind of bias.

How could such bias have been created? Through the aggregation of issues. This is the first main characteristic of aggregation bias, it emerges through the aggregation of issues within a system.

Search Company Inc. offered a proprietary service, they kept the algorithm behind the service opaque. The process of formalization was automated, the system was fed new data regularly and it adapted its metrics and scores accordingly. The needs between user and provider were not misaligned in our case. The service serves a clear gatekeeper function and it filters the data. The way I see it, there is no clear cause of bias in this example. Even if one would argue that the resulting bias was introduced by a skewed dataset, my answer would be that for one, a skewed dataset is not preexisting bias (it is not individual bias, it is system bias) and that a skewed dataset will do nothing of harm in a properly working algorithm31. Only a skewed dataset that is combined with a lack of transparency and improper (or missing) feedback loops can produce biased results, otherwise the results would be caught beforehand. Only if the algorithm serves a gatekeeper and filter function is the resulting bias problematic. In short: Only if issues arise manifold and if they aggregate, do we speak of this fourth kind of bias.

Bias & Responsibility

I said earlier, that I will argue for two things, one that a fourth kind of bias exists. I did just that in the previous section. The second was that this fourth kind of bias can be traced back to the aggregation of issues. I did start with such an argument. That argument did not yet touch upon the main characteristic of this aggregation: a diffusion of responsibility32.

Aggregation Bias is the only form of bias that has no one clearly responsible for it. There are two main characteristics of aggregation bias, one is its emergence. It emerges through the aggregation of issues. The other is the just named diffusion of responsibility. Technical bias happens through the use of technology that is not yet fully capable of the desired outcome. The creator is to blame for choosing a technology that is incapable or for creating a system that should not have been implemented given the currently present technology. Preexisting bias is attributable to the creators, designers, developers and formalizers. Emergent bias happens through misuse and is mostly the fault of the users, while some part of the blame can also be attributed to the operator since he did not prohibit malicious use. Aggregation bias on the other hand does not have a clear someone responsible for it: everyone in the system and using the system holds some part of responsibility.

Think back to example above about Search Company Inc.’s recidivism score. Who is causally responsible for what in that example? Are the developers at fault for their faulty algorithm even tough they were very careful and managed to avoid any kind of preexisting bias? Is the judge or advisory board at fault for blindly following the algorithms suggestions even tough they did not misappropriate the algorithm in any way? Or is it the politician or manager that decided to use that product? Is the technology as a whole at fault even tough it was up to the task? The aim of this paper is of course not to give exhaustive answers to those questions, it is merely to raise them and to establish those questions as part of aggregation bias.

This diffusion of responsibility in aggregation bias is why I introduced Floridi’s theory of distributed morality. The diffusion of responsibility in aggregation bias means that we need this or a similar theory that is able to handle a diffusion of responsibility, otherwise we will be unsuccessful in ascribing blameworthiness and praiseworthiness. Only if we accept that actions of agents within a system can accumulate and that there is some sort of responsibility as a whole are we able to ascribe responsibility in cases of aggregation bias properly.


Our main question of how bias arises in personalization algorithms can now be answered. There are four distinct forms of bias in such algorithms, bias can be preexisting, technical, emergent or aggregated. We identified different paths as to how those forms of bias can develop. The differentiation between individual bias and system bias allows to explain the development of the four forms of bias.

Preexisting bias is individual bias, that enters a system through the process of formalization in the design phase. Once preexisting bias has entered a system, it becomes system bias because such a systems will produce systematically biased results. Technical bias is system bias and it’s the only form of bias that is already present in most systems, it does not develop. It is inherent to the technology used but a clever use of said technology and the knowledge of its limitations makes this form of bias manageable. Emergent bias develops in the use phase, it is system bias that gets turned into individual bias. The fourth form is the one I argued for. I called this form aggregation bias because I showed that the aggregation of certain ethical issues in personalization algorithms such as user profiling, the gatekeeper function or a lack of transparency can give rise to a form of bias that is non-reducible to its single issues and that does carry different characteristics compared to the other three forms. It emerges through the aggregation of issues and it is the only form of bias that leads to a diffusion of responsibility. No one is directly responsible for the resulting aggregation bias.

So to summarize this: Bias can either enter a system through its creators and their preexisting biases, it can already be present, it can emerge from the actual use of the algorithm or it can aggregate through the coupling of many ethical issues.

Bias Responsibility From To
Emergence Phase
Technical Technology & Creator
System Bias
Already present
Preexisting Creator Individual Bias To System Bias
Emergent User System Bias To Individual Bias
User Use
Aggregated Diffusion Of Responsibility
Individual Bias To System Bias
System Bias To Individual Bias
System Bias
Individual Bias
Aggregation Of Issues
Design & Use

I understand Aggregation Bias as the most dangerous form of bias any algorithmic system can contain. Not only do we not really know how it developed, we are also badly equipped to deal with the issues of responsibility that can arise from this kind of bias. The interesting questions about moral responsibility and shared agency were only mentioned in the introduction of this paper and at least causal responsibility was looked at briefly. I believe the theory of distributed morality to be a start to try and answer those, much bigger questions. I do not yet know, how the theory would hold, would we really try and put it under scrutiny. A comprehensive account of an ethics of algorithms (in the wide understanding we used here) needs to be based on a sound theory of morality that can not only ascribe single actions of individual actors but that can also say something about moral responsibility and its distribution for all the actors (human & artificial) in a system33.


Ananny, Mike. 2016. “Toward an Ethics of Algorithms: Convening, Observation, Probability, and Timeliness.” Science, Technology, & Human Values 41 (1):93–117.

Barocas, Solon, and Andrew D. Selbst. 2016. “Big Data’s Disparate Impact.”

Bozdag, Engin. 2013. “Bias in Algorithmic Filtering and Personalization.” Ethics and Information Technology 15 (3):209–27.

Diaz, A. 2008. “Through the Google Goggles: Sociopolitical Bias in Search Engine Design.” Web Search: Multidisciplinary Perspectives 14. Springer Science & Business Media:11.

Floridi, Luciano. 2013. “Distributed Morality in an Information Society.” Science and Engineering Ethics 19 (3). Springer:727–43.

Friedman, Batya, and Helen Nissenbaum. 1996. “Bias in Computer Systems.” ACM Transactions on Information Systems (TOIS) 14 (3):330–47.

Gauch, Susan, Mirco Speretta, Aravind Chandramouli, and Alessandro Micarelli. 2007. “User Profiles for Personalized Information Access.” The Adaptive Web. Springer, 54–89.

Hill, Robin K. 2016. “What an Algorithm Is.” Philosophy & Technology 29 (1):35–59.

Hinman, Lawrence M. 2005. “Esse Est Indicato in Google: Ethical and Political Issues in Search Engines.” International Review of Information Ethics 3 (6):19–25.

Huff, Charles, and Joel Cooper. 1987. “Sex Bias in Educational Software: The Effect of Designers’ Stereotypes on the Software They Design.” Journal of Applied Social Psychology 17 (6). Wiley Online Library:519–32.

James, Lawrence R. 1982. “Aggregation Bias in Estimates of Perceptual Agreement.” Journal of Applied Psychology 67 (2). American Psychological Association:219.

Jonas, Hans. 2015. Das Prinzip Verantwortung. Suhrkamp.

Levy, Neil. 2017. “Implicit Bias and Moral Responsibility: Probing the Data.” Philosophy and Phenomenological Research 94 (1):3–26.

Mandelbaum, Eric. 2016. “Attitude, Inference, Association: On the Propositional Structure of Implicit Bias.” Noûs 50 (3):629–58.

Mittelstadt, Brent Daniel, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. 2016. “The Ethics of Algorithms: Mapping the Debate.” Big Data & Society 3 (2):1–21.

Newell, Sue, and Marco Marabelli. 2015. “Strategic Opportunities (and Challenges) of Algorithmic Decision-Making: A Call for Action on the Long-Term Societal Effects of ‘Datification’.” The Journal of Strategic Information Systems 24 (1):3–14.

O’Neil, Cathy. 2017. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Broadway Books.

Pariser, Eli. 2011. The Filter Bubble: What the Internet Is Hiding from You. Penguin Books.

Sunstein, Cass R. 2007. 2.0. Princeton University Press.

Zarsky, Tal. 2016. “The Trouble with Algorithmic Decisions: An Analytic Road Map to Examine Efficiency and Fairness in Automated and Opaque Decision Making.” Science, Technology, & Human Values 41 (1):118–32.

  1. Hill 2016, 47.

  2. Mittelstadt et al. 2016, 2.

  3. Mittelstadt et al. 2016, 2.

  4. This is only a definition of one form of a personalization algorithm. There are also other personalization algorithms that work under different provisions.

  5. Hinman 2005, 22.

  6. See Pariser 2011, 230 on exactly this problem put forward by Google as the reason why they need to keep their personalization algorithm opaque..

  7. Hinman 2005, 22.

  8. See Mandelbaum 2016 on the relation of attitudes and implicit bias..

  9. Levy 2017.

  10. This is of course just one possible scenario of how bias can be introduced. As the section will show, System Bias can be created without Individual Bias of the creators.

  11. Friedman and Nissenbaum 1996, 334f.

  12. The reality is such that the biggest implementations of personalization algorithms we have today (Facebook & Google) serve for one the need of their creators, which is in the end to serve the most ads because this is the core part of their business, but also the needs of their users which is usually coupled to some need for information. See also Diaz 2008, 23 on this.

  13. Friedman and Nissenbaum 1996, 335.

  14. See also Huff and Cooper 1987 on how improper formalization can introduce biases into a computer system.

  15. Zarsky 2016, 127.

  16. Gauch et al. 2007, 56.

  17. Bozdag 2013, 213.

  18. Newell and Marabelli 2015, 5.

  19. Bozdag 2013, 214.

  20. Pariser 2011, 151–56.

  21. Pariser 2011, 155.

  22. Pariser 2011, 229.

  23. Sunstein 2007, 44.

  24. O’Neil 2017, 27.

  25. Floridi 2013, 729.

  26. Floridi 2013, 728.

  27. The first idea will be of much greater importance for our argument than the second. But the second will be of vital importance for a comprehensive ethics of algorithms. Only if artificial agents are granted some sort of morality or at least of making morally relevant actions will we be able to sort out responsibility in algorithmic systems.

  28. The term Aggregation Bias also refers to a fallacy in statistics. See for example James 1982 on this, this use of the term is differing from our use. Aggregation Bias in our case has a different meaning, we do not treat it as a statistical problem but as an undesired outcome of a system.

  29. All the remarks made here were discussed at length in so I will not make any unnecessary explanations here.

  30. This here may be an imaginary example but the uses of algorithms for exactly such services is very real. See for example O’Neil 2017, 24–27.

  31. Barocas and Selbst 2016 would disagree with my argument here. They see data as the most important factor deciding an algorithm’s outcome. I do not dispute this point, correct data is very important. But my argument stands, as long as an algorithm is issue-free, skewed data will not corrupt its results.

  32. The problem of responsibility in technology in general has already been raised by Jonas 2015

  33. A first try of an ethics of algorithms is provided by Ananny 2016.