A Classification of Feedback Loops and Their Relation to Biases in Automated Decision-Making Systems

Prediction-based decision-making systems are becoming increasingly prevalent in various domains. Previous studies have demonstrated that such systems are vulnerable to runaway feedback loops, e.g., when police are repeatedly sent back to the same neighborhoods regardless of the actual rate of criminal activity, which exacerbate existing biases. In practice, the automated decisions have dynamic feedback effects on the system itself – which in ML literature is sometimes referred to as performative predictions – that can perpetuate over time, making it difficult for short-sighted de-sign choices to control the system’s evolution. While researchers started proposing longer-term solutions to prevent adverse outcomes (such as bias towards certain groups), these interventions largely depend on ad hoc modeling assumptions and a rigorous theoretical understanding of the feedback dynamics in ML-based decision-making systems is currently missing. In this paper, we use the language of dynamical systems theory, a branch of applied mathematics that deals with the analysis of the interconnection of systems with dynamic behaviors, to rigorously classify the different types of feedback loops in the ML-based decision-making pipeline. By reviewing existing scholarly work, we show that this classification covers many examples discussed in the algorithmic fairness community, thereby providing a unifying and principled framework to study feedback loops. By qualitative analysis, and through a simulation example of recommender systems, we show which specific types of ML biases are affected by each type of feedback loop. We find that the existence of feedback loops in the ML-based decision-making pipeline can perpetuate, reinforce, or even reduce ML biases.


INTRODUCTION
Many of today's automated processes rely on machine learning (ML) algorithms to inform decisions that have a profound impact on people's lives.For instance, they are employed to evaluate whether an individual should be admitted to a certain college [42], be granted a loan [25], or treated as high risk of recidivism [6].The advantage of these ML-based decision-making systems is their scalability, i.e., the capability to handle a vast number of decisions in an efficient manner.However, researchers have found evidence that these algorithms often exacerbate existing biases that underlie human decisions [18,27,41] and even introduce new ones [1,8,16,62].
To solve this problem, a recent line of research in algorithmic fairness started investigating solutions that can mitigate these biases by enforcing some metrics of individual or group fairness [4,9,15,30,50].Although these attempts prove to be successful in the short term, they often do not perform equally well in the long term, i.e., after multiple rounds Pagan and Baumann, et al.
of the decision-making process [46,65]. 1 The underlying reason seems to lie in the fact that the mitigating solutions are designed for stationary systems [13,52], while the system itself dynamically evolves over time.More specifically, the system changes over time because the output (the decision) feeds back as input to the system itself, thus creating what researchers refer to as a "feedback loop".The result is that biases are perpetuated (or even reinforced) due to the existence of the feedback loop, despite enforcing the mitigation techniques.Even though researchers recently started studying the long-term effects of sequential decision-making algorithms (e.g., [17,35,47], see [75] for a recent survey), the proposed simulation-based solutions are drawn on ad hoc models which prevent a comparison of their underlying assumptions and a deep interpretation of the driving factors, i.e., what causes the feedback loops and which components of the system are involved.As a result, to date, we lack a comprehensive classification and theoretical understanding of these feedback loops, and how they relate to the amplification of different types of bias.This is a necessary first step to change the research perspective from developing short-sighted solutions aimed at identifying and fixing existing biases to a more long-term-oriented view that strives to anticipate and prevent biases.
Compared to previous work in the field, in this paper, we do not attempt to provide a simulation-based solution to the existence of feedback loops, rather we fill this theoretical gap by providing a formal definition and a rigorous taxonomy of feedback loops in the ML-based decision-making pipeline, and by linking them to the biases they affect.
To do so, we first clarify the difference between open-loop and closed-loop (or feedback-loop) systems by borrowing the language and the tools from dynamical systems theory, the discipline that focuses on the analysis of systems with dynamical behavior (and their interconnection).Then, we apply this system-theoretic framework to the decision-making pipeline, which is composed of different sub-systems: the individuals' sampling process, the individuals' characteristics representing the decision-relevant construct, the observed features and outcomes, the ML model, and the final decision.
The final decision can feed back into any of these sub-systems, thus forming different types of feedback loops.This, in turn, means that the ultimate effect on the whole pipeline and the amplification of biases depend on what types of feedback loops are simultaneously present in the system.
The first contribution of this paper (see Sec. 2) is to cast the ML-based decision-making pipeline into a system-theoretic framework that emphasizes the different components.Our second contribution (see Sec. 3) consists in providing a classification of the different types of feedback loops, which we call sampling, individual, feature, outcome, and ML model feedback loop depending on which component is affected.Additionally, we introduce the notion of "adversarial feedback loops, " which represent special cases of feedback loops in which the final decision feeds back into the system as a consequence of some strategic action of the affected individual(s).As a third contribution (see Sec. 4), we provide an overview of the different types of bias that can be affected by each of the five feedback loops we introduce.As a fourth and final contribution (see Sec. 5), we demonstrate the potential of our classification framework in the context of news recommender systems.Specifically, we demonstrate that different types of feedback loops can affect distinct parts of the decision-making pipeline, resulting in system dynamics that produce various forms of bias.

THE ML-BASED DECISION-MAKING PIPELINE IN THE FRAME OF DYNAMICAL SYSTEMS THEORY
Accounting for the fact that ML-based decision-making systems are usually not static but evolve over time, we follow Dobbe et al. [19] in using the language of dynamical systems theory to describe them.A dynamical system is a process that relates a set of input signals to a set of output signals.A signal is a variable or quantity of interest that may vary over time.Thus, an algorithm is an example of a dynamical system that receives observable features as input signals and produces predictions or decisions as output signals.Dynamical systems theory is concerned with the mathematical modeling of dynamical systems with the objective of understanding and/or manipulating fundamental properties, such as whether the system reaches a predictable operating point or exhibits oscillatory behaviors.
It is common to represent dynamical systems in block diagrams, where blocks denote systems and arrows denote signals, as a way to provide a high-level graphical representation of a real-world system.Block diagrams are particularly useful to understand and study the interconnection of different (sub-)systems, which are composed to form larger systems.A series interconnection occurs when the output of a system (or algorithm) is the input for another one.A parallel interconnection occurs when the same input enters two systems whose outputs are then combined.In a feedback interconnection, the output of a system is injected back as an input to one (or more) of its components, creating a feedback loop.Series and parallel interconnections lead to open-loop systems, whereas feedback interconnections lead to closed-loop systems -see Fig. 4 in Appendix B for a visual representation.
The prototypical ML-based decision-making pipeline can also be represented as a block diagram.We start by describing its open-loop components, shown in Fig. 1, before characterizing possible feedback interconnections in Section 3. At the beginning of the pipeline, an individual  is sampled from the world (i.e., the environment) I, which represents a signal entering in the sampling function block  : I → .Let  be the individual's identity -i.e., its index in the population, which [34] call potential space (PS) -and let  :  →  be a function that returns the individual's attributes.More precisely,  denotes the construct that is relevant for the prediction -what [24] call construct space (CS).The features , extracted through the function  :  → , , and the outcome  (also called label or target), realized through the function  :  → , are imperfect proxies that can be measured -what [24] call observed space (OS).For instance,  can represent whether or not an individual repays a granted loan and  is a set of features (for example, the credit score, as widely used in the US) that are used by the decision-maker to predict the repayment probability ŷ in order to decide whether to grant the loan or not.For each sampled individual, the final decision  is informed by the prediction ŷ, which is produced based on the observed features  to approximate  using a learned function  :  → ŷ.
Once the outcome is observed, i.e., after one time-unit of delay, the past time's feature label pair ( x, ỹ) can end up as a sample in the dataset (,  ) that is used to (re)train and (re)evaluate an ML model (more details on the ML model development process are discussed in Appendix C).In fully-automated decision-making systems, the decision rule ℎ is solely based on the prediction (ℎ : ŷ → ), usually taking the form of a simple threshold rule, e.g.,  = 1 if and only if ŷ ≥ ȳ.The symbol  indicates the sensitive attribute of the individual (e.g., race or gender) and can possibly also be incorporated in the features .More precisely, the training, evaluation, prediction, or decision-making can use the information on the individual group memberships. 2

FEEDBACK LOOPS IN THE ML-BASED DECISION-MAKING PIPELINE
In contrast to ML, in the field of dynamical systems theory, feedback loops are not always seen as an undesirable feature of a system.Lots of the emphasis of dynamical systems theory is on relating properties of the open-loop system, i.e., the system without a feedback loop, to those of the closed-loop system, i.e., the system with a feedback loop.In this paper, we leverage the idea of closed-loop system properties to define feedback mechanisms in ML-based decision-making systems.Interestingly, closed-loop systems may exhibit desirable properties compared to their open-loop counterparts.
In this section, we complete the specification of the ML pipeline as a dynamical system by considering the feedback interconnections that could be present.We first define various types of feedback loops depending on the component of the ML pipeline affected by the outcome of the system (i.e., the final decision of the decision-maker).Next, we introduce the concept of adversarial feedback loops.Then, we describe how different types of feedback loops can coexist.Finally, we clarify some terminology with respect to positive and negative feedback loops.

Feedback Loops
In many real-life settings, the decision taken at the end of the ML pipeline may feed back into some of its blocks.Every block in the ML pipeline (except the prediction block, as this usually simply consists of applying  to a new input example ) can be affected by the decision, each forming a different type of feedback loop, as depicted in Fig. 2. In what follows, we classify these feedback loops to provide a vocabulary and some examples.To validate this terminology, we reviewed a total of 24 recent relevant papers that discuss issues of feedback loops in the context of ML-based decision-making systems -many of which particularly focus on fairness aspects.These papers are listed in Table 1 (we describe the literature search process in more detail in Appendix A).We emphasize that the classification of the five The first type of feedback loop we introduce is the one that comprises the effects of the decision on the sampling of the individual from the population.This influences the retention rate of different groups and modifies their representation.Consider the following example of a college admission scenario discussed in [53].
First, let the total population be partitioned into two groups  and .The population undergoes a selection process in which an institution, the decision-maker, designs a policy that maps each individual to a probability of being selected, possibly depending on the group identity  and on observable attributes  that bear information about qualification, e.g., GPA, SAT, or recommendation letters.According to the authors of [53], the selection process at time  might change the qualification profiles of either group at time  + 1 through a self-selection process acting in the form of filtering the pool of individuals available at the next iteration.In other words, with the existence of a sampling feedback loop, individuals belonging to a group that had received lower admission rates at the previous iteration might be discouraged from applying as candidates at the next iteration, thus affecting the application rates from the two groups (and ultimately the selection rates).Note that, this feedback loop might lead to one of the two groups disappearing from the candidate pool.To understand this, consider a similar example related to speech recognition products such as Amazon's Alexa and Google Home, which have been shown to have accent bias against non-native speakers [31], with native speakers experiencing much higher quality than non-native speakers.This difference can lead to a sampling feedback loop, where non-native speakers cease to use such products.This may be hard to detect because the speech recognition model, from that point on, only receives input and training data from native speakers, potentially resulting in a model that is even more skewed towards the remaining users, i.e., the native speakers.Without intervention, the model becomes even less accurate for non-native speakers, which reinforces the initial user experience [32].Additional examples of the sampling feedback loop can also be found in [73,74].
3.1.2Individual Feedback Loop.Another possible effect of the decision acts directly on the individual's characteristics  , i.e., through the function .An example of this type of feedback loop can be found in the users' reactions to personalized recommendations.As discussed in [58,61], a user's opinion on, e.g., a certain political issue, is influenced by the news articles received.Therefore, the decision of the recommender system to promote a certain type of content has the effect 3 Notice that certain blocks of the ML pipeline depicted in Fig. 2 aggregate several processes that could be split up into several blocks, potentially resulting in a more nuanced classification of feedback loops.For example, the block denominated "feature extraction" could be split up into measurement followed by feature engineering, creating two subcategories of the feature feedback loop.However, existing works modeling feature feedback loops (e.g., [46,51]) consider the effect of the decision on the distribution of , , which forms the input for the prediction model, without differentiating between the effects of the decision on the measurement and the engineered features.Similarly, we consider the entire ML model development process as one block with (one or more) new feature-label pairs as input and a learned prediction function as output (see Appendix C for more details).
of shifting the opinion of the individuals that receive such a recommendation.Additional examples of the individual feedback loop are discussed in the context of adversarial feedback loops (see Sec. 3.2).
3.1.3Feature Feedback Loop.The third type of feedback loop is relatively close to the previous one.However, in contrast to the individual feedback loop, the decision has an effect on the observable characteristics of the individual rather than on the actual ones, i.e., on  rather than  .One of the most common examples of this feature feedback loop can be found in credit lending scenarios in which a lender decides whether or not to approve a loan application based on the applicant's credit score, which is interpreted as a measurable and observable proxy for the individual's capability of paying back a granted loan [46].For any positive decision, we observe a feature feedback loop: if the loan is repaid, the credit score increases; otherwise, if the applicant defaults, the credit score decreases.Note that, in this example, the feedback loop takes place only if the decision is positive, and it also requires information on the actual outcome .
However, none of these conditions is strictly necessary for a feature feedback loop to occur. 4nother example is constituted by content recommender systems where the time a user looks at some content is part of the observation captured in the feature  [10,63].However, the time explicitly depends on what the recommender system has previously suggested, thus closing a feature feedback loop.This happens irrespective of whether this recommendation affects the individual's interests, i.e., even in the absence of an individual feedback loop.
Additional examples of the feature feedback loop can also be found in [17,64,65,73].Furthermore, similarly to the individual feedback loop, also for the feature feedback loop, there exists an adversarial counterpart (see Sec. 3.2).

ML Model Feedback
Loop.In the ML model feedback loop, the final decision  affects the ML model by modifying the training or the validation data sets (,  ) that will be used for future predictions.Typical examples in this category are known as ML-based decision-making with limited [22] or partial feedback [5] and the reason is that ML models are retrained using newly available data.ML model feedback loops describe the case when the data that becomes newly available over time depends on the decision taken.For example, hiring algorithms only learn about the skills of the candidates who were hired, credit lending algorithms only receive repaying probability information from people who received the loan, and predictive policing algorithms only register crime in patrolled neighborhoods.In all these scenarios, the decision will create a gate to the pair (, ), which will be added to the existing data set (,  ) only when the decision is positive ( = 1).Notice that, when the retraining of the model does not depend on the decision (i.e., if the feature-label pair (, ) is added to the existing data set independently of ), there is no ML model feedback loop.Using the language of dynamical systems theory, this case is simply viewed as an open-loop system with memory where the state variable (,  ) evolves according to the inner dynamics, but independently of the output variable (the decision ).
Additional examples of the ML model feedback loop can also be found in [21,23,63].
3.1.5Outcome Feedback Loop.Finally, in the outcome feedback loop, the decision  affects the outcome  before it is realized and ultimately observed.Notice that this observed outcome then needs to be reused in some form in order to close the loop.Namely, it only forms a loop if the outcome is used, e.g., as part of the training or validation data when retraining the model5 .To see how an outcome feedback loop can arise, consider again the credit lending scenario: if a person is predicted at high risk of default, the loan might be granted, but at a higher interest rate.However, the decision to enforce a higher interest rate further increases the chances that the customer defaults [57].In contrast to the example provided in Section 3.1.3,here we assume that the lender's decision  has an effect on the realization of the outcome , i.e., whether the loan is paid back or not, rather than on the features (credit score).

Adversarial Feedback Loops
Some of the previously described feedback loops can take the form of what we call adversarial feedback loops, which can arise if the decision  is intertwined with an adversarial reaction to it.While there is no visual difference with respect to the block-diagram representation, adversarial feedback loops differ from their non-adversarial counterpart in that the decision triggers the reaction of the individual(s) subjected to the decision-making process, which then affects the ML-pipeline.In practice, in adversarial feedback loops, individuals subjected to the decision-making process react strategically to the previous decisions by taking actions that increase their chances of receiving favorable decisions.
Notice that this is an important distinction for the design of measures that intend to control the system's dynamics, e.g., through bias mitigation techniques (as we will discuss in Section 6 in more detail).For instance, consider the attention allocation problem discussed in [17].Here, the decision-maker has limited (insufficient) resources to exhaustively inspect  different locations, and therefore they have to decide where to (dynamically) allocate the attention.As the authors argue, the incident rate of each of the  sites responds dynamically (and adversarially) to the previous allocation, i.e., it increases where there was absolutely no control, and vice-versa it decreases proportionally to the amount of inspection.In essence, this example describes the case of an adversarial individual feedback loop, because the decision ultimately affects the incident rate, i.e.,  .
To give another example, consider a college that publishes the decision rule for its admission policy.Prospective students can strategically invest in their own qualifications in order to meet the requirements.If this action truly changes the preparation level of the student [48], then it is again an adversarial individual feedback loop.However, it is also possible that only the observable features of the individual are changed [33], e.g., if the students invest in SAT exam preparation without changing their actual qualification for the college.Then, we are facing an adversarial feature feedback loop.Similarly, if an individual is applying for a loan, it might be beneficial to open multiple credit lines to improve their observable features [57].This action is not truly modifying the individual's capability of paying back the loan, but it is only a way to game the decision-making policy, thus we have an adversarial feature feedback loop.
Additional examples of adversarial individual and feature feedback loops can be found in [33,35,43,76] and [17,36,43,51,67], respectively.However, we emphasize that it is not always easy to distinguish between the individual and the feature adversarial feedback loops, because many of these works assume that the decision affects the qualification  of the individuals, but oftentimes they intend that it only affects its observable features .

Coexistence of Feedback Loops
As seen in the previous sections, different feedback loops can coexist within the same application domain.Thus, which type(s) of feedback loop(s) (co)exists solely depends on the context-specific assumptions regarding the underlying causal effects of the decision.The possibility of the coexistence of different combinations of feedback loops gives rise to coupled behavior and even more complex dynamics.

Positive/Negative Feedback Loops and Relation to Stability
In many disciplines, including the ML community, a considerable emphasis is placed on classifying feedback loops as either positive or negative [38,54,60].This is often accompanied by some ambiguity in the definition of these notions.In systems theory, a positive feedback loop (also known as reinforcing) amplifies the effect of inputs on the outputs, while a negative feedback loop (also known as balancing) attenuates it.In other domains, the notion of a positive/negative feedback loop is sometimes associated with desirable/undesirable outcomes, regardless of how it acts to amplify/attenuate inputs.For example, the feedback loop that increases recidivism due to incarcerated individuals' reduced access to finance is referred to as a negative feedback loop in [75, p. 2].This ambiguity is problematic, especially considering that in systems theory the desired goal is often to make the output a predictable function of the input and independent from other exogenous but inevitable inputs (considered as disturbances).For this reason, properly designed negative feedback loops are deemed preferable, while positive feedback loops are often considered problematic.
However, systems theory often places more emphasis on the stability of the closed-loop system rather than classifying feedback loops as positive or negative.A stable system converges to a predictable equilibrium point, while an unstable system either oscillates or grows beyond bounds.It is intuitive to associate positive feedback with instability and negative feedback with stability, however, this intuition is not universal [2,72].On the one hand, positive feedback is guaranteed to lead to instability only in the special class of linear systems.The presence of non-linearity (e.g., saturation or hysteresis) can stabilize a positive feedback loop, which is intentionally introduced in many cases (e.g., the design of signal amplifiers).On the other hand, negative feedback does not guarantee stability (even in linear systems).Moreover, the same system could be in either positive or negative feedback depending on the operating regime (e.g., the frequency of the input signal).Thus, in this paper, we shift the focus from classifying feedback loops as positive/negative to asking whether the closed-loop system converges (or not) to a (desirable) state.As we will see in the examples in Section 5, feedback loops often drive the ML-based decision system to stable equilibrium points in the long run.

FEEDBACK LOOPS AND ALGORITHMIC BIASES
Being able to reason about what caused certain types of bias is of incredible practical importance in order to avoid or counteract them in the long term.Otherwise, ML-based decision-making systems can result in socially undesirable outcomes over time.Many works claim that those biases can be perpetuated or even reinforced due to feedback loops [3,13,14,40,49,50,55,56,68].However, a clear understanding of the causal effects of feedback loops on algorithmic biases is currently missing.We fill this gap by connecting the classification of feedback loops (which we introduced in Section 3.1) to algorithmic biases and explain in more detail which types of bias they affect.Table 2 provides an overview of the connections we establish.However, the term 'bias' can have different meanings and be used interchangeably with synonyms for different types of bias.To ensure consistency, we adopt the concepts and terminology introduced by Suresh and Guttag [66].
Representation Bias.According to [66], there are different nuances of representation bias: Representation bias can arise (i) if the defined target population does not reflect the use population, (ii) if the target population contains Sampling feedback loops can affect representation bias.Sampling feedback loops affect the sampling function  that outputs a set of individuals on which an ML-based decision-making system acts.A sampling feedback loop changes the sample of individuals for whom a prediction and, ultimately, a decision is made (i.e., those who get a chance to be selected).Thus, it can result in representation bias, which describes the situation in which  undersamples some part of the population.As a result, the available data is not representative of I and, for this reason, the ML model likely does not generalize well for the disadvantaged group [66].
ML model feedback loops can also affect representation bias.The ML model feedback loop changes the sample of individuals whose realized outcome becomes observable, i.e., those that are selected and can thus be added as a new feature-label pair (, ) to the sample (,  ) -see Fig. 5 in Appendix C for a visualization of this process.Therefore, it can affect representation bias, which stems from a shift in the training data distributions. 6  Historical Bias.Individual feedback loops can affect historical bias.Individual feedback loops act on the construct space (CS) of an individual, i.e., the inherent properties of an individual  change and not only the observed proxies , , which are measured in the observed space (OS) [24].This can result in historical bias (also called "life bias" [34]), which describes injustices that manifest in inequality between groups in the CS.As decisions can change individuals' properties  , which can manifest in altered future features , it becomes more difficult to treat individuals fairly since the decision actually changed them.This means that the world is accurately represented by the data (i.e., the measurement functions  and  are acceptable), but the state of the world (i.e., an individual's inherent decision-relevant attributes  ) is the result of unfair treatments in previous decision rounds [66].For example, not considering counterfactual decisions for individuals (i.e., assuming that individuals would have evolved identically over time, even if they had been assigned different decisions) can drive the decision system to a state in which individuals are disadvantaged solely because of an unlucky event in the past, even if their attributes are perfectly measurable.
These two feedback loops act on the measurement functions  and  and thus affect an individual's observable properties , , .The outcome feedback loop changes the realization of the outcome ().In contrast, the feature feedback loop changes the observable attributes that are fed into the prediction model ( and, potentially, ), i.e., the features for future decisions.Thus, both types of feedback loops can affect measurement bias: the features  and labels  are usually just proxies as they try to measure an inherent property of an individual, which might represent a construct that is not directly measurable or even observable ( ) [66].Measurement bias describes the transition between CS and OS [24].Thus, it describes a situation in which those proxies less closely approximate the intended attribute for certain 6 Additionally, ML model feedback loops can affect evaluation bias.More generally, evaluation bias exists if the sample used to evaluate an ML-based decision system does not represent the population it is used for, i.e., it stems from a shift in the evaluation data distributions [50,55,66,68].Hence, ML model feedback loops can affect evaluation bias if the sample used for evaluation (  ,   ) is influenced by past decisions.individuals or groups, which means that  or  (or both) are not appropriate to capture the relevant construct. 7For example, using arrests as a proxy for the risk of committing a crime (as is the case in the recidivism risk prediction tool COMPAS [1]) is problematic if there are groups that are much more likely to be arrested for certain crimes.

CASE STUDY: FEEDBACK LOOPS IN RECOMMENDER SYSTEMS
To demonstrate the potential of our classification of feedback loops and their relation to the different biases, we present a unifying case study on recommender systems (RS). 8We consider the case of an online platform where the RS is used to provide content the users are interested in.For simplicity, we consider just one relevant item (e.g., a specific video) and denote a user's interest in this item with  ∈ [0, 1], where larger  corresponds to higher interest.The realized outcome  denotes whether a user shows interest (e.g., clicks on the relevant item in question),  = 1, or not,  = 0.
The platform uses an RS to predict a user's interest ŷ =  (), where the feature  ∈ [0, 1] represents the user's past clicking behavior on the platform.For this simple example,  is the percentage of recommended relevant items that the user has clicked on in the past and thus serves as a proxy of the user's interest in the relevant item.The function is learned through a logistic regression (LR) algorithm (which is fitted to a sigmoid function) trained on data (,  ), which consists of a collection of feature-label pairs (, ).To decide whether the relevant item should be shown as one of the top recommendations ( = 1) or not ( = 0), the following threshold rule is used:  = 1 if ŷ > 0.5, and  = 0 otherwise.After each recommendation round,  is observed, (, ) is added to the existing dataset (,  ), and the RS is retrained.We consider two groups of users  ∈ {G1, G2}.For simplicity,  is not used as an input for the RS.We now provide one example for each type of feedback loop described in Section 3.1 to illustrate how they are associated with different biases.The initial conditions specific to each of these simulation examples are described in Table 3 and the initial  distribution is shown in Fig. 6 in Appendix D. Notice that the mean of  is higher for group G1 (i.e.,  ,1 = 0.7,  ,2 = 0.3), which means that individuals of group G1 are more interested in the item, on average.
Sampling Feedback Loop.First, we look at a special case in which  = 0 corresponds to not receiving any recommendation, leading to users leaving the platform.Instead, when receiving  = 1, users stay on the platform.Initially, about 50% of the active users on the platform are from group 1 and 50% from group 2:  1 = 496 and  2 = 504.Every time someone leaves the platform, a new user replaces them.To mimic users' homophily, the new user is drawn from 7 Notice that the list of the biases in Table 2 is non-exhaustive since some biases are connected: feedback loops could also indirectly affect learning bias or aggregation bias.Learning bias represents a limitation of the learned function  that occurs when erroneously assuming that  ( |) is homogeneous across groups [66].Aggregation bias arises if the ML-based system fails to draw the correct conclusions for certain individuals and, therefore, results in disproportionately worse decisions for some group [66].For example, feature or outcome feedback loops directly affect measurement bias and indirectly affect learning bias at the same time if the shift of the distribution ( ,  ) -which is connected to measurement bias -also results in heterogeneous conditional probabilities,  ( |), of getting a certain output for a given input across groups.Similarly, they could indirectly result in learning bias if the way the learned function  is optimized is less suited for the new distribution ( ,  ). 8 The code to simulate this case study is publicly available at https://github.com/paganick/feedback-loops-and-biases. group 1 with probability  =   1  (else, from group 2), i.e., the higher the percentage of users from group 1 in the platform, the higher the probability the new user belongs to group 1.As can be seen in Fig. 3a, this phenomenon leads to the reduction of  2 from 504 to 89 individuals after 10,000 time-steps.This distribution persists in future time-steps, suggesting that it is a (locally) stable equilibrium point of the dynamical system.Group 2 is underrepresented on the platform in the long term with just 8.9% of the platform users.This corresponds to nuance (ii) of the representation bias as described in Section 4.However, at the same time, nuance (iii) of the representation bias is present for both groups: since only those given  = 1 stay on the platform, the sample of active users becomes less representative over time, i.e., only interested users (those with high values for  ) stay on the platform (see Fig. 3b).Notice that it is difficult to classify the sampling feedback loop as positive or negative in this case, as there is no initial representation bias against Group 2 that gets amplified by the loop.The resulting biased equilibrium point is simply a property of the closed-loop dynamics.
Individual Feedback Loop.An example of an individual feedback loop is when the recommended content influences the user's opinion  , which we model by letting the new opinion be a convex combination of the previous one and the recommended content.Fig. 3c shows that this results in a polarization of interests on the platform.Namely, users with high initial interest (i.e.,  > 0.5) are more likely to be recommended the item and, as a result of this, their  further increases over time, and vice versa for users with low initial interest. 9Due to the initial difference in the distributions of  (see Table 3), historical bias increases.Namely, results show bigger group-level disparities with a very high  for group 1 users and a very low  for group 2 users, on average.The steady-state value reached by the trajectories in Fig. 3c represents a biased stable equilibrium point of the closed-loop ML system in which the opinions are polarized.
Feature Feedback Loop.Fig. 3d shows the result of an example in which the content recommendation feeds back into the feature extraction block  (rather than acting on the actual opinion  , as was the case in the previous example), thus forming a feature feedback loop.Compared to all the other examples, there is no difference in the mean of the initial  distribution across groups G1 and G2  ,1 =  ,2 = 0.5 .However, initially,  is a noisy observation of  for both groups (i.e., there is measurement error  ,1 =  ,2 = 0.1) and there is measurement bias on the feature observation of group 2  ,1 = 0,  ,2 = −0.2 .This means that the interest of individuals of group 2 is systematically underestimated.
Over time, the true interest of users is learned, since the feature feedback loop updates  using new accurate information on the users' interests.This reduces the measurement error, which can be seen by the reduced variance in Fig. 3d.Fig. 3d further shows that the feature feedback loop reduces the measurement bias (which is measured as  −  ) of group 2 over time, i.e.,  −  ∼ 0 after 50, 000 time-steps.Hence, the system converges to an unbiased stable equilibrium point, and the feature feedback loop leads to eliminating the measurement bias in this case.
ML Model Feedback Loop.Next, we consider the case in which the content recommendation feeds back into the ML model.For the initial training, we specify a high noise (   = 1) such that the model  does not contain any information to predict  from , resulting in ŷ = 0.5 for all individuals (see solid black line in Fig. 3e).Over time, the model observes new data and becomes more accurate, i.e., the predictor approximates the true outcome realization  ( ).
In this example, the training dataset (,  ) is enriched with feature-label pairs (, ) only if the relevant item was recommended to the user (i.e., if  = 1, namely, partial feedback), assuming that the platform cannot measure the user's interest in an item that was not recommended.This forms an ML model feedback loop.The result is that after an initial period of exploration, the ML model quickly learns how to predict  for individuals with large values of , as those become the ones more likely to receive positive decisions.Here, we measure the prediction error as ŷ − E[], however, in the absence of any measurement error in the outcome realization ( ( ) = ),  is approximately equivalent to E[] except for some noise, which is negligible for the average over a group of individuals.As can be seen in Fig. 3f, the prediction error quickly approaches 0 for G1, but the LR algorithm continues to perform poorly for G2 in the short to medium term.In the long term, thanks to the noise in the observation of 10 it eventually approaches 0 also for G2.
Retraining the ML model over time reduces the representation bias, nuance (ii) of the representation bias as described in Section 4.However, it is due to the ML model feedback loop that the sample (,  ) becomes more representative of group 1 after just very few time-steps while taking much longer to reduce representation bias for group 2.
Outcome Feedback Loop.Finally, we consider an example using the same initial conditions as in the sampling and individual feedback loops, but this time the RS's decision affects the outcome realization .Namely, the probability of the realized outcome to be  = 1 increases/decreases by 20% for positive/negative decisions, respectively.This means that the realized outcomes  (, ) are more extreme than they would be if there were no outcome feedback loop (see dashed lines in Fig. 3g).Despite starting with an unbiased ML model, over time, the retrained ML model approximates  (, ), i.e., the initial predictor is a much flatter sigmoid function compared to the final predictor.Namely, the outcome feedback loop introduces a measurement bias on the realized outcome  for both groups G1 and G2.Thus, as is visible in Fig. 3h, the prediction error ŷ −  diverges from 0 (as ŷ predicts the realized outcome  and not  ) until it reaches a stable equilibrium point after approximately 10,000 time-steps (at approximately 0.2 and -0.2 for G1 and G2).From the perspective of platform users, an outcome feedback loop can result in a situation in which one keeps receiving recommendations due to having clicked on similar content in the past, despite not being interested in it.

RELATED WORK AND DISCUSSION
Along with providing a common system-theoretical framework for the growing literature on fairness in sequential decision-making (already discussed in Sec.1), our aim is to pave the way for a fruitful collaboration with different communities working on, e.g., distribution shifts, adversarial machine learning, control theory, and optimal transport.
In what follows, we elaborate more on these research directions and on future interdisciplinary approaches.Distribution Shifts.Many works have investigated ML under different types of distribution shifts over time.The problem of concept drift is broadly defined as a shift of the target distribution over time [26,70].This is a rather broad definition, which includes distribution shifts due to exogenous effects, e.g., a pandemic or a financial crisis.However, such shifts can be arbitrary and do not assume that feedback loops are present in general.Recently, endogenous distribution shifts, i.e., target distribution shifts caused by the deployed prediction model, have been investigated more thoroughly.The concept of performative predictions acknowledges the fact that ML-based decision-making systems can affect the outcome they try to predict [57].The notion of performative stability, which is defined as a predictor that is not only calibrated against historical data but also against future outcomes that are produced by acting based on the prediction, is a possible solution that achieves a stable point for retraining [57].This stable point means that a model remains exactly the same if it is retrained on future outcomes.Performative prediction is an umbrella term for a situation where ML-based decisions cause a shift in the outcome distribution.However, this distribution shift can occur through any type of feedback loop we introduced in Section 3.1. 11As we showed in Section 5, these feedback loops have different properties and implications.For example, changing a platform user's opinion (through an individual feedback loop) is very different from changing the individual's realized outcome (through an outcome feedback loop).In all cases, the recommendation changes the individual's consumption.In the former case, this is caused by shaped preferences.
In contrast, the decision-relevant individual attributes remain unaltered in the latter case.More research is needed to study the effects of the specific feedback loops we classify on the concept of performative power [28], which only considers shifting outcome distributions in its more general understanding as in the performative prediction literature.
Adversarial Machine Learning.Adversarial ML studies attacks on ML algorithms and how they can be defended [37,69].
The idea is that adversarial attacks are executed by an attacker who intends to influence some part of the ML pipeline, whereas the developer of the ML algorithm thwarts the attacker's objective.In contrast, feedback loops do not occur due to malicious external manipulation but are a direct consequence of the dynamics in sequential decision-making systems.Yet, the outcomes of certain adversarial attacks are closely related to the feedback loops we classify in this paper.For example, data poisoning attacks are associated with ML model feedback loops in that they modify the data used for training.Applying measures designed to counter adversarial attacks to deal with feedback loops in sequential decision-making systems represents an interesting avenue for future research -for example, robust learning through data sub-sampling [39] or trimmed optimization [45] to counter ML model feedback loops.First results have shown that this becomes more complicated if the fairness of the decision-making systems is a concern [71].
Control Theory and Optimal Transport.Control Theory provides the tools to drive dynamical systems towards a state with a desired behavior.These goals are reached by designing a controller with the required corrective behavior.
Our framework provides a basis to interpret the decision-maker as a controller that can be purposely designed to mutually achieve desired performances and bias mitigation, enabling the use of tools from Control Systems theory, such as Economic Model Predictive Control [20] or Optimal Control [44].Using these tools, one could incorporate fairness guarantees as constraints and performance guarantees, e.g., engagement on an online platform, as the objective function, possibly in a Regret minimization fashion [7].For the case of adversarial feedback loops, one can think about the decision-maker and the external environment as the two players of a zero-sum game where Robust Control [77] techniques find their natural expression.For example, referring to the selection process example in Section 3.1.1,one can model the adversarial actions taken by the candidates as a disturbance for the decision-maker in achieving the best candidates selection.When looking at bias mitigation techniques at the group level, tools from Optimal Transport can also come in handy [11].Optimal Transport allows quantifying the violation of fairness constraints as the distance between the group's current distribution and a desired one [12].This tool would allow the design of a controller (decision maker) that drives the initial distribution towards an ideal one, fulfilling fairness constraints.

CONCLUSION
The output of ML-based decision-making systems, i.e., the decision, often affects various parts of the system itself, creating a so-called feedback loop.Yet, ML evaluation techniques usually omit potentially important temporal dynamics [13,46,52] and taking feedback loops into account is crucial to avoid unintended consequences [17,46,65,74].In this work, we build on dynamical systems theory to provide a general framework that sheds light on the different types of feedback loops that can occur throughout the ML pipeline.We identify five distinct types of feedback loops, some of which can be classified as "adversarial" whenever the decision feeds back into the system as a consequence of some strategic action of the affected individual(s).Furthermore, we associate the different types of feedback loops with the corresponding biases they affect, and we demonstrate these dynamics using a recommender system example.
By rigorously analyzing the ML pipeline, we believe that our framework is a necessary preliminary step towards (i) understanding the exact role of the feedback loops and (ii) shifting the research focus from short-sighted solutions that aim to identify and correct existing biases to a more forward-looking approach that seeks to anticipate and prevent biases in the long term.First, providing a rigorous classification of feedback loops will pave the way for a systematic review of existing works in the ML literature and it will allow putting their results into the perspective of their assumptions (e.g., which types of feedback loops are considered and which are not).Second, with the help of additional tools, e.g., dynamical systems and control theory, it will be possible to fully exploit the potential of our framework in the purposeful design of feedback loops, and for the development of effective long-term unfairness mitigation techniques.

A LITERATURE SEARCH PROCESS
We conducted a literature search to identify relevant articles for our analysis.We started by performing a forward and backward search using an initially small set of important papers [17,23,46,75].Then, we performed a keyword search on the ACM Digital Library and google scholar using various combinations of the following keyword: "feedback loop, " "algorithmic bias, " "algorithmic fairness, " "machine learning, " and "feedback." We performed the classification of existing works in three stages: First, we skimmed each paper and added it to a list of potentially relevant papers if it contained some mention of feedback effects and ML -this list consisted of 75 papers.Next, we looked at each paper in more detail and only included it in the final base of literature to be considered if it investigates feedback loops as part of ML-based decision-making systems.Finally, looked at each description of feedback loops in those papers and mapped it to our conceptualization of feedback loops.This resulted in the 24 papers listed in Table 1 and, through an iterative process, ultimately also in Figure 2 presented in Section 3.1.

B OPEN-AND CLOSED-LOOP DYNAMICAL SYSTEMS
In Fig. 4, we provide a simple visualization of the two types of series and feedback interconnections.A series interconnection (Fig. 4a) composes two systems into an open-loop system.A feedback interconnection (Fig. 4b) composes them into a closed-loop system: the output is injected back as an input to one (or more) of the components, creating a feedback loop.
Unlike their open-loop counterpart, closed-loop systems are not straightforward to predict from their components and require specialized techniques to analyze.
The interconnections of systems are systems themselves, and a model for the interconnected system can be derived from the models of the individual subsystems that compose the interconnection.This derivation is straightforward in the case of series interconnection, but it becomes significantly more involved in the case of feedback interconnection.
Only for special classes of systems, for example, systems where the input-output relation of each subsystem is linear, a model for the resulting interconnected system can be derived analytically.For general dynamical systems, a tractable direct derivation is typically very difficult, and numerical methods (including simulations) come to help.

Fig. 1 .
Fig. 1.The ML-based decision-making pipeline as an open-loop system.

Fig. 2 .
Fig. 2. The ML-based decision-making pipeline as a closed-loop system in which different feedback loops can emerge.
For instance, the recommender systems for an online platform can affect the opinion of the users  (individual feedback loop) or just their representation in the feature space  (feature feedback loop).College admission policies can induce students to improve their qualification (adversarial individual feedback loop) or just their representation  (adversarial feature feedback loop).Alternatively, they can also lead to different retention rates across groups (sampling feedback loop).Lending decisions can affect an individual's creditworthiness  (individual feedback loop), credit score  (feature feedback loop), realized outcome  (i.e., whether or not the granted loan is paid back, representing an outcome feedback loop), or even the data used for the ML model development (,  ) (resulting in an ML model feedback loop) or the sample of individuals applying for a loan in the first place (causing a sampling feedback loop).All five classified feedback loops represent some causal effect of the final decision on another component of the ML-based decision-making pipeline.

PPFig. 3 .
Fig. 3. Dynamic effects of different types of feedback loops (FL) acting on an RS pipeline for an online platform.Circles in the box plots denote outliers.

Table 1 .
Overview of feedback loops in the algorithmic fairness literature

Table 2 .
Feedback loops and the ML biases they affect (iii) if the sampled group of individuals is not representative of the target population.All three versions represent some difference between the used dataset (,  ) and the population I.