Stochastic Machine Witnesses at Work: Today's Critiques of Taylorism are Inadequate for Workplace Surveillance Epistemologies of the Future

I argue that epistemologies of workplace surveillance are shifting in fundamental ways, and so critiques must shift accordingly. I begin the paper by relating Scientific Management to Human-Centred Computing’s ways of knowing through a study of ‘metaverse’ virtual reality workplaces. From this, I develop two observations. The first is that today’s workplace measurement science does not resemble the science that Taylor developed for Scientific Management. Contemporary workplace science is more passive, more intermediated and less controlled. The second observation is that new forms of workplace measurement challenge the norms of empirical science. Instead of having credentialed human witnesses observe phenomena and agree facts about them, we instead make outsourced, uncredentialed stochastic machine witnesses responsible for producing facts about work. With these observations in mind, I assert that critiques of workplace surveillance still framed by Taylorism will not be fit for interrogating workplace surveillance practices of the future.


INTRODUCTION
The use of digital technology for monitoring, tracking and surveilling staf is a standard feature of modern workplaces.How has digital surveillance changed, and how might it change in the future?And critically, do we have the tools as a discipline for reasoning about these changes?This paper takes the form of a metascientifc study.It is not an empirical study in the typical ACM CHI style.I have not interviewed workers or managers for this study.I have not surveyed researchers.Instead, this is an investigation of scientifc practice at work, drawing on past empirical work and contemporary theory to make an argument contribution 1 .
I build my study in several stages.First, I will give some context on the history of measurement in workplaces and the ways that critiques of measurement practices have evolved alongside them.I focus on Scientifc Management (i.e., Taylorism), its descendants, and its critiques (e.g., neo-Taylorism), but also reference the ethnomethodological tradition of workplace research.This sets up the context of contemporary workplace surveillance and the conceptual framings we use to talk about it.I then take a disciplinary turn, exploring how measurement has been constructed in Human-Centred Computing (HCC2 ) over time.This provides the context in which I build connections between the epistemologies of HCC research and workplace surveillance.The aim is to take a critique that is grounded in the sociology of work (neo-Taylorism) into work-based research that draws on HCC epistemological traditions.
To make the connection between emerging technologies, neo-Taylorism and HCC concrete, the next stage of the study develops a vignette of workplace surveillance in virtual reality workplaces that exist in metaverses, like the 'Metaverse' developed by Meta 3 .Do these workplaces, instrumented with the kinds of context sensors pioneered in ubiquitous computing and with interaction design informed by HCC, provide unlimited scope for workplace surveillance?Are they the kinds of workplaces that Frederick Taylor would have built 110 years ago, if only technology had permitted it?I suggest that a neo-Taylorist account of workplace surveillance would say yes; a VR workplace provides almost unlimited scope for observation and control.
I fnish the study by examining whether the emergent changes to workplace surveillance technology really lend themselves to analysis by contemporary critiques of Taylorism.I argue that the epistemology of modern workplaces is changing, and that the science of work is increasingly part of a stochastic zeitgeist (after [15]).Data-driven science is often the norm, in contrast to Taylorism's demand for planned, controlled experimentation.I claim that observation of workplaces will increasingly be undertaken by artifcial witnesses, often outside the direct control of employers.With these changes, I assert that we need to be careful about returning to Taylor, recognising that by framing workplace surveillance in early 20th century management theories (augmented as they may be), means we may not be alert to changes that fall outside the scope of such theories.If we are not alert, then we may see workplaces change in unanticipated ways, compromising our capacity to defend the right of workers.

Positionality
I have come to this work having experienced a realist, often positivist training.My empirical research has largely retained a postpostivist character across a variety of research methods.Throughout this paper, I use the word 'measure'.I view this term as being aligned with realist epistemologies, implying that phenomena exist 'out there', and that we just need to fnd a way to access (i.e., measure!) them.I have chosen to use 'measure' in this paper because, I believe, it is most apt to the discussions of contemporary surveillance at work, Taylorism, datafcation and HCC that follow.Ways of framing what happens in work research other than 'measurement' would admit more diverse interpretivist or constructivist stances, and while I touch on these, they are not the focus of this piece.I acknowledge that, though I make a substantive argument for 'measure' being appropriate, my training and research experiences will also have signifcantly infuenced my choice of language and constructs in this critique.
This paper is about the nature of work and its future.My orientation in discussions of work is to centre the perspectives and needs of those assuming the role of 'managed' over those assuming the role of manager (this is not an uncommon perspective in the CHI community [39]).This manifests in the tenor of the paper, which is critical of management and tech-frm innovation, and undertakes its conceptual development to 'protect the interests of workers'.I accept that there are alternative framings of what follows that could, say, centre the interests of organisations, managers, shareholders, or states.I acknowledge that choosing not to take up such framings refects my preferences and disposition, rather than any intrinsic 'correctness' of a position.Other papers could be written that centre the perspectives of those actors and with them in mind enumerate the consequences of the changes I articulate here.They would be useful.

Provenance
What is the motivation for this work, and what is the need for it 4 ?The original motivation for this work came from observing the increasing use (and consciousness) of workplace surveillance tools ('bossware') stemming from the pandemic [30].I also noticed metaverses entering the news media consciousness [74].Initially, I had simply planned to use critiques from the sociology of work (neo-Talyorism, digital Taylorism) to connect metaverses and bossware in a way that would be useful for a CHI audience.In attempting this critique, though, I came to the conclusion that critiques of work measurement, as commonly formulated, are premised on ideas about measurement and workplaces that may no longer hold.At this point, metaverses went from being the primary focus of this work to a vignette that supports a broader argument about workplace research epistemologies and work surveillance critiques.I hope that that has ultimately led to a contribution that is more essential than the application of a particular critique of a particular work context would be (even if this work also doubles as that).
What about the need for this work, beyond satisfying my own curiosity?CHI has published many critical essays (or 'arguments') over the years (e.g., [5,19,37,49,56,81,84]), including ones on work (e.g., [7]).I have valued these contributions because they have been able to advance arguments without being tightly bound to particular empirical results.For the parts of the CHI community that are interested in the future of work (including the computersupported co-operative kind), virtual environments, and sensing, I felt there was a gap in the literature for a paper that related the development of new workplace tools and systems (and our understanding of them) to fundamental questions about ways of knowing (i.e., epistemologies).I have written this paper with the aim of prompting readers to consider these fundamental questions about ways of knowing as they develop productivity tools, as they capture something about avatars in virtual environments, or as they instrument working environments with new sensors.This paper should, I hope, give some ideas to readers about what they should be considering as they do so.

HUMAN-CENTRED COMPUTING AND SURVEILLANCE
The potential for connected digital technologies to surveil was recognised in the HCC literature as soon as networked digital technology achieved commodity status.Agre [4] ofered 'capture' over surveillance when he discussed the potential for networked machines to measure, track or surveil.The point of the distinction was to emphasise the capacity of new technologies to actively reconstruct environments based on their capacity to sense: 'activity is reconstructed through assimilation to a transcendent ("virtual") order of mathematical formalism' [4].Agre saw this happening everywhere, including at work.Since Agre's work, digital technology has tremendously increased both the potential scope of workplace surveillance (or capture, as Agre would have it) and the ease with which it can be conducted [13].Human-Centred Computing researchers have explicitly investigated the surveillance of emotion [93] in the workplace, as well as the power relations surveillance manifests [10,95].Perspectives are as varied as methodologies, whether researchers are using justice as a frame for understanding workplace surveillance [57], or trying to entertain the idea that workplace tracking could, in particular situations, empower workers [55,110].Though not described explicitly as surveillance, the HCC literature on algorithmic management and other kinds of technologies that constitute or rely on workplace measurement (e.g., [25,68,113]) have also implicitly been studying surveillance.
Researchers have examined the surveillance of workers in roles like public transport [87], most published work on surveillance in human-centred computing 5 has focused on knowledge workers (e.g., [32]) or platform workers (e.g., 'ride-sharing', online microtask platforms).Sannon et al. [95], for example, found platform workers are exposed to surveillance as part of the measurement practices of platforms.They also found that, as such workers are 'customerfacing', they are also subject to (often gendered [10]) surveillance from their clients, too (i.e., the person commissioning a website or receiving a delivery).Altenried describes this surveillance as a way to homogenise the highly heterogenous demographic groups that these platforms require in order to maintain a sufciently liquid labour pool [8].Surveillance, unsurprisingly given its negative connotations, normally appears as a point of critique in these works.
These (conscious or unconscious) investigations of surveillance at work are a subset of broader work at CHI and other HCC venues that have assessed the power of technology to surveil [23] and the actors involved (e.g., [47,69,75,94,109]).The capacity of new technologies and platforms to surveil inside and outside the workplace is clearly an area of interest (or concern) for the CHI community.This paper aims to further develop the community's thinking about workplace surveillance.
Is there anything connecting these various accounts of workplace surveillance, something that helps us to formulate some underlying principles of surveillance?With some exceptions (e.g., [4,7,14,61]), the historical context of workplace measurement, control, surveillance does not feature strongly in HCC discourses on the instrumentation of workplaces for the measurement of staf.'Taylorism' is sometimes used just as a synonym for workplace surveillance.In the sections that follow, I will go into some detail on Taylorism, and then attempt to connect it to epistemological preferences in HCC.

WORK SURVEILLANCE, SCIENTIFIC MANAGEMENT, TAYLORISM AND NEO-TAYLORISM
In the sociology of work literature, the increasing use of sensing technologies for quantifcation [78,79] and the algorithmic control of work [111,112]

Scientifc Management
Scientifc Management has a positivist approach to epistemology, and so a positivist approach to understanding work.Taylor (the originator of the theory), in defending his ideas, claimed to have implemented Scientifc Management "with the object of arriving at the exact truth as to the efect of the system upon the prosperity, wages, health and contentment and satisfactory conditions of the men Taylorist approaches to understanding work can be seen as in contrast to more constructivist traditions to understanding workplaces.Garfnkel's ethnomethodological approach, for example, resists the controlled, positivist aspirations of Taylorism and its descendants (see [106]).There is no sense of a dispassionate observer with truth-revealing measurements.The locus of where understanding comes from moves.As Rawls, puts it, "[t]he observer is not constructing the situation they are analysing, the participants are.Focusing on the observer at all is a problem in itself" [89, p.724].Suchman's work on situated action [98], so infuential beyond work studies, takes place in this tradition.
Interpretivist approaches to understanding work have undoubtedly given us new knowledge (and new kinds of knowledge) about how work happens.The focus of this paper, though, is not on how researchers do or ought to study workplaces.The focus is on how workplaces are studied by the people that run them (i.e., managers).Whether such studies take the form of tracking, monitoring, or outright surveillance, critiques of these Taylorist-like approaches to understanding (and controlling) work are based on what is happening in workplaces.If we are talking about how work is understood within workplaces (rather than by, say, work anthropologists going into them), then, by and large, Taylorism and descendant ideas tend to dominate [17,54].This domination has, if anything, increased with the availability of digital tools that ofer quantifcation and analysis of work activities.

Neo-Taylorism
Critiques like neo-Taylorism have been developed to analyse modern work that is organised along Taylorist lines.The idea is that the changes to work and workplace technology are "no more than a superfcial change, leaving the essential aspects of the traditional Taylorian division of labour intact" [71, p.71], while post-Taylorism suggests changes to working practices have changed too much for Taylorist ideas to retain relevance [71].Some neo-Taylorists argue that not only are Taylorist ideas still relevant, but that changes to work "refect a revitalization of scientifc management" [31, p.422].Gautié et al. [45] describe the specifc role of technology in this revitalization of Taylorism, writing that it "is facilitated, but not primarily caused, by the spread of new digital technologies'' [45, p.777].This implies a kind of latent Taylorism, the reactivation of which is catalysed by technology.Advocates of neo-Taylorism have focused on the decomposition of tasks into atomic units as the hallmark of Taylorism, something that has become even more apparent and specialised in gig economy work [7,26,111].In this paper, I revisit the suitability of Taylorism (and so neo-Taylorism) as a lens for understanding contemporary work measurement.My target is less the structure of work implied by Taylorism, though, and more its epistemology.I will argue that atomisation and control in Taylorism is, to a signifcant degree, a product of its epistemology; the modes through which it seeks to generate knowledge about what is happening at work.This has implications for the suitability and applicability of neo-Taylorist critiques of workplace technology to changing workplaces.
If we are interested in the direction of travel of workplace surveillance technologies, then it makes sense to consider developments through what is ostensibly the dominant realist-positivist lens.This is the lens that is most likely to be used within organisations as they make use of these technologies.It is not about whether work ought to be understood in this way.Given that that work is being understood in this way, what are the interactions between this form of understanding and new technologies going to be?In particular, how will these changes infuence the ways that measures of work are constructed?
Digital technology is at the centre of changes to workplace surveillance.Human-centred computing provides the tools to design, explore and critique these new technologies.If we are going to use HCC research methods to understand the changing nature of measurement in workplaces (and we should), then we also need to consider the epistemological approaches of HCC research traditions.This will put us in a better position to treat the neo-Taylorist and HCC traditions simultaneously as we explore change to work measurement.

MEASUREMENT AND DIGITAL SENSING IN HCC
CHI and adjacent HCC communities are made of up complex and varied research traditions.The publication outputs of the CHI conference suggest that we do not have a single discipline [70].Some researchers argue this is desirable [18,90].These research traditions bring with them a variety of methods and epistemologies [22,42,52].Work research in HCC is a microcosm of HCC more broadly.Work is understood through the ethnographic methods developed by Garfnkel, and transported into HCC by Suchman (e.g., [91]).We see controlled, interventionist, positivist studies (e.g., [72]) sitting next to those ethnographies.And we see studies in the ubiquitous computing tradition focused on instrumenting workplaces with technology, then materialising the 'context' as measured by those instruments (e.g., [44]).In this section, I draw connections between broader epistemological traditions in HCC research, and those that dominate contemporary understandings of workplaces by those responsible for managing them.I focus, therefore, on the more positivist experimental and ubicomp traditions in HCC research, because these, respectively, mirror the more formalised Taylorist traditions of workplace measurement and the less formal, goingwith-the-data-at-hand, improvisational approach that manifests in situ.
The evolution of research in HCC has been described as taking place in 'waves' [21].'First wave' HCC work followed in the tradition of positivist, realist science, using the epistemological machinery of psychology and engineering to understand interactions with technology [35].Other 'waves' have since arrived, but this kind of frst wave work is still taking place, and still making a contribution to knowledge.
First wave research has dealt with the challenges of measurement by going through a process of rarifcation (or, depending on your stance, reductionism).The environment is controlled.Changes to those environments are explored systematically through experimentation.Tasks might be broken down to reduce the confounding efect of one on another.If this sounds familiar, it is because workplace Taylorism is conducted on the same epistemological basis.Planning, control, experimentation, and incentive are central to generating new knowledge, whether in an HCC study of pointing performance or a Taylorist study of a production line.
The advantage of understanding workplaces through this rarifying process is that, so long as the rarifcation can be maintained, manipulations of the environment can have measurable efects (or non-efects) on outcomes.So long as some productivity-enhancing intervention that has been tested in a lab stays in an environment that is like the lab, then we can be confdent that it will work.Of course, the disadvantage of this approach to studying interactions with technology (or within workplaces) is that it is often not possible to maintain this rarifcation in practice.In most workplaces, reality returns and the interventions no longer work so well [83].
There has been another tradition in HCC that, rather than attempt the rarifcation of controlled experimental work, has instead opted for reifcation, to make things more concrete, and ftted to reality as it is found, rather than building rarifed sandboxed realities.This 'third wave' has "partly moved away from a commitment to users towards a more exploratory take-it-or-leave-it approach where designers seek inspiration from use" [20, p.2] where "[n]ew technologies servicing these developments have appeared; pervasive technologies, augmented reality, small interfaces, tangible interfaces" [20, p.2].As part of this wave, the ubicomp tradition has focused on engineering systems that develop machines, tools and systems for building context awareness [1,33].
The advantage of conducting research or understanding workplaces through this reifcation is that it is not necessary to try and control the environment.The goal is to sense it as it is, to be able to measure it as it exists.It's not this straightforward, though.First wave methods are limited by the demands of their epistemologies for control, third wave ubicomp methods are limited by technical ceilings on sensing.When engineering a ubicomp system, it is necessary to consider constraints like power consumption, system availability and processing power.These constraints infuence what systems are able to sense, and how they are able to sense it.For instance, for a device with little memory and a small battery, sampling from a sensor at the maximum sample rate may deplete the battery and fll the memory so quickly as to render the device useless.The challenge for those doing this kind of research is, therefore, construct validity: you know what you want to measure, but that is ultimately inaccessible, and you instead end up using proxy measures.This is what happens in the workplace; managers want to measure productivity, but in many cases that is not a construct that can be measured directly.They instead have to rely on other measures that they hope will sum to productivity, even though the individual measures are hit-and-miss [100].
I have considered on two epistemological impulses in HCC research and related them to our understanding of workplaces.One is the rarifying tendency.In this tendency, control is exerted over the environment to make the environment easier to measure.This comes at the cost of a poor understanding of reality where control is disapplied (e.g., when an experimenter or manager stop looking).The other tendency is the reifying tendency, in which researchers have favoured trying to measure reality (i.e., context) as accurately as possible with increasingly sophisticated sensors.This comes at the cost of construct validity, with the gap between an uncontrolled reality and what it is possible to measure making it difcult to be sure that what has been measured is sufcient to characterise the context.As a result, you end up not measuring the things you need to, or, worse, having what you can measure become an accidental target [103].
If you want to understand a workplace, then these rarifying and reifying impulses seem to be in a kind of contradiction.You can attempt to exert control to construct an abstract reality that you can reliably measure, but accept that if your control fails, the reality falls apart.Or you try to measure reality 'as it is', but accept that your instruments are inadequate to some degree and that you can't really measure the thing you want to 7 .Could the solution to these tensions be to create an environment that yields more control (no wondering what staf are ingesting, imbibing or inhaling on their breaks) that also ofers greater capacity, ease and fdelity of instrumentation (no annoying sample rates or batteries to worry about)?If such an environment exists, would it ofer the aspiring 'Scientifc Manager' a way to 'do' Taylorism, but without having to set up an expensive paternalistic model village next to a factory to try and control the environment as much as possible [58]?I will consider whether virtual reality workplaces, something pioneered by HCC researchers, ft this bill.

WORKING IN VIRTUAL REALITY WORKPLACES
The 'Metaverse' is the way that Meta, the company that owns Facebook and other social media platforms, has been thinking about human interaction of the future.The idea is that people will enter digital worlds through virtual reality devices, which will mediate interactions with other people.This idea is not new.Platforms like Second Life have in the past ofered similar digital worlds [16].
Metaverses are useful here because it has been proposed as the venue for virtual working environments [88].The idea is that rather than working from home or in the ofce and then making use of tools like Microsoft Teams or Zoom for video calls with your online colleagues, or having in-person meeting with others in your ofce, your workplace is in a metaverse.Regardless of where you are, you don VR equipment, and your meeting spaces are in the virtual world.Your software tools are in the virtual world, and you interact with them through your avatar.You no longer have a real workplace, you just have locations where you can join your (virtual) workplace.(Though see Richardson [92] on how infrastructures of work, rather than sites of work, are responsible work's construction.)Let's set aside whether your workplace existing only in virtual reality will actually 'work'.Most likely, it will kinda-sorta-maybebut-not-really work.There will be some use cases where it seems to do the job, and others where its use will be catastrophic in some way.
As it has done for the last thirty years, research in the Computer-Supported Co-operative Work (CSCW) literature will continue to explore which contexts are best for VR work, why the things that work, and how we can translate the things that work to other domains [41,43,108].I am not trying to make any determination about the efcacy or otherwise of these technologies in this work.Metaverses just provide an excellent lens through which we can simultaneously view workplace surveillance, Taylorism and its critiques, and epistemologies of HCC.

Surveillance in metaverses
An employer decides they have a problem with a business process.They decide they need to conduct a study in order to understand what is going wrong.Another employer is anxious about whether staf are productive while they are working at home.They decide they need to keep an eye on staf in order to make sure that they are working 8 .In both cases, employers will want to identify things to measure, because measurement, in the tradition of Scientifc Management, is typically how evidence is created in workplaces.Measuring things about work is difcult, though [50], as we have already considered.
Metaverse technologies are often promoted by their creators as being "primarily as an extension of workers' physical and cognitive agency over a variety of workplace materials and activities [...] rather than a more radical transformation of workplaces' ability to monitor, track, and evaluate worker efciency" [51, p.19]. Park et al.'s work [85, p.3], published at CHI 2023, points to a similar orientation in pre-pandemic academic literature on virtual working environments: that metaverses exist to free workers from the spatial, temporal and social constraints of the physical workplace.
Some recent work has continued to present metaverses as locations for more productive, more fexible, 'better' work.Adhiatma et al. [2] describe the ways in which metaverse workplaces can be 'maximised' to encourage productivity, fexibility and connectivity.The need to train employees so that they have the requisite skills to be productive has been identifed [53].The potential (need!) to develop new key performance indicators that are suitable for (and that leverage) metaverses are described from the perspective of productivity [104], without considering their potential for surveillance: "Immersive engaging interactive experiences can be determined in virtual recruitment by use of behavioral analytics, haptic technologies, deep and machine learning algorithms, and emotional state prediction tools" [46, p.22].Those engaging experiences are, presumably, to the beneft of workers, given that the problems such an environment might produce for workers are not enumerated.
At the same time, researchers are also developing critiques of metaverse workplaces.Szakolczai, after Agre, describes these technologies as being part of a 'captaverse' [99] which normalises abuses of data.Egliston and Carter [36], coming from a Critical Data Studies perspective, focused specifcally on the capacity of metaverses for mass data collection, enumerating the ways in which the capability to sense gives platforms power over those subject to them.Park et al. 's empirical work on perceptions of metaverse workspaces yields a less consistently sunny picture, too: workers are concerned about the potential for surveillance and tracking, and have resisted accordingly [85].

Metaverses and HCC epistemologies
Metaverses bridge the rarifying and reifying impulses in HCC research.In this environment, there is no need to create constrained worlds in which constructs can be more reliably and robustly developed.There is no need to be constrained by the limits of what physical sensors are able to sense.Everything that exists in the virtual world can be instrumented.Things that we might normally need an experiment to be able to sense, like reaction times, become trivially accessible in an authentic working context.We no longer need to be constrained by the physical limits of sensors and batteries.We can develop complex sensors in the ubicomp tradition that allow us to sense context, but we no longer need complex, expensive and error-prone machine vision systems to keep track of where people are.We don't need to rely on proxy measures (like, say, pressure changes in a ventilation system [24]) in order to track people's journeys through environments.Arditi has described these environments as a new form of enclosure: taking the spaces through which people live their lives and subjecting them to new schemes of control and ownership [11].
Lee et al. [67] argue that while the term 'metaverse' was originally coined to refer to "a massive virtual environment parallel to the physical world, in which users interact through digital avatars" [67, p.1], the metaverse of the future will blend physical and virtual worlds seamlessly.As Wang et al. [107] put it, this kind of metaverse is "a fully immersive, hyper spatiotemporal, and self-sustaining virtual shared space blending the ternary physical, human, and digital worlds" [107, p.317].For such shared space to exist, we'd have to conceive a way to instrument the physical and human worlds such that they were entirely intelligible to the digital world.This would mean solving the problems of ubicomp, and implies, once again, an infnitely instrumentable world that where everything that one needs to measure can be measured.(How else could the connection between those three spaces be seamless?) The challenges of construct validity remain, of course.If you take it as read that phenomenological states are largely inaccessible by quantifying behaviours or physical attributes (e.g., using facial recognition to 'detect' emotion), then there remains plenty inaccessible to those running a metaverse.Taylorism and its descendants have generally shown little interest in being able to measure these kinds of things, though.Where they have, they have generally taken a realist position on them -that all of the world is accessible and that you can measure emotion from people's faces.
Given my analysis, metaverses seem the ideal environment for a Taylorist approach to measuring work, and consequently for surveilling staf.There are no unobserved corners for workers to take themselves to.Brain sensing isn't quite there yet [3], but it might be eventually.There are no parts of production that are too challenging to observe.Interactions between workers that might otherwise be challenging to monitor are now mediated only through a metaverse.If those interactions are hard to measure, then you can adjust how interactions are mediated to make it easier for you to measure (e.g., through task design).Everything can be done with code.A Disneyland for an aspiring Scientifc Manager.

RETOOLING NEO-TAYLORISM FOR A STOCHASTIC ZEITGEIST
On the surface, then, a VR workplace seems like it would be the ideal territory for Scientifc Management of one kind or another.
Almost unlimited potential to instrument every tool and every task means a commensurate potential for work measurement, and hypothesis testing.There is a sense in which Taylorism has become a byword for workplace surveillance.More sophisticated technologies that permit an expansion of surveillance, means, ultimately, a reinscribing of Taylorist principles into new working domains [8,29,63].I am not sure that what is happening is as straightforward as 'new Taylorism, same as the old Taylorism', though.And I am concerned that channelling critiques of workplace measurement through neo-Taylorism means that we risk applying arguments couched in older positions on the structure of work, measurement and science to contemporary contexts where there are changes in epistemology.The risk is that we miss out on important aspects of what is changing and so are in a worse position to design for (or against) it.My focus for the rest of this paper is not on whether employers seek to discipline through new digital surveillance, but is instead on what a Taylorist framing of this relation does for our ability to understand the ways in which it is changing.

The epistemology of 'true' Taylorism
As discussed earlier, Taylorism is typifed by the control, planning and atomisation of work that allow it to be pushed through experiments that provide measurements for management.Following Braverman [59], it has been described as the "appropriation of workers' autonomy and control over their work, the construction of a politics and a technology of the disciplined body at work, constituted [one of the] principle[s] of Taylor's system" [12, p.55].This control is then viewed as proceeding "by acting on class and sexed subjectivity" [12, p.63].Without seeking to minimise these aspects of Taylorism, control in Scientifc Management and its descendants is also being exercised in the most positivist sense of experimental control, that is to say, exerting control over people and their environments to make the workplace more laboratory-like and allowing for experiments that test hypotheses and allow causal relationships to be established.
Employers designing rigorous experiments, with clearly operationalised measures and testable hypotheses: if that was ever happening at large, is that what is happening today, and does it look like what is going to happen in the future?It is not clear that it is what is happening, or that it will be what happens in the future [61].Instead, we are at a point where a common strategy is to measure whatever can be measured, and try and rake through what has been collected using machine learning and other statistical tools to try and work out what might be happening [82].This isn't about the use of those statistical techniques in science: as Gigerenzer et al. [48] discuss in their history of probabilistic techniques in science, relying on the fuzziness of probability and statistical techniques has been essential to a variety of scientifc endeavours for hundreds of years.It is rather about the larger process of scientifc practice, and what research communities consider acceptable epistemologies.Moves in how science is undertaken will have signifcant implications for 'Scientifc Management' and critiques of it.

Data frst, ask questions later
Kitchin's [62] infuential work, building on ideas by Anderson and others [9], elucidates this point clearly.It makes the point that the capacity to store and process massive amounts of data has changed the epistemology of (some) scientifc practice.Kitchin notes the position of some researchers "suggest[s] that a new mode of science is being created, one in which the modus operandi is purely inductive in nature" [62, p.4].The paper argues that this is not what is happening -there is no unplanned emergence of theory from a bag of data -but instead this new kind of epistemology "is situated and contextualized within a highly evolved theoretical domain.As such, the epistemological strategy adopted within data-driven science is to use guided knowledge discovery techniques to identify potential questions" [62, p.6].This is more nuanced than 'data frst, ask questions later', but it is a far cry from the heavily controlled positivism that characterised Taylorism and that provides the fodder for neo-Taylorist critiques of workplace surveillance.Can we even be said to be engaging in a neo-Taylorist critique if contemporary workplaces do not require control as an epistemological prerequisite?Employers may still crave control, absolutely, but not necessarily because control is required before scientifc investigations of a workplace can take place.
Kitchin's paper was published a decade ago, at a time when Big Data was the buzzword."Artifcial intelligence" appears in the paper once, and "machine learning" twice.This doesn't dilute the signifcance of the contribution: the kind of data-driven science that the paper describes is increasingly infuential, and, with datafcation an activity seemingly promoted as something for every organisation to engage in [28], increasingly the starting point of investigations.However, the rapid advances of AI and ML technologies over the last few years is changing not just how science is being practised, but who or what is practising it.

Stochastic machine witnesses
Latour relates the development of empirical science to Boyle: "Instead of seeking to ground his work in logic, mathematics or rhetoric, Boyle relied on a parajudicial metaphor: credible, trustworthy, wellto-do witnesses gathered at the scene of the action can attest to the existence of a fact, the matter of a fact, even if they do not know its true nature.So he invented the empirical style that we still used today" [65, p.18].The key to this invention was the acceptability of reliable witnesses as a means of producing new knowledge.Empirical science still works in this way; we have mechanisms for assessing the standing of witnesses, and we use these to decide whether we trust the way that they have witnessed phenomena 9 . 9Reading this paper, you yourself are right now engaged in this kind of witnessing Machine learning can aid the analysis of data in ways that are not all that far from traditional statistical techniques.These kinds of statistics are an accepted part of conducting realist, positivist science.The current direction of AI technologies suggests a departure from this, though.If we are, say, to use large language models (LLMs) to help make sense of large datasets10 , then we get to the point where we are introducing new actors in the process of doing science who are now taking on the role of witnesses in the scientifc process.This fundamentally changes the relationship between phenomena and their witnesses.We have established ways of establishing the credibility of witnesses to phenomena 11 .One way is to require witnesses to complete research degrees before we assess credibility to them.Will we permit artifcial witnesses, and, if so, how will we assess credibility to them?This is not just about the capacity of things like LLMs to 'hallucinate'.Human witnesses to phenomena do the same thing.It is a more basic than that, it is about who is permitted to observe, who is permitted to 'do' science [66].

What this means for critiques of workplace surveillance
What does this mean for Scientifc Management, or neo-Taylorism?It means that the way we are doing science has changed, and the way we might be doing science in the future might change even further.The ability to instrument everything, collect all possible measurements and work out what it means is not compatible with the Taylorist, positivist, realist approach of Scientifc Management.
The means to create witnesses to phenomena that are not human could fundamentally change all scientifc enterprise.So what does the 'Scientifc' in 'Scientifc Management' actually mean?It points to a conceptualisation of science that has dominated the last hundred years.Now we're moving into a more stochastic zeitgeist, where we rely on non-human witnesses who, just like human witnesses, can return diferent responses to the same stimulus, it seems that we shouldn't take it for granted that the possibilities of a metaverse, as interpreted in our new zeitgeist, can be properly explored with conceptual tools like neo-Taylorism that have been developed under certain assumptions about measurement and workplace science.
Manokha [73] has recently argued (after Foucault) that digital technology has fundamentally altered power relations in the workplace, and that technology afords employers a fully-featured panopticon in which their disciplining gaze is omni-present.Similar arguments have been made about the power of algorithmic management, and its capacity to disenfranchise workers [40].I do not disagree that the trajectory of these technologies has been to change power relations in workplaces.This fts with a neo-Taylorist account of what has been happening in workplaces over the last couple of decades; more measurement, more employer control, more employer power.Given the account I have given of the changing epistemology of the workplace, though, that neo-Taylorist account feels disintermediated.The implication is that the disempowerment of workers and the empowerment of employers is dyadic.One's loss is another's gain.Yes, workers have lost power.And so far it looks like that power has transferred to employers.At this stage in the development of these technologies, it looks like this because the employers who have had the capacity to implement the most powerful AI technologies have done so in a vertically integrated way.When Uber develops new methods of algorithmic management, or Amazon builds new sensors for monitoring staf, they do so with the capacity to make sense of the data collected.They do so with more organisational understanding of the new (and still limited) artifcial witnesses to phenomena it is deploying to measure the world.The power of these companies to use vertical integration is not limited to controlling consumer markets [114], it can be used internally to control labour, too.
How do the power relations look outside organisations without the capacity to vertically integrate sensing, measurement, analysis and action?Some -most, the vast majority, even-of organisations will not have this kind of capacity.Instead, they will be consumers of these workplace technologies [50].What will it mean when organisations, convinced of the necessity to instrument their workplaces and surveil staf to stay competitive, contract-out the witnessing of phenomena in their workplace to artifcial agents (leaving aside the question of whether more data can even solve their problems [105])?How will they assess the credibility of those witnesses?What would an LLM, or whatever, for surveilling staf need to do to be awarded "chartered status" like HR professionals 12 ?Where does the power lie when these agents come with a monthly subscription and their proprietary nature makes it impossible to interrogate them?Yes, of course, still to the employer, but also to another set of entities not under the control of employers, the organisations selling access to the artifcial actors whose behaviour is not deterministic.
For a neo-Taylorist account of work surveillance to be complete, then, it needs to be able to expand to ft these new actors.An account that considers a simple tug-of-war between employers and workers is going to end up missing critical activity that is not taking place within that relation.We will perhaps need to augment these accounts of workplace surveillance with our understanding of trust in human-in-the-loop systems [77].We may need to look to the Business Management literature on outsourcing [64] to try and make sense, operationally, of what kinds of control systems might be established.And all of this has to be connected to the essential epistemological question of what can be measured, and how it can be measured.It's only by doing this that we will be in a position to really understand what we are talking about when we're talking about workplace surveillance.Until we have that nailed down, neo-Taylorist accounts of the erosion of good work by surveillance are not going to be adequate to critically evaluate the changes that are coming with AI in the workplace.Ironically, our capacity to defend the rights of workers at work might end up being contingent on resisting data-driven science, where employers get to decide what to investigate post-hoc, and re-embrace the upfront, controlled, planned -and potentially negotiable!-scientifc practices of Taylorism.

DISCUSSION
Having made my argument about workplace surveillance and the ways in which we think about it, there are two things that it would be valuable to consider.The frst is what people designing and building the future of work should take away from this argument.The second is to refect on how my argument relates to the humancentred computing community beyond work and workplace studies.Unfortunately, my thesis here is that machine intelligence has the potential to remake these things.New epistemologies.
Critiques unmoored from the state of the art.Data produced by artifcial witnesses. . .and we're not sure how they are witnessing.We will need to iterate as the capabilities of these machines change over time.What "collect", "process", "store", "analyse" mean for workplace data will not be static.When they change, how will the tools and systems you've built look?What assumptions are you relying on?As a discipline, we have developed ways of using speculation that might be useful for exploring these kinds of challenges [38].

Connections beyond the workplace?
This paper is about how we measure work.I have used virtual work environments as a lens with which to examine critiques of work measurement, assessing these critiques against the state of the art in workplace surveillance.How do these ideas about work connect to other kinds of HCC and HCC-adjacent research?The purpose of this paper is not to provide a generalisable account of surveillance or attendant epistemologies or critiques.The focus is specifcally on workplaces.However, there are related ideas that members of the CHI and broader HCC communities have been working on, and it makes sense to consider how the argument I have made here relates to this work.This paper is closest to publications at CHI on datafcation and algorithmic decision-making.This work is often about how models of the world are created through diferent ways of knowing.Muller and Strohmayer [81], for instance, write about 'forgetting practices' in data science.Which data in a given context is kept.Which is thrown away.This is a critical issue in the workplace, where workers may have no control over how their organisation chooses what to 'remember' or 'forget'.How will artifcial witnesses modulate the 'forgettance stack' that Muller and Strohmayer describe?Given the complexity of this stack, are our current epistemologies sufcient to even know?
There are connections 14 between what I have presented here and Alkhatib and Bernstein's work on algorithmic decision-making [6]. 14With thanks to an anonymous reviewer for the suggestion.
They note that it is often the case that individuals close to decisions make 'street-level' judgements to bridge policies, laws and reality: "A police ofcer chooses whether to issue a warning or a trafc citation; a judge decides whether to allow a defendant to pay bail or to have them remanded to jail; a teacher determines whether to waive a course's prerequisites for a student.These decisions often involve nuance or extenuating circumstances, making it all but impossible to prescribe the right response for all situations" [6, p.2].The point is that algorithms are not capable of making such adjustments.This is relevant to the crowdworking context that Alkhatib and Bernstein explore, but also to workplaces more generally.Evaluating performance at work often involves street-level decisions by managers -accounting for the broader and individual context of a particular marginal case, where protocol and reality sit uneasily together 15 .Street-level arbitrations are not always fair.Employers often selectively enforce policies to target specifc individuals capriciously, but what about where you have artifcial witnesses reporting to street-level algorithms, and this is siloed in a proprietary decision-making stack?We might, as with forgettance, not even be in a position to inspect the underpinnings of workplace decisions.
This work also connects with the Science and Technology Studies work Pine and Liboiron have published at CHI [86].Their work deals with fundamental questions about measurement and the construction of new things through measurement.The contention is that decisions about what to measure are intrinsically political, and that the ability of measurement to construct new things can be used to advance political goals.In the context of workplace measurement, this implies ontological questions alongside epistemological ones: the decisions that are made about measurement are not just about ways of knowing what is happening in a workplace.The measures themselves -of course they do-produce new things in the workplace.The corollary is clear: if you outsource work measurement in your organisation, you are outsourcing the creation of new things (actors?beings?) in your organisation, too.The essentially political nature of measurement (including that by artifcial witnesses: their make-up is political, too [76]) means that the creation of new things will be political, too.In future work, it would be useful to consider the connection between ontology and epistemology in the workplace in more depth.
Given the interest of the CHI community in datafcation and epistemologies more generally, there are almost certainly other connections to make with work already published.It may be that I have yet to encounter this work, it may be that I have but have yet to appreciate its relevance.My hope is that this paper will be of interest to a broad CHI audience and that perhaps, down the line, other authors will read and cite this work and help me make some of those missing connections.

CONCLUSION
The future of work is an important topic of research for CHI and CHI-related communities.A topic of such importance, especially one in which researchers are enacting change, requires a well develop metascientifc underpinning.We need to understand what it is that we as a community are doing and why we are doing it in the 15 Perhaps because policy has poor construct validity?way that we are doing it.The purpose of this paper was to explore a particular aspect of future of work research that has growing salience: digital surveillance.
Via a vignette of infnitely instrumentable workplaces in metaverses, I have made the argument that neo-Taylorist critiques of workplace monitoring, and their focus on control, are not, at the moment, equipped to defend the interests of workers in changing workplaces.Epistemologies are changing.The controlled, positivist science of Taylorism is getting harder to make out in the contemporary practice of workplace science.Artifcial witnesses to phenomena will change the way that we credential the actors that are trusted to observe workplace phenomena.I have sketched the limits of our current tools for critique.I hope this is another potential starting point for the collective development of our critique of work technology, a development that will help to protect the interests of workers as workplaces change.
7.1 The design of the future of work Suppose that you are involved in designing, understanding and building workplace technologies.Lots of people at CHI are!And then suppose that you accept the argument that I have made here.So what?What now?I do not think there are 'implications for design' from this work; the argument that I am making is not specifc enough to a particular context.However, I do think there are 'implications for thinking about design' that are relevant to those creating workplace technologies.To keep these implications somewhat digestible, I ofer an enumeration: (1) Epistemologies of work apply.Yes, your work too!These are not remote ideas for academic-types to mull over.If you have a new workplace productivity tool that, say, monitors activity to make recommendations about when to take breaks, then the things that you are able to measure and decide to measure during development could end up setting normative expectations for behaviour in workplaces.And vice versa: what you measure is infuenced by existing normative practices of measurement in workplaces.Be refexive about the potential for these scenarios.Critiques scafold thinking.We need to form habits of critique.Critiques of workplace surveillance like neo-Taylorism or digital Taylorism provide ways of thinking about work, power and measurement in ways that are consistent over time and contexts.This is useful, given the breadth of workplaces that the CHI community is interested in; software engineering, healthcare, crowdwork, education, research.The full gamut.Applying well-developed critical perspectives provides a starting point for comparison and shared practice.Members of the CHI community have been pushing critical practice for twenty years (e.g., [34, 96]).The conference has a submission track specifcally for critical research.In the same way that working with people implies a set of ethical practices and norms, I would like us to get to the same place with work research.If you design or build workplace technologies, you would get into the habit of applying critiques.You don't have to agree with what those critiques say about what you've created, but if we can make refexivity a habit, we may be able to better anticipate harm [97].The regular application of critique keeps those critiques in good fettle, too; the more of us that apply them, the more we can hear them groan and strain as the world changes.(4) Machine intelligence will mean revisiting the previous three!If you've set up a workfow that lets you think about epistemology, critique and data when you're specifying, designing, building workplace technologies, then job done?