In any attempt to understand or intervene in the design and conduct of work, we can consider several kinds of ‘work’. We are not usually considering actual purposeful activity – work-as-done. Rather, we use ‘proxies’ for work-as-done as the basis for understanding and intervention. In this series of short posts, I outline briefly some of these proxies. (See here for a fuller introduction to the series.)
Other Posts in the Series
- Work-as-Imagined
- Work-as-Prescribed
- Work-as-Disclosed
- Work-as-Analysed
- Work-as-Observed
- Work-as-Simulated
- Work-as-Instructed
- Work-as-Measured
- Work-as-Judged (this post)
Work-as-Judged
Work-as-judged is the judgement, evaluation or appraisal of work, via other proxies for work-as-done.
Function and Purpose: Work-as-judged is the judgement, evaluation, or appraisal of work, via other proxies for work-as-done. Work-as-judged may relate to various aspects of work (e.g., productivity, efficiency, safety, wellbeing, customer satisfaction), and the necessity for workers and work. It has several purposes relating to the selection, evaluation of workers, tools and procedures, working methods, working environments, and jobs, and associated interventions (e.g., policy, procedures, recruitment, training, equipment design, reward, punishment).
Form: Work-as-judged takes the form of judgements and evaluations by individuals and groups. Work-as-judged must, however, employ other proxies, and may be integrated into other proxies. Work-as-judged typically invokes a standard of performance in work-as-prescribed or work-as-imagined, against which to compare work-as-observed (e.g., competency checks, video recordings), work-as-disclosed (e.g., incident reports, interviews), work-as-measured (e.g. output, data logs), and work-as-analysed (e.g., investigation reports, just culture algorithms).
Agency: Work may be judged by anyone, including front-line staff and organisational functions such as human resources, quality, safety, security, design and engineering, and company management. Outside of organisations in which the work in question is done, work may be judged by regulators, supervisory authorities, investigatory bodies, researchers, the media, service users, the public, government, the courts, and the judiciary.
Variety: Work is judged in a variety of ways, including via private opinions and conversations, and the processes and outputs associated with social media posts, surveys, performance appraisals, promotions, investigations, audits, inquiries, committees, media reporting, judicial proceedings, and court judgements. The criteria used for judgement also vary (again, including safety, quality, productivity, regulatory compliance, legality, etc). For the same work-as-done and associated proxy, work-as-judged may be very different (see ‘Stability’ and ‘Fidelity’), both over time and between different people.
Stability: The stability of work-as-judged will depend largely on the stability of underlying proxies. Still, work-as-judged will also change over time depending on the contexts of judgement. For instance, what we judge as acceptable changes with the shifting personal, social, cultural and societal contexts (e.g., values, attitudes, and norms). The informational and technological contexts will obviously affect how work is judged (e.g., logs, recordings). The temporal context will also be influential. For instance, time pressure affects judgement, and as some outcomes emerge over long periods (e.g., major projects, regulatory decisions, government policy), work may be judged very differently at different points in time as consequences unfold. Senior decision makers may not even be in position as some outcomes emerge, and the work associated may be unavailable for judgement when there are no associated proxies (e.g., recordings). Front-line workers are rarely in this situation.
Fidelity: Work-as-judged differs from other proxies for work-as-done. It is not a described or enacted representation of how work is, was or should be done, but instead forms an ‘evaluative representation’ (with emotional and cognitive elements) of work based on other proxies. ‘Fidelity’ therefore applies to other proxies in a way that does not quite fit work-as-judged. It is the case, however, that the fidelity of other proxies strongly affects fairness and justice when it comes to work-as-judged. For instance, work-as-measured for aspects of tasks or output that do not reliably and validly represent work-as-done are often used to form judgments about work and workers, which will be unfair since they lack fidelity. Work-as-judged also has a powerful effect on other proxies. Work-as-judged affects work-as-measured, for instance, because prejudgements about pertinence and possibility affect what we measure and how. Because the measures are self-reinforcing, they in turn further affect work-as-judged. As an example, prejudgements about certain work or workers may result in the reliance on deficit-based approaches to work-as-measured (e.g., incidents, errors), which of course reveal what they are design to reveal: deficits. In this case, far more has been ignored than has been attended to, and most day-to-day work activity is seen as ‘how things ought to be’ rather than ‘how things are made to be despite the the conditions’. Since the way we imagine work is so closely related to our evaluations of work, work-as-judged is inseparable from work-as-imagined.
Work-as-judged is affected by a range of heuristics and biases. These include biases concerning personal characteristics (e.g., sex and gender, ethnicity, nationality, social class, accent), and cognitive biases and heuristics. The following biases and heuristics seem to have a reasonable evidence base in terms of judgement (noting that there are hundreds of claimed ‘biases’, but the evidence base for many is poor, especially in real-world work contexts):
- Outcome bias – We tend to judge a decision based on the eventual outcome instead of the quality of the decision at the time it was made.
- Neglect of probability – We tend to disregard probability when making a decision under uncertainty.
- Omission bias – We tend to judge harmful actions as worse, or less moral, than equally harmful omissions.
- Naïve realism – We tend to think we are objective, but we are not.
- Overconfidence effect – We tend to be overconfident in the accuracy of our judgements.
- Bandwagon effect – We tend to believe things because many others do.
- Confirmation bias – We tend to search for, interpret, focus on, and remember information in a way that confirms our preconceptions.
- Hindsight bias – We tend to believe that events were predictable at the time that they happened.
- Continued influence effect – We tend to believe previously learned misinformation even after it has been corrected.
- Illusory truth effect – We tend to believe that a statement is true if it has been stated multiple times.
- Framing effect – We tend to draw different conclusions from the same information, depending on how that information is presented or ‘framed’.
- Group attribution error – We tend to make assumptions about people based on group membership.
- Defensive attribution hypothesis – We tend to be biased against people who are different to us when evaluating an event.
- Just world hypothesis – We tend to assume that a person’s actions inherently bring morally fair consequences to that person.
In short, work-as-judged is affected by how we think about outcome and baseline frequency, the quality of our judgement, our understanding of others’ mental states, information presentation, individual characteristics, and penalties and rewards.
Finally – related to the framing effect – language, tools and professions have a strong influence on judgment. For instance, the safety literature is replete with negatively-framed vocabulary and concepts, such as human error, non-compliance, unsafe act, violation, never event, and zero accidents. Similarly, tools for judging work (past, present and future) via proxies are mostly deficit-based, and so in some sense ‘what you look for is what you find’. Professions also distort judgment via so-called ‘déformation professionnelle‘. People of different professions attend to, perceive, understand, and judge the same work differently.
Completeness: Work-as-judged should, via other proxies, refer to a representative sample of the work-as-done that is being judged or evaluated. In practice, limited fragments of work proxies tend to be judged. These are, of course, only those fragments that are available, and are often proximate to the outcomes that follow (but that may not be causally attributable). More generally, work-as-judged should take proper account of the people and contexts relevant to work-as-done. However, the proxies that are used to understand work-as-done are limited in their capacity to represent these, and so judgements about work remain partial as well as distorted by a lack of fidelity.
Granularity: Work may be judged based on proxies at different levels of granularity from course (e.g., beliefs about work) to fine (e.g., an observed error). In practice, work-as-judged tends to comprise combined fragments of representations from different proxies (e.g., recorded data, interview responses, analyses, procedures).
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. CC BY-NC-ND 4.0
Humanistic Systems 