The Quantified Self in a Complex System: A Systems Perspective on Mental Workload

In the last few years, many of us have started to quantify ourselves. We have purchased activity trackers to monitor and track health and fitness metrics such as distance walked and run, calorie consumption, heart rate and sleep quality. By quantifying inputs, outputs and what goes on in between, it is possible to set a desired goal, adjust, and track progress toward it. This occurs within a complex system (a person), but one where it is usually possible to control the inputs and outputs fairly well. Sometimes, we like to think of socio-technical systems in the same sort of way. We measure concrete things like traffic and RT (radio-telephony) load, less concrete things like traffic complexity, and sometimes rather abstract intermediary things like ‘mental workload’. Yet many remain sceptical of the quantification of aspects of human experience in complex systems.

From a systems perspective, human performance exists in the context of a dynamic (and often messy) system. For complex systems such as aviation, everything connects with something. When there are changes in one part, there are adjustments elsewhere. So ‘human performance’ is only relevant in the context of the system: other humans, a variety of equipment, procedural constraints, working environments, demands, and so on. All of these aspects of the system interact in variable ways, over time and in different situations. From a humanistic perspective, human beings supersede the sum of their parts. We cannot be reduced to components or concepts, nor can we be dislocated from our human and environmental context. The trouble with many measures in socio-technical systems is that they can dislocate, mask and distort the human, system and environmental context.

Quantitative data about humans and systems look scientific because they take on a certain (often spurious) accuracy in black and white, with all their decimal places. But such data are as political as they are scientific, or at least they become so because the search is sometimes not for an answer to a question but the desired answer to a question (e.g. that a change is safe or acceptable). Unravelling the history of the numbers can reveal some inconvenient truths. And once something is measured, it can be tempting to prescribe a maximum, minimum, or target. All of these can create problems in socio-technical systems, which do not have hard physical parameters, and which can change their behaviour in response to being measured, and in response to arbitrary quantified targets. Numbers can take on a life of their own.

But it is naïve to think that we can or should completely avoid numbers. Unfortunately, there remains an attitude among some that “If you can’t measure it you can’t manage it.” This is despite everyday evidence to the contrary, and despite the thinking of management and quality guru and statistical professor, W. Edwards Deming who remarked that “the most important things cannot be measured”. And qualitative data – of the sort that I tend to prefer – don’t always penetrate the management-by-numbers or hard engineering mindsets. Qualitative data are messy and might not reduce uncertainty in the same way as numbers, and uncertainty is a key source of anxiety for decision makers. It’s also worth remembering that quantitative measures can also suggest that workload is too high, and this might carry more weight for some than a story or controller comments. Numbers may also be the only thing that some have any time or inclination to digest when it comes to decision making. The quantification of performance is really a trade-off in data collection. Such data can often be gathered from more people, more efficiently.

For these reasons, most numerical measures concerning human experience and system parameters should be treated as social objects. Any data on mental workload, sector capacity values, traffic numbers, or whatever, are a reason for a conversation, the start of a conversation – not an end point. We can’t measure workload like we can measure our heart rates, calorific intake or physical activity, but we can do what we can to try to make sense of our experience, accepting that any data collection is a compromise, and there are nearly always social and political implications.

In practice, how we human factors specialists measure, assess or understand your workload – or anything else – is secondary. This is because only a small minority of European ANSPs (air navigation service providers) employ human factors specialists in the first place, and those ANSPs who do any kind of ‘mental workload assessment’ could be counted on one hand (with fingers to spare). Decisions about changes to technology and procedures are, in the majority of cases, made with no input from human factors specialists in ANSPs. Decisions are made on the basis of a perceived business or operational need and an available technological or procedural solution (which sometimes creates a ‘need’), and the solution undergoes some form of design process and safety assessment. Technological solutions are increasingly commercial-off-the-shelf, with little room for adaptation.

After over one thousand hours talking with operational staff (and managers) all over Europe, and hundreds of hours observing controllers, my questions on workload rarely concern numbers, even though so much research on workload is aimed at measurement. To you air traffic controllers, some of these are worth asking prior to the introduction of changes. For instance:

• Ask the proposer about the purpose(s) of the change – the answers may change over time.
• Ask designers and engineers about the requirements engineering process, user needs analysis, prototyping, interaction design, testing and simulation.
• Ask training specialists about the training needs analysis, the length and timing of training and familiarisation, its design, method and platform.
• Ask HR and planning about the staffing, stress management and fatigue implications, including shift work and breaks from operational duty.
• Ask operational management about how demand can be reduced or varied when needed (e.g. high-workload training flights in small airports).

And finally, ask yourself, your colleagues and all of the above about your involvement in all aspects of the change. You are the experts in your work, and you will inherit the result of any changes…and have to adapt to them.

Reference

Shorrock, S. (2015). The quantified self in a complex system: A systems perspective on mental workload [pdf]. HindSight 21, June. Brussels: EUROCONTROL.

Afterword

Note that the article is written for an air traffic control context for HindSight magazine – hence the references to aviation – but hopefully applies more generally across other domains. The full issues if this HindSight magazine on ‘Workload’ will be available sometime in June 2015 here.