Testing saccadic peak velocity as an index of operator mental workload: A partial overview

Di Stasi, Leandro Luigi

Cognitive Ergonomics Group. Department of Experimental Psychology. Faculty of Psychology - University of Granada

Campus de Cartuja s/n, 18071 Granada

+34 958243767/ distasi@ugr.es

Marchitto, Mauro

Cognitive Ergonomics Group. Department of Experimental Psychology. Faculty of Psychology - University of Granada

Campus de Cartuja s/n, 18071 Granada

+34 958243767/ maurom@correo.ugr.es

Antolí, Adoración

Cognitive Ergonomics Group. Department of Experimental Psychology. Faculty of Psychology - University of Granada

Campus de Cartuja s/n, 18071 Granada

+34 958243767/ antoli@ugr.es

Cañas Delgado, José Juan

Cognitive Ergonomics Group. Department of Experimental Psychology. Faculty of Psychology - University of Granada

Campus de Cartuja s/n, 18071 Granada

+34 958243767/ delagado@ugr.es

ABSTRACT

Automation research has identified the need to monitor operator functional states in real time as a basis for determining the most appropriate type and level of automated assistance for operators doing complex tasks. For this reason, the development of a methodology that is able to detect on-line operator attentional resource variations could represent a good starting point to solve this critical issue. We present an experimental series that demonstrates the validity and sensitivity of a specific eye movement index; i.e. saccadic peak velocity (PV), that is able to detect variations in mental state (workload/fatigue) while doing complex and ecological tasks. PV was tested in different experimental contexts, as well as air traffic control simulated tasks and driving simulator sessions. This research could provide several guidelines for designing adaptive systems (able to allocate tasks between operators and machine in a dynamic way) and early fatigue-and-distraction warning systems to reduce accident risk.

Keywords

Fatigue; eye movements; main sequence; microworld; ATC; driving.

INTRODUCTION

In Cognitive Psychology and Cognitive Ergonomics, the idea that mental activity can be envisioned in an “energetic” perspective in which quantities of activity can be measured, can be found in the first pioneer works of the two disciplines, together with studies on factors that can increase or decrease mind activity levels. From both a purely theoretical point of view (and consequently in the relative framework of basicinvestigation) as well as in real work context application studies (for increasing or at least preserving comfort of operators, developing high quality instrumentation or ameliorating interactions with technologies), mental activity has been regarded for many years as a set of processes for which quantitative measures could be determined. Thus, the notion of “load” is one of the core issues in mental activity investigation and modelling. According to this dynamic conception of mental activity, mental workload has progressively been given more importance, especially in ergonomic studies of operator performance in safety-critical contexts, such as industrial plants (e.g., processing industries and nuclear power plants) and transportation systems (especially aviation).

The existent literature on workload topics is very wide, and a number of reviews on the concept have existed for years. As with all constructs that are not directly observable, there are several definitions of mental workload. A common trait of almost all definitions is the presence of a task (or, more generally, a situation) that requires an effort (mental or physical) that a subject has to sustain, according to his/her possibilities. The quantity of effort that can be directed to the task depends on many factors related to the task as well as to the subject him/herself. One of the definitions that more explicitly covers the aforementioned aspects is that provided by Parasuraman and Caggiano (2002): “Mental workload refers to a composite brain state or set of states that mediates human performance of perceptual, cognitive, and motortasks”. Moreover, mental workload can be driven exogenously (or “bottom‐up”) byenvironmental sources, namely, by task load, as well as endogenously (or “top‐down”) by voluntary mental effort. In this definition it is clear that the connection between mental activity and brain activity, for which many measurement techniques have been developed throughout the last decades, de facto, mind is a function of brain.

MEASURING ATTENTIONAL ACTIVATION IN REAL WORK CONTEXTS

Currently, three sets of methods are most prevalently used to evaluate mental workload: (1) subjective reports of perceived effort by the operator; (2) methods based on the performance level reached by the operator in a secondary task; and (3) psychophysiological indices that reflect the cognitive state of the operator.

Primarily, mental workload is measured using subjective tests. A variety of tests and questionnaires have been developed to quantify this subjective rating. Some of these instruments use subscales to provide separate indices of the different dimensions of mental workload. They have the advantage of being relatively easy to administer and to interpret, and they do not require extensive training or expensive equipment. However, although these subjective techniques are popular, there are several methodological problems associated with their use (Tsang & Velazquez 1996). First, they tend to be specific to the situation and frequently fail to be influenced by adaptability, learning, experience, innate ability, and changes in the emotional state of the person performing a task. Second, the subjective measures generally perform poorly when there is little variation in mental workload. And third, they do not assess evolution over time. Furthermore, correlations with performance measures are not perfect (Yeh & Wickens, 1988). These are problems common to a variety of research fields in cognitive ergonomics. The main problem with subjective techniques, however, is their off-line nature, which often make them unpractical or intrusive, as when for instance control operators are asked to fill in mental workload questionnaires off-duty, during rest breaks or even worse, at lunch time. Indeed, intrusiveness can convert subjective questionnaires into requiring further effort.

Performance based techniques for assessing mental workload are based on measuring response times and performance accuracy while the subject simultaneously performs a secondary task. Their validity has been supported by extensiveinvestigations. Their most important advantage is that they provide data that are less influenced by personal and confounding factors, and they are objectively verifiable by outside observers. However, they are demanding, tend to be more specialized, and require expensive equipment and extensive training of the person administering them (Christodoulou, 2005). Furthermore, in most situations they cannot detect small variations in mental workload, they do not always or necessarily evaluate mental workload directly, and they can be insensitive to variations in the difficulty of the task. In conditions where multiple tasks must be undertaken, the operator’s sustained attention may get increasingly involved, making the main task performance more difficult.

Because of the problems associated with subjective and performance measures, researchers and practitioners have turned their attention to the promising area of psychophysiological indices. In recent years, there have been numerous reviews of the application of psychophysiological techniques to ergonomics and, in particular, to the area of mental workload (for examples, see Kramer, 1991; Wilson & Eggemeier, 1991; Gevins et al., 1995). Several indices of mental workload and performance have been developed, based on studies conducted in the laboratory and in simulation and operational environments. The more important indices that have been employed in the assessment of mental workload in applied settings are respiration, heart rate, electrodermal activity, some components of electroencephalic activity, and some parameters related to eye activities (Kramer & Weber, 2000).

In the last twenty years, the commonly used indices of mental workload have been heart rate and respiration (Wientjes, 1992; Jorna, 1992). These have been used in a variety of experimental settings, from the early experiment of Gemelli (1917) involving real-flight performance of military pilots to the most recent work of Pattyn et al. (2008) in a laboratory context (for a review, see Kramer & Weber, 2000). However, the use of heart rate has been often criticized (e.g., Kahneman 1973) and respiratory measurements are still rather uncommon in applied research. Furthermore, a number of technical problems associated with acquiring these two indices make it impossible to apply them in working environments. Two common sources of artifacts for these indices in applied settings come from speaking and muscle activity (Jorna, 1992; Porges & Byrne; 1992, Wientjes, 1992). It is impossible for an air-traffic controller or for refinery operators, for example, to be assessed without moving or speaking. Furthermore, as happens with most common physiological measures, cardiorespiratory indices fail to produce reliable results because they lack specificity and have hyper- hypo sensitivity (Roscoe, 1992). These problems are important because in applied contexts it is impossible to define stable task loads and it is therefore impossible to define clear causal relations (Mulder, 1992).

Skin conductance and other electrodermal measurements have been extensively used to study variations in cognitive demand during a task (e.g., Naccache et al., 2005; Gould et al., 2009; for more specific information about this measure, see Dawson et al., 2007). However, even if there are many advantages in using this technique, including low-weight devices and relative lack of intrusiveness, an important problem makes the application of this technique outside of a laboratory context impossible. Indeed, as happens when measuring heart rate and respiration, the body creates noise that interferes with skin activity recording (Min et al., 2002; Yoshino et al., 2007).

Cerebral activity measurements, such as functional magnetic resonance imaging (fMRI) or electroencephalography (EEG), provide an opportunity for a more direct assessment of mental workload (Ryu & Myung, 2005). Several studies have shown that the fMRI signal in regions that are sensitive for workload (the prefrontal and parietal cortices) increases as the demands imposed on working memory increase (e.g., Rypma et al., 1999; Jansma et al., 2000). However, EEG and fMRI are difficult to monitor outside specialized laboratory environments. Functional neuroimagingtechniques require massive machinery, large multidisciplinary teams of technicians, and in some cases complete immobilization of the subject (Gevins & Smith, 2003). This makes assessment in industrial operations close to impossible.

Furthermore, despite the promising results from some studies using psychophysiological indices, the types of tasks studied in research settings, such as performing verbal calculations, observing photographs, or troubleshooting, make it difficult to extend the findings to real-life ergonomic situations. Also, these psychophysiological indices have been obtained in well-controlled experimental laboratory settings that are very different from the natural settings in which cognitive ergonomists and human factors professionals hope to assess. For example, Murata (2005) recently showed that EEG signals could be analyzed to discriminate cognitive task loads, with increasing cognitive task load seeming to delay the time at which the central nervous system works most actively (the appearance time of the θ, α, and β frequency bands increased as task difficulty increased). However, this author used an experimental task in which participants had to simply indicate whether a current stimulus matched a stimulus presented on a previous trial.

Therefore, we need to look for an alternative method to evaluate mental workload, one that can be used in more ecological and less artificial conditions and that avoids the problems mentioned above. In this paper, we present some experimental data in favour of such an alternative method, using a parameter of eye movement activity known as saccadic peak velocity (PV). Our proposal is based on the assumption that brain activity offers the best estimation of mental workload, and since the eye is, embryologically, an extension of the brain (Wilson & O’Donnell, 1988), some eye activity parameter may be a suitable index. We propose that videooculography may be an optimal solution because it supplies a continuous, reliable measurement that can monitor unexpected and continuous changes in mental workload.

OCULOMOTOR INDICES

Eye tracking has been a tool in Human Factors since the 1950s, when for instance eye movements of pilots flying landing approaches were studied. Ocular movements are often studied to understand perceptual‐cognitive processes and strategies mediating performance in complex tasks (Parasuraman & Rizzo, 2007).

In recent years, thanks to technological progress (which reduced intrusiveness), eye tracking methods for mental load assessment have received increasing attention. The basic assumption is that brain activity offers the best estimation of mental workload and since the eye is an extension of the brain (Wilson & O’Donnell, 1988) ocular indexes can reflect changes in mental activity related to the task being performed.

Eye tracking introduces three new potential sources of information about user mental workload: blink rate, pupil size, and parameters related to eye dynamics. Moreover, gaze behaviour (durations and directions) have been traditionally studied. Pupil diameter and blink rate are the most popular eye-movement indices for mapping mental workload (Wickens & Hollands, 2000; Ahlstrom & Friedman-Berg, 2006).

The positive relationship between cognition and pupil dilation has been well established by psychological research (Marshall, 2007), and some researchers have also found a link between eye blinks and cognition (e.g., Ryu & Myung, 2005). However, some problems make these indices difficult to use with dynamic and complex tasks. For example, using blink rate as an indicator of mental workload is problematic because with the closed eye a lot of information is lost. According to Velichkovsky et al. (2002), if we consider only the number of blinks that are made in a minute, a person is ‘blind’ for up to 4% of the time; this means that during a complex task, such as air traffic control, information is lost for approximately 15 minutes (considering dutytimes for an air traffic controller in a busy daytime) (Gander, 2001). Considering that the majority of errors are caused by inattention and that blink durations and rate increase as functions of time-on-task [TOT] and fatigue (Morris & Miller, 1996), it should be recognized that measures that avoid information blindness is needed. We need to detect dangerous levels of fatigue before inattention or errors happen. For this reason, among others, most researchers have directed their attention to pupil diameter.

Several researchers have demonstrated a strong correlation between variations in pupil amplitude and the amount of cognitive resources used to perform a task (Kahneman & Beatty, 1966; Beatty & Lucero-Wagoner, 2000; Le Duc et al., 2005). However, recent data cast doubts on the validity of this index for studying human– machine interactions. Schultheis and Jameson (2004) measured pupil diameter while people read texts of different difficulty, to evaluate the validity of this index in adaptive hypertext systems. They found no significant differences in pupil diameter when text difficulty was changing. More recently, Conati and Merten (2007) explored the validity of pupil diameter for online assessment of user meta-cognitive behaviour during exploration based learning. They also found that pupil size was not a reliable predictor of mental workload.

From the early years of ergonomic research, gaze measures, as gaze duration or gaze direction, were often used to assess mental states of operators (for more details see Kramer & Parasuraman, 2007). As recent studies have confirmed it is known that visual scanning behaviour is sensitive to variations in MW. For example, Di Stasi et al. (submitted) have reported an experimental study in which participants interacted with an e-commerce website in two searching tasks (goal-oriented shopping and experiential shopping), each demanding different amounts of cognitive resources. In this study, results showed visual scanning behaviour coincided with subjective test scores and performance data in showing a higher information processing load in goal- oriented shopping. We may assume that experiential shopping is easier than goal- oriented shopping. Therefore, in experiential shopping there is an optimal level of arousal and, consequently, better planning of the visual behaviour. On the other hand, in goal-oriented shopping the level of arousal is higher due to the task aim: to buy a product taking care of specific features (under the same temporal constraint). Even if the above cited investigation confirmed that MW affects eye fixation variability, the relation between attentional state and fixation duration is still not clear. Some authors have shown an increase in fixation duration under high MW conditions (e.g. Bellenkes& Wickens, 1997); others have found the opposite result; i.e., that more frequent fixations reflect additional effort when processing visual information load (Camilli et al., 2008).

Nevertheless, eye movement parameters related to saccadic movement dynamics have received less attention from researchers investigating mental workload. It could be the case that some parameters related to saccadic movements are influenced by mental workload and that such a parameter could be a good alternative to blinking rate or pupil size for measuring mental workload in natural settings.

THE MAIN SEQUENCE

The relationships between duration and magnitude and between peak velocity and magnitude over a wide range of human saccades are indicated as Main Sequence, and they have been used to interrelate several hypotheses concerning the generation and control of saccades (Bahill et al., 1975). During the first 20 ms (more or less) of a saccadic movement, velocity tends to be the same regardless of target position. However, for the next 80 ms (or more), target position affects saccadic acceleration, which increases up to a point before velocity declines slightly and is maintained until reaching the target. Peak velocity is the point at which acceleration turns to negative, namely the point at which maximum saccade velocity is reached. It is independent of saccade duration since it is not linked a priori to it by a mathematical relation. Peak velocity measurement does not depend on thresholds used to define start and end points of a saccade, while saccade duration does (Becker, 1989). Saccadic parameters have been used as markers of task performance (Galley, 1998). Moreover, empirical research has also shown relations between saccadic dynamics and activation state (Galley, 1989; 1993; Galley & Andrès, 1996; Schleicher et al., 2008). While saccadic amplitude and saccade latency can be used as indicators of performance, saccadic speed is related to the activation state in visual performance tasks (App & Debus, 1998). It is also related to mental fatigue (Schmidt et al., 1979) and road accidents (Di Stasi et al., 2009). It is therefore reasonable to hypothesize that, in visual tasks of long length (e.g. monitoring of devices), saccadic speed could change according to the state of activation. However, as App and Debus (1998) have suggested, saccadic (average) velocity has not been used as an indicator of mental state because it is strongly dependent on saccadic amplitude and orbit direction, two variables that in real contexts are usually uncontrolled. For this reason, PV, which is not linked a priori by a mathematical formula to saccadic amplitude or duration, could represent a possible sensitive index of task complexity.

SACCADIC VELOCITY: THE ORIGINAL WORKS

During the first decade of the 20th century, Dodge and Cline (1901) and Dodge (1917), using a photographic technique of corneal reflexion recording, studied the dynamics of saccadic movements. The authors noted that saccade generation was influenced by the organism’s state of “arousal” and that it may be impaired (among all, reduction in angle velocity) by mental fatigue.

In the 1970s, Bahill and Stark (1975) concluded their work discussing the great potential utility of using saccadic eye movement indices as indicators of general psychological state while performing real tasks; notwithstanding, in the field of neuroergonomics, this suggestion still needs to be considered (Parasuraman & Rizzo, 2007; Schleicher et al., 2008).

Even if some authors studied the relationship between performance indicators and activation indicators, such as saccadic behavior (for more details see Galley, 1998), researchers have generally designed experiments such that oculomotor performance is dissociated from the natural role of the saccades; i.e. to make crucial perceptual information rapidly available for high resolution (Montagnini & Chelazzi, 2005). For example, in visually complex and dynamic tasks, such as driving a car, saccadic eye movements play an important role, namely to direct the foveal gaze to the area of interest, which has direct consequences for task performance.

To the best of our knowledge, only a few researchers have investigated the saccadic dynamics in complex tasks such as driving simulations (Galley, 1993; Schleicher et al., 2008) or in real road environments (Galley & Andrès, 1996).

The aim of the original work of Galley (1993) was to test electro-oculograms [EOG] as a sensitive tool for measuring online driver gaze behaviour. Gaze andblinking behaviour were reordered during three different simulated conditions, differing on the “secondary task” that participants had to perform. The experimental design “forced” the participant’s gaze behaviour to look for information present in different positions on the dashboard of the simulator (for example, several digital displays or lateral mirrors). The results, considering the secondary task, showed that blink rate went up when the concurrent task finished and that saccadic velocity decreased according to mental fatigue (time-on-task). The author concluded that blinking behaviour could be a good indicator of visual behaviour interruption costs (derived by performance of the secondary task) and that saccadic velocity could represent a sensitive index of driver (de)activation. Even if this study represented the first great approximation of saccadic behaviour study in complex and dynamic conditions, the use of the EOG, induced to “force” participant gaze behaviour, created unnatural driver interaction (by defining several distant targets from the projected road simulation).

Some years later the investigation of Galley and Andrès (1996) overcame the caveats of Galley’s work. Authors applied the same methodology (now including fixation durations among variables) to study the effect of long-term driving (drowsiness) on motorways in natural driving conditions (without forcing any secondary task). Fifteen participants drove at least 6 hours per day during 5 days. In this study, authors used the saccadic parameters as indicators of changing information processing, manipulating three main factors: the road environment (city vs. motorway); tiredness (time-on-task: five to one hour blocks) and consumption of alcohol (0 mg vs. <0.5 mg). Authors found effects with city vs. motorway and with vigilance (time-on-task and alcohol intake). Regarding saccadic velocity, a clear increase of mean values reflected an increase of information processing while driving in the city, and only moderated decrease of it, due to vigilance changing (reduction) in other cases.

Finally, Schleicher et al. (2008) examined the changes in several oculomotor variables (including main sequence parameters and blinking behaviour) as a function of increasing sleepiness in simulated traffic situations. Also, in this experiment, the EOGs were used as a psychophysiological measuring instrument. Participants had to drive for about two hours in a monotonous road circuit, without any secondary task. The results showed that blinking behaviour (blink duration, delay of lid reopening, blink interval and standardised lid closure speed) was the best indicator of subjective as well as objective sleepiness. Among saccadic dynamics parameters, mean saccadic duration showed only modest changes with increasing sleepiness.

These studies are quite similar. Without considering the experimental context and psychophysiological measuring instrument (EOG), researchers used a standardization procedure Schleicher et al. (2008) to eliminate the influence of changing amplitudes on saccadic duration and velocity, and both considered the mean saccadic velocity as a third element of the main sequence relation, showing a general decrease of mean saccadic velocity as driver fatigue (or deactivation) increases.

SACCADIC (PEAK) VELOCITY AS ATTENTIONAL INDEX IN COMPLEX TASKS

In this short overview, we present data from an ongoing research project on the validity and sensitivity of the saccadic main sequence, and in particular PV (instead of saccadic mean velocity) as an attentional state index, in several experimental settings (from simulated air traffic control tasks to driving simulator sessions).

There are three main differences compared to Galley’s works (see above): 1) the psychophysiological measuring instrument (in this case video-oculography at 500 Hz of sample rating - Eye Link II system), 2) the procedure of data analysis. In their works Di Stasi and colleagues used the saccadic-bin analysis (i.e. the analysis of PV and saccadic duration as a function of saccade length, in order to control the influence of saccadic magnitude). 3) Authors analyzed the saccadic peak velocity (that was moresensitive) instead of the average saccadic velocity.

In each experiment, Di Stasi and colleagues (see below) evaluated the variation of main sequence parameters (saccadic amplitude, duration and PV) induced by task load manipulations, multidimensionally, using subjective ratings, performance in primary or secondary tasks and other behavioural indices.

In an experiment that simulated multitasking performance in an ATC setting, Di Stasi et al. (2010b) studied the relation between the main sequence parameters and task load. The created tasks demanded different perceptual and central processing resources, as well as response resources. Three different levels of task complexity (low, medium and high) were created by manipulating the number of simultaneous tasks to be carried out. The number of simultaneous tasks was assumed to affect information processing load (Wickens, 2002). Low task complexity was defined as a monitoring/decision task. Medium task complexity involved a low-complexity task and a digit code task. In the high task complexity condition, the task was a combination of low- and medium-complexity tasks and a mathematical operation (paper and pencil) secondary task. All subjects started with low task complexity and proceeded through the same order. Results obtained from the subjective ratings Mental Workload Test [MWT, for more details see Di Stasi et al., 2009] and behavioral measures (number of errors and delayed answers) confirmed that MW levels varied according to task demand. These different levels of MW were reflected in PV values. The authors found that there was a 6.3 º/s reduction in PV when task complexity assessed by MWT increased by 10.6 and performance was also affected (6 delayed answers). However, there was one limitation in this work. The authors were unable to distinguish between the effects of task complexity and time-on-task [TOT], probably due to the nature of the experimental design. Indeed, to avoid any effect of task switching during the experimental session, the order of task complexity levels was not balanced across participants.

On the basis of these results, authors designed a well-controlled experiment to surmount the methodological problem encountered in the previous study, and particularly the influence of TOT on the disruption of the main sequence rules. In this research (Di Stasi et al., submitted), screen visual configuration was manipulated between groups to create two different levels of task complexity. Also the TOT was manipulated (within group), participants had to perform the same simulation (eye movement and subjective measure were recorded during the 1st and 10th trial) ten times. In more detail, the Firechief (Omodei & Wearing, 1995; Cañas et al., 2003; Cañas et al., 2005) incident simulator (microworld) was chosen as a complex and dynamic problem-solving task. Microworlds are an appropriate research environment to test hypotheses about MW because they are based on simulations of real tasks that change dynamically and are designed to reproduce the important characteristics of real situations. The experiment was set up to test the validity and sensitivity of PV compared with the results of performance and subjective ratings in response to the main manipulations. Within the limits imposed by the experimental task and the sensitivity of the no-psychophysiological measurements; i.e. the different demands on cognitive resources were not reflected in different levels of performance or in different MWT ratings, probably due to the general ease of the task, PV was sensitive to the manipulations of the screen configuration and TOT. Consistent with previous studies, we found that saccadic movements were lower (276 º/s vs. 290 º/s) while mental workload was higher (Di Stasi et al., 2010a; Di Stasi, 2010b), and with an increase of time on task; i.e. tiredness (1st trial 287 º/s vs. 10th 279 º/s), as could be expected (DeLuca, 2005).

Finally, similar results were obtained by Di Stasi et al. (2010a) in a more complex experimental setting. In this study, the authors demonstrated that PV was sensitive to variations in MW during ecological driving tasks, showing again an inverse relation between PV and task complexity. In this experiment, three different levels oftask complexity (low, medium and high) were created by manipulating traffic density and adding a secondary task. Traffic density could either be low (no other cars) or high, and it was assumed to affect the load of information processing (Wickens, 2002). As a secondary task, potentially hazardous situations were produced by pop-up events that appeared in the central field of view. The events were red squares and rectangles. Each type of event was associated with a specific reaction. Participants were asked to press the left or the right button of the steering wheel in response to the squares and rectangles, respectively. Low task complexity was defined as driving with low traffic density and no secondary task. Medium task complexity involved low traffic density and a secondary task. In the high complexity task condition, high traffic density was combined with the secondary task. To disentangle mental workload and fatigue, the order of tasks was varied in the following way: all subjects started with low task complexity, while the order of medium and high complexity tasks was balanced across the subjects. Results showed that PV decreased by 7.2 º/s as the MWT score of MW increased by 15.2 and reaction time for the secondary task increased by 46 msec. Saccade duration and velocity were not affected by differences in task complexity. The design of this experimental investigation allowed the authors to differentiate between the effects of TOT and changes in MW from the same dataset. In this experiment, no effect of fatigue was found. Even if the analyses for influences of TOT on PV revealed no effects, the authors suggested that the relationship between fatigue and MW requires further investigation in a more controlled experimental setting.

Figure 1: Examples of screenshots used (left) and a resume of the relative experimental design(right).

CONCLUSION

In a technological information society, changes in mental workload can have significant impact on operator performance, possibly leading to delays in information processing or even causing operators to ignore incoming information (Ryu & Myung, 2005). Consequently, there is a need to monitor operator functional states in real time to determine the type and level of automated assistance most appropriate in helping operators to complete tasks (Langan-Fox et al., 2009). Development of a method able to detect operator attentional states in real time during interactions with complex and dynamic systems could be a good starting point for this critical issue.

On the basis of these confirmations, we investigated all parameters of the saccadic main sequence (duration, amplitude and PV) with respect to mental state. We found a clearer effect of task complexity for PV only. Based on these findings, we suggest that saccadic peak velocity could be a useful general indicator for assessment of operator mental workload and attentional state in highly dynamic environments (Di Stasi et al., 2009; 2010a; 2010b; submitted).

Theoretically, our results are compatible with an explanation of mental activation in terms of a multiple resources model developed by Wickens (2002), which provides a framework for workload assessment intimately related with human information processing. The model is able to provide an explanation for mental activity changes that can occur when operational conditions change, for instance, task difficulty and time-on-task. In our experiments, the global results could be explained with a reduction of arousal state, due to the increase of task complexity and time-on-task that is reflected in a reduction of saccadic peak velocity.

Our research is relevant to a variety of domains, ranging from ATC towers to call centres. For example, using real-time main sequence measures, neuroergonomists could better evaluate when an operator’s attentional state is changing (mental under/overload), helping in the design of systems able to allocate tasks in a dynamic way between the operator and the machine. The current work could be a starting point for further research on understanding how variations in cognitive processing and mental workload can be used to build a model to manage the dynamic allocation of tasks between human operators and support systems (Crévits et al., 2002). We are now trying to validate these results, applying the same methodology during complex interactions task, to develop an assistance system able to reduce the risk of accidents by assessing operators’ mental workloads and attentional states in hazardous conditions.

ACKNOWLEDGEMENTS

This study was in part supported by Spanish national project SEJ2007- 63850/PSIC (J.J. Cañas), and by national grant FPU AP2006-03664, awarded to the first author.

REFERENCES

Ahlstrom, U., & Friedman-Berg, F.J. (2006). Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics, 36, 623-636.

App, E., & Debus, G. (1998). Saccadic velocity and activation: development of a diagnostic tool for assessing energy regulation. Ergonomics, 41, 689-697.

Bahill, A. T., & Stark, L. (1975). Overlapping saccades and glissades are produced by fatigue in the saccadic eye movement system. Experimental Neurology, 48, 95- 106.

Beatty, J., & Lucero-Wagoner, B. (2000). The pupillary system. In J.T. Cacioppo, L.G. Tassinary, and G.G. Bernston (Eds.), Handbook of Psychophysiology (pp. 142- 162). N.Y.: N.Y., Cambridge University Press.

Becker, W. (1989). Saccadic eye movements as a control system: Metrics. In R.H, Wurtz, and M.E., Goldberg (Ed.), Reviews of Oculomotor Research, vol 3: The Neurobiology of Saccadic Eye Movements (pp. 13-67). Elsevier, Hillsdale.

Bellenkes, A.H. , Wickens, C.D., & Kramer, A.F. (1997). Visual scanning and pilot expertise: The role of attentional flexibility and mental model development. Aviation, Space, and Environmental Medicine, 68, 569-579.

Camilli, M., Nacchia, R., Terenzi, M., & Di Nocera, F. (2008). ASTEF: A simple tool for examining fixations. Behavior Research Methods, 40, 373-382.

Cañas, J.J., Antolí, A., Fajardo, I., & Salmerón, L. (2005). Cognitive inflexibility and the development and use of strategies for solving complex dynamic problems: effects of different types of training. Theoretical Issues in Ergonomics Science, 6, 95- 108.

Cañas, J.J., Quesada J.F., Antolí, A., & Fajardo I. (2003). Cognitive flexibility and adaptability to environmental changes in dynamic complex problem-solving tasks. Ergonomics, 46, 482-501.

Christodoulou, C., 2005. The assessment and measurement of fatigue. In DeLuca J. (Ed.), Fatigue as a Window to the Brain (pp. 19-35). New York: MIT Press.

Conati, C., & Merten, C. (2007). Eye-Tracking for User Modeling in Exploratory Learning Environments: an Empirical Evaluation. Knowledge Based Systems, 20, 557-574

Crévits, I., Debernard, S., & Denecker, P. (2002). Model building for air-traffic controllers workload regulation. European Journal of Operational Research, 136, 324-332.

Dawson, M.E., Schell, A.M., & Filion, D.L. (2007). The electrodermal system. In J.T. Cacioppo, L.G. Tassinary, and G.G. Berntson (Eds.), Handbook of Psychophysiology (pp. 160-181). New York: Cambridge University Press.

Dodge, R., & Cline, T.S. (1901). The angle velocity of eye movements. Psychological Review, 8, 145-157.

Dodge, R. (1917). The laws of relative fatigue. Psychological Review, 14, 89-113.

Di Stasi, L.L., Alvarez, V. Cañas, J.J., Maldonado, A., Catena., A., Antolí, A., & Candido,

A.  (2009).  Risk  Behaviour  and  mental  workload:  multimodal  assessment techniques applied to motorbike riding simulation. Transportation Research, Part

F, 12, 361-370.

Di Stasi, L.L., Renner, R., Staehr, P., Helmert, J.R., Velichkovsky, B.M., Cañas, J.J., Catena, A. & Pannasch, S. (2010 a). Saccadic peak velocity is sensitive to variations in mental workload while driving in a virtual environment. Aviation, Space, and Environmental Medicine, 81, 413-417.

Di Stasi, L.L., Marchitto, M., Antolí, A., Baccino, T., & Cañas, J.J. (2010 b).

Approximation of on-line mental workload index in ATC simulated multitasks.

Journal of Air Transport Management. DOI: 10.1016/j.jairtraman.2010.02.004.

Di Stasi, L.L., Antoli, A., Gea, M., & Cañas, J.J. Submitted. A neuroergonomic approach to evaluating mental workload in hypermedia interactions.

Galley, N. (1989). Saccadic eye movement velocity as an indicator of (de)activation. A review and some speculations. Journal of Psychophysiology, 3, 229-244.

Galley, N. (1993). The evaluation of the electrooculogram as a psychophysiological measuring instrument in the driver study of driver behaviour. Ergonomics, 36, 1063-1070.

Galley, N. (1998). An enquiry into the relationship between activation and performance using saccadic eye movement parameters. Ergonomics, 41, 698-720.

Galley, N., & Andrés, G. (1996). Saccadic eye movements and blinks during long term driving on the autobahn with minimal alcohol ingestion. In A.G. Gale, I.D. Brown,

C.M. Haslegrave, and S.P. Taylor (Eds), Vision in Vehicles V (pp. 381-388). Amsterdam: Elsevier.

Gander, F., (2001). Fatigue management in air traffic control: the New Zealand approach. Transportation Research Part F, 4, 49-62.

Gemelli, A. (1917). Sull’applicazione dei metodi psico-fisici all’esame dei candidati all’aviazione militare. (On the application of psycho-physical methods of examining candidates as military pilots). Milan: Vita e pensiero.

Gevins, A.S., & Smith, M.E. (2003). Neurophysiological measures of cognitive workload during human-computer interaction. Theoretical Issues in Ergonomics Science, 4, 113–131.

Gould, K.S., Røed, B.K., Saus, E.R., Koefoed, V.F., Bridger, R.S., & Moen, B.E. (2009).

Effects of navigation method on workload and performance in simulated high- speed ship navigation. Applied Ergonomics, 40, 103–114.

Jansma, J.M, Ramsey, N.F., Coppola, R., & Kahn, R.S. (2000). Specific versus nonspecific brain activity in a parametric N-back task. Neuroimage, 12, 688–697.

Jorna, P.G.A.M. (1992). Spectral analysis of heart rate and psychological state: a review of its validity as a workload index. Biological Psychology, 34, 237–257.

Kahneman, D. (1973). Attention and effort. Englewood Cliffs, New Jersey: Prentice Hall.

Kahneman, D., & Beatty, J. (1966). Pupil diameter and load on memory. Science, 154, 1583-1585.

Kramer, A.F., & Parasuraman, R. (2007). Neuroergonomics: Application of Neuroscience to Human Factors. In J.T. Cacioppo, L.G. Tassinary and G. Bertson (Eds.), Handbook of Psychophysiology. (pp. 704-721). N.Y.: N.Y., Cambridge University Press.

Kramer, A.F., & Spinks, J.A. (1991). Capacity views of information processing: Central nervous systems measures. In: J.R. Jennings and M.G.H. Coles (Eds.), Handbook of cognitive psychology: Central and autonomic nervous system approaches (pp. 194-208). New York: Wiley.

Kramer, A.F., & Weber, T. (2000). Applications of Psychophysiology to Human Factors.

In J. T. Cacioppo, L.G. Tassinary, and G. Bertson (Eds.), Handbook of Psychophysiology. (pp. 794-814). N.Y.: N.Y., Cambridge University Press.

Langan-Fox, J., Canty, J.M., & Sankey, M.J. (2009). Human-automation teams and adaptable control for future air traffic management. International Journal of Industrial Ergonomics, 39, 894-903.

Le Duc, P.A., Greig, J.L., & Dumond, S.L. (2005). Involuntary eye responses as measures of fatigue in U.S. Army Apache aviators. Aviation, Space, and Environmental Medicine, 76, 86-92.

Marshall, S.P. (2007). Identifying cognitive state from eye metrics. Aviation, Space, and Environmental Medicine, 78, 165-75.

Min, B.C., Chung, S.C., Park, S.J., Kim C.J., Sim, M.K. & Sakamoto, K. (2002).

Autonomic Responses of Young Passengers Contingent to the Speed and Driving Mode of a Vehicle. International Journal of Industrial Ergonomics, 29, 187-198.

Montagnini, A., & Chelazzi, L. (2005). The urgency to look: Prompt saccades to the benefit of perception. Vision Research, 45, 3391–3401.

Morris, T.L., & Miller, J.C. (1996). Electrooculographic and Performance Indices of Fatigue during Simulated Flight. Biological Psychology, 42, 343-60.

Mulder, L.J.M. (1992). Measurement and analysis methods of heart rate and respiration for use in applied environments. Biological Psychology, 34, 205-236.

Murata, A. (2005). An attempt to evaluate mental workload using wavelet transform of EEG. Human Factors, 47, 498-508.

Naccache, L., Dehaene, S., Cohen, L., Habert, M., Guichart-Gomez, E., Galanaud, D., & Willer, J., (2005). Effortless control: executive attention and conscious feeling of mental effort are dissociable. Neuropsychologia, 43, 1318–1328.

Omodei, M.Y., & Wearing, A. (1995). The Firechief Microworld generating program: An illustration of computer simulated microworld as an experimental paradigm for studying complex decision-making behaviour. Behaviour Research Methods, Instruments and Computers, 27, 303-316.

Parasuraman, R., & Caggiano, D. (2002). Mental Workload. In V.S. Ramachandran (Eds.), Encyclopedia of human brain (pp. 17-27). San Diego, CA: Academic Press.

Parasuraman, R., & Rizzo, M. (2007). Neuroergonomics: The Brain at Work. UK: Oxford University Press.

Pattyn, N., Neyt, X., Henderickx, D., & Soetens E. (2008). Psychophysiological investigation of vigilance decrement: Boredom or cognitive fatigue? Physiology and Behavior, 93, 369-378.

Porges, S.W., & Byrne, E. A., 1992. Research methods for measurement of heart rate and respiration. Biological Psychology, 34, 93-130.

Roscoe, A.H. (1992). Assessing pilot workload. Why measure heart rate, HRV and respiration? Biological Psychology, 34, 259-287.

Rypma, B., Prabhakaran, V., Desmond, J. E., Glover, G. H., & Gabrieli, J. D. (1999).

Load-dependent roles of frontal brain regions in the maintenance of working memory. NeuroImage, 9, 216–226.

Ryu, K., & Myung, R. (2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics, 35, 991–1009.

Schleicher, R., Galley, N., Briest, S., & Galley, L. (2008). Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics, 51, 982- 1010.

Schmidt, D., Abel, L.A., Dell’Osso, F., & Daroff, R.B. (1979). Saccadic velocity characteristics: intrinsic variability and fatigue. Aviation, Space, and Environmental Medicine, 50, 393–395.

Schultheis, H., & Jameson, A. (2004). Assessing cognitive load in adaptive hypermedia systems: Physiological and behavioral methods. Lecture Notes in Computer Science, 3137, 225-234.

Tsang, P.S., & Velazquez, V.L. (1996). Diagnosticity and multidimensional subjective workload ratings. Ergonomics, 39, 358-381.

Velichkovsky, B.M., Dornhoefer, S.M., Kopf, M., Helmert, J., & Joos, M. (2002). Change detection and occlusion modes in static and dynamic road-traffic scenarios. Transportation Research, Part F, 5, 99-109.

Wickens, C.D. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomic Science, 3, 159-177.

Wickens, C.D., & Hollands, J.G. (2000). Engineering Psychology and Human Performance (3rd ed.). Upper Saddle River, New Jersey: Prentice Hall.

Wientjes, C.J.E. (1992). Respiration in psychophysiology: methods and applications.

Biological Psychology, 34, 179-204.

Wilson, G.F, & Eggemeier, F.T. (1991). Physiological measures of workload in multi- task environments. In D. Damos (Ed.), Multiple-task performance (pp. 329-360). London: Taylor and Francis.

Wilson, G.F., & O'Donnell, R.D. (1988). Measurement of operator workload with the neuropsychological workload test battery. In Hancock P.A., Meshkati, N. (Eds.), Human mental workload (pp. 63-100). Amsterdam: Elsevier.

Yeh, Y., & Wickens, C.D. (1988). Dissociation of performance and subjective measures of workload. Human Factors, 30, 111-120.

Yoshino, K., Edamatsu, M., Yoshida, M., & Matsuoka, K. (2007). An algorithm for detecting startle state based on physiological signals. Accident Analysis and Prevention, 39, 308–312.