An Enhanced Human-Machine Interface Enabled by Augmented Reality – A New Approach for Human Augmentation

3. Literature Review

This brief literature review approaches AR technology, HMI, human augmentation concept as well as augmented operator. These topics support the ongoing project and the new approach to be developed.

3.1. Augmented Reality

One of the emerging disruptive technologies that are being enabled by Industry 4.0 is AR, which intends to combine the physical word with computer generated texts and images or animations, providing an intuitive interaction experience to the users. This technology can be defined as a real-time direct or indirect view of an enhanced or augmented real world environment, combining real and virtual objects [7]. These objects are able to interact in real-time, allowing higher work performance and efficiency in workplaces [8]. The aim of AR is supplementing physical environment through the overlaying of digital computer generated information, such as, sound , images, video and graphics [14], simplifying user’s experience and enhancing their perception and  interaction with physical world by augmenting their sense of reality in real-time [7].

This technology is often related with Virtual Reality (VR), since these are two closely related areas. However, AR and VR are different concepts. VR is a technology that completely immerses their user in a virtual environment generated by computers in a 360-degree views of a simulated world, together with other sensorial experiences. Moreover, Mixed Reality (MR) consists in the intersection between AR and VR technologies, intending to merge physical and virtual worlds and generating new environments where virtual and physical objects interact in real-time [14].

AR can also be understood ass a variation of VR, however, unlike VR whose goal is to totally immerse users in a virtual world, AR aims to augment the real world adding computer generated information over real objects [15]. This information is added three dimensionally, in order to create a visual space and assist human behavior and movements [9]. In other words, AR consists in VR technology applied to real world, while VR is represented only in a simulated environment, AR has a strong focus on the physical environment, augmenting the sense of reality through the attachment of computer-generated information [16].

In the last years, the way of representing work instructions and information to operators has been changing [17]. The emerging technological developments have allowed the transition from traditional paper-based instruction to the application of 3D visualization techniques in order to increase productivity, cost savings and control regarding error traceability. Furthermore, the use of advanced visualization technologies allows real-time changes in instructions and processes available to be seen by workers, avoiding wastes caused by delays and errors [18]. There are many studies about the effectiveness of this technology to represent work instructions and relevant information to operators [19]. The integration of disruptive visualization techniques in work instructions can provide a higher level of quality, reduced times, more precise task performance, and cost effectiveness [20].

Over the last years, advanced techniques, such as AR, are being introduced in industrial environments in order to enhance visualization capabilities. One of the most usual application of AR technology is industrial environment consists in the use of wearable devices based on Head-Mounted Displays (HMD), such as smart glasses and helmets. Wearable devices hold a particular interest to industrial applications, providing hand-free solutions that allow users to communicate effectively with the physical world [21]. Furthermore, mobile hand-held displays, such as smartphones or tablets provide real-time interaction into one single device that overlays real environment by graphical augmentations [22]. The ubiquity and advanced features of mobile devices provide a good opportunity to implement AR in industrial environments for tasks automation and mitigation of information availability deficit for workers [23]. These devices have a large number of applications being more socially accepted when compared with HMD. Due to their easy transportation, these devices are widely used in industry, capturing the real environment through the device camera and providing superimposed real-time information on the display [24]. Spatial Augmented Reality (SAR) has been introduced by Bimber and Raskar [22] as a solution to merge real and virtual worlds. This technology proposes a projector-based AR that can overcome some limitations of the abovementioned types of AR. Despite of wearable and hand-held devices holding huge potential to industrial applications, these solutions are not group-oriented and do not facilitate social interactions. Projector-based AR or SAR can overcome this social gap, creating a space-efficient and seamless visual displays that are able to merge augmented physical objects and real environment in a shared workplace [25].

However, AR technology holds a huge potential to augment all human senses and capabilities, being not limited to sight sense [26]. The concept of AR audio, or augmented audio reality, is characterized by an extended real sound environment, where virtual and real sounds are mixed, allowing augmentation of hearing sense, in order to perceive virtual sounds as an extension to the natural ones, creating a hybrid augmented environment [27]. On the other hand, AR technology allows humans to become stronger and safer in manufacturing environment. Exoskeletons are wearable robots directly controlled by their users that hold a great potential in human physical capabilities augmentation regarding to strength, endurance, durability and speed [28]. This technology consists in a robotic extension of human body, helping in overcoming disabilities or enhance physical performance and capabilities in workplaces, supporting existing human limbs and replacing the lost ones [29]. The creation of super-strong humans in industrial environment is allowed by the use of wearable, lightweight, flexible and mobile exoskeletons that are enable by a biomechanical system powered by motors, pneumatics, levers or hydraulics that provide a cooperative human-robotic system [30].

AR is an emerging technology that intends to seamlessly enhance physical environments with virtual objects, creating a bridge between virtual and physical world and bringing them closer together. However, the importance of the interaction between human and machine has been growing as the technology develops. There are a lot of critical factors that have to be considered during the design of a technological solution that interacts with workers in an industrial environment. It is vital to ensure the accuracy of the provided information in order to allow a correct understanding of the instructions and simplify tasks execution. A lack of communication between machine and human can result in error that can pledge the quality of final product or service. For this reason, it is essential to make sure that the system provides every important information to the workers, in order to avoid human errors [31].

3.2. Human- Machine Interface

Moving toward Industry 4.0 paradigm and implementing AR will enable new type of interaction between humans and machines and, eventually, transform the current industrial workforce and workplace. The work environment is rapidly changing in the last few years due to disruptive technological advancements and Industry 4.0 is transforming jobs and required skills.

The most significant changes regards the new HMI paradigm that embraces the interaction between workers and a set of new ways of collaborative work [5]. This new paradigm will lead to deep impacts in worker tasks and demands in work environment, which will be characterized by the cooperation between smart machines and humans [10], [11]. The number of robots and smart machines is increasing, while physical and virtual worlds are merging, which means that a significant transformation is being launched in the current work environment. The increasing relevance of HMI will promote the interaction between both production elements and the required communication between smart machines, smart products and employees, enhanced by the vision of IoT and IoS that is enabled by CPS.

For that reason, ergonomic issues should be taken into account in the context of industry 4.0 and future systems should have a focus on workers and their importance [32], [33]. The integration of Industry 4.0 technologies, namely AR, in manufacturing systems and the increasing implementation of new technologies will have an impact on job profiles, as well as on work management, organization and planning. The main challenge in this context is to avoid what is known as technological unemployment, redefining current jobs and taking measures to adapt the workforce for the new jobs that will be created [34].

To mitigate these impacts, there are several aspects to consider for AR implementation. The AR tools must be developed with functionalities that allow a user-friendly collaboration between human and technology, in order to enhance their experience and improve their performance and awareness in a non-intrusive way. Thus, it will be possible to meet the industrial requirements, allowing people to be more efficient and effective in their tasks [35]. Regarding to human factors, the adoption of AR demands further analysis, since there are some critical factors such as the use of wearables, fatigue effects and optical quality [36]. Furthermore, during the design process of an AR application, there are some important issues to take into account regarding HMI. It is crucial to define who are the users, their needs, system effectiveness metrics, tasks to be performed and user’s capabilities [37].

In order to achieve a human-machine symbiosis that allows higher workforce capabilities and increased manufacturing flexibility in future production systems, it is essential taking into account several aspects regarding technical and economic benefits for companies, such as, higher quality, shorter production times, optimized processes, increased responsiveness and innovation and continuous improvement capacity. However, workers should be the focus of every manufacturing systems and social-human benefits for workforce should be considered, including well-being and quality of working life, job satisfaction, improved ability and skills and higher personal flexibility and adaptation [10].

In augmentation and enhancement of human performance context, Human Cyber-Physical Systems (H-CPS) are the new approach for HMI, bringing together digital and physical worlds. H-CPS aim to achieve higher safety systems for workers, providing a sustainable human-centric production system where humans, machines and software dynamically interact within a cyber-physical world [38]. Based on this context, H-CPS are designed to improve human abilities in order to interact with smart machines within a smart factory which are engineered to fit operator’s cognitive and physical need. Furthermore, these systems intend to enhance cognitive capabilities through the use of technologies, such as wearable devices [10].

Lean Production, an organizational approach that resulted from the Toyota Production System (TPS) [13], plays and important role ensuring safety and enhanced human factors in this context. Ohno [13] has considered wastes the activities do not add value to the products in a client point of view, classifying them into seven categories: overproduction; over processing; transportation; defects; motion; inventory and waiting. Later on, Liker [12] has identified an extra waste, i.e. untapped human potential. Nevertheless, already in 1977, TPS has been defined as a respect-for-human system in a first published English paper because workers were allowed to apply their full potential and actively participate in improving their own workshops [39]. TPS main goal is “doing more with less”, where less means less human effort, less stocks, less resources, less space, less operation time. For this reason, was called Lean Production [40]. In other words, TPS was designed in a way that fewer and fewer resources would be required, in order to deliver the right products at the right time and at the shortest possible deadline, through the elimination of all types of wastes. Taking this into account, TPS also enables a greater production flexibility, while meeting quality standards and deadlines. Additionally, it tries to enable a greater production flexibility, while meeting quality standards and deadlines. After decades, enhancing Lean Production solutions represents a huge potential for current industrial landscape and Lean Automation, which consists in the automation integration into Lean Production, brings several opportunities for the smart factories context [41].

In HMI context, it is crucial to ensure that people develop their tasks without wastes and symptoms of wastes (muda, in Japanese). Beyond muda, there are the mura and muri that are considered the symptoms of muda. Muri is the overburden or stress or, even, accidents that could occur in the workplace due to other symptom of waste, the unevenness, i.e. mura. All together these three Japanese words are called 3M [12]. As so, Lean impacts ergonomics of workplaces [42]. To systematically eliminate wastes, Womack and Jones [43] have designed the Lean principles: Value; Value Stream; Flow; Pull production and Pursuit of Perfection. These principles happen cyclically and allow the continuous improvement, known as kaizen through the people heads, heart and hands (3H) because only people has the capability to think, promoting companies agility [44].

3.3. Human Augmentation

Human augmentation techniques rely on the use of technologies that are able to augment human actions, senses, capabilities and cognition, allowing humans to perceive the real environment in a new and enhanced way. Based on augmenting technologies, such as AR, VR and MR, relevant information is provided to operators, in order to enhance human life and allow new HMI solutions [45]. This approach is centered on the AR users and based on human-centered real world merged with an information world [9].

An augmented human has extended capabilities regarding physical, sensing and cognitive abilities [10]. Physical capabilities refer to operator’s capacity to undertake physical activities required for daily work, such as, lifting, walking, manipulating and assembling. Regarding enhancement of physical capabilities, human augmentation will allow the creation of super-strong workers encased in exoskeletons, being able to safely move and lift more heavy items. Exoskeletons in industrial environment allow humans and technology to cooperate in order to simplify tasks and reduce physical stress, while offering additional protection, support and strength to operators. Enhanced physical capabilities using exoskeletons technology provide improved ergonomic conditions, reduced injuries, accidents and safety risks, higher productivity and quality. Furthermore, with a reduced physical workload, operators can relocate their energy to sensorial and cognitive capabilities, which promotes the sustainability of the workforce, allowing people to perform their tasks longer [38]. Furthermore, people with special needs or elderly people will have their quality of life improved with these new solutions that allow new forms of human actions [45].

Alternatively, AR holds a great potential in sensory augmentation, which consists in the extension of human senses. The sensory augmentation relies on the use of IoT-enabled sensor devices that collect, convert and aggregate external signals that would not be accessible to operators, due to several reasons, such as available data, human limitations or personal limitations. These devices are able to transform one signal into another, allowing humans to identify relevant information and simplifying decision-making processes [10]. AR is able to augment every human sense, however, sight sense augmentation is one of the most studied field. Applications that convert light within a spectrum not visible to human eye into visible light, visualization of dark scenes through infrared or superimposed virtual images provided by wearables or projections are some examples about extensions of the human visual function. Nevertheless, augmentation is not limited to sight sense and other applications can extend hearing function by immersing users in an augmented environment by sound in place of information. Furthermore, the sense of touch can also be augmented providing textures, sensations or radiant temperature that provide information to operators [9].

On the other hand, cognitive capability relies on the ability to undertake mental tasks, such as memory, decision-making, responsiveness, perception and reasoning, that are essential to perform tasks [46]. In Industry 4.0 and smart factories context, the increasing demand for mental tasks can be address by AR technology and new approaches to HMI that support the increased cognitive workload, while considering operators well-being and performance and reducing mental stress [10]. It is possible to intellectually augment humans in a human-centered environment superimposing information into real world regarding work instructions, personal information, directions or safety instructions [9]. In this context, AR technology provides a new HMI, displaying real-time information to operators, which improves decision-making and create new interactions between humans and products due to the available information about them, which allows their configuration and monitoring [38].

However, human augmentation goes beyond the scope of AR, being able to augment spatial and temporal abilities. human augmentation in time and space is known as telexistence. This is a concept that regards to a technology, such as AR, that can free humans from the constraints of time and space, allowing them to experience a real-time perception of being in other place and interact with a remote real, virtual or mixed environment [9].

Operator 4.0 is a concept that has emerged in Industry 4.0 context and can be understood as a smart and skilled operator that performs collaborative work with machines and robots, being enabled by CPS and advanced technologies [38]. AR is a critical enabling technology for improving information transfer between digital world and smart operators in physical world [47].