Available online at www.sciencedirect.

com

Procedia Computer Science 15 (2012) 248 255

Virtual Worlds for Serious Applications (VS-GAMES'12)

Development of a mechanical maintenance training simulator in OpenSimulator for F-16 aircraft engines
Andr Pinheiroa, Paulo Fernandesa, Ana Maiab, Gonalo Cruzb, Daniela Pedrosac, Benjamim Fonsecad, Hugo Paredesd, Paulo Martinsd, Leonel Morgadod*, Jorge Rafaele
b a Dep. Engenharias, Escola de Cincias e Tecnologia, UTAD University of Trs-os-Montes e Alto Douro, 5001-801 Vila Real, Portugal Pro-Chancellery for Innovation & Information Management, UTAD University of Trs-os-Montes e Alto Douro, 5001-801 Vila Real, Portugal c Faculdade de Psicologia e de Cincias da Educao, UC University of Coimbra, 3000 Coimbra, Portugal d INESC TEC (formerly INESC Porto) / UTAD University of Trs-os-Montes e Alto Douro, 5001-801 Vila Real, Portugal e Portuguese Air Force Air Base Nr. 5, Serra de Porto de Urso, 2425-022 Monte Real, Portugal

Abstract Mechanical maintenance of F-16 engines is carried out as a team effort involving 3 to 4 skilled engine technicians. This paper presents the development of a mechanical maintenance simulator for their training. This simulator aims to enable technician training to be enhanced with cooperation and context prior to the training phase with actual physical engines. We describe the requirements that were identified with the Portuguese Air Force, the overall software architecture of the system, the current stage of the prototype, and the outcomes of the first field tests with users. 2012 The Authors. Published by Elsevier peer-review under responsibility of the scientic programme committee of 2012 Elsevier B.V...Selection and/or B.V. Selection and/or peer-review under responsibility of the scientific programme committee of VS-Games 2012 VS-Games 2012
Keywords: virtual worlds; OpenSimulator; virtual learning; cooperation; task coordination; aircraft engine maintenance

1. Introduction At the Portuguese Air Force, engine technicians go through an initial training process at the Centre for Military and Technical Training Air Force (CFMTFA), and are subsequently placed at different air bases, with specific engines and requirements. At each of these bases, they receive further training, focused on the specific engines and aircraft deployed and serviced there. In the case of the F-16 aircraft, this takes place at Air Base Nr. 5, near Monte Real. Since technicians may be re-deployed to other bases, training of technical procedures for maintenance of specific engines is a common and frequent process. The training process has an initial theory phase, based on -the-job training phase ensues, with trainees acting directly on an engine, in actual maintenance circumstances. This final on-the-job training phase is resource-demanding, since it requires engines to be available for training, and consequently not to be available for service. Also, procedure errors in training may in some cases produce costly component damage. Further, several of the technical procedures need to be executed by a team, meaning that time allocation of different trainees, trainers, and experienced technicians needs to be managed, in order for a full team to

* Corresponding author. Tel.: +351-259350369; fax: +351-259350356 E-mail address: leonelm@utad.pt.

Andr Pinheiro et al. / Procedia Computer Science 15 (2012) 248 255

249

be available for on-the-job training to ensue. These various resource requirements place constraints on the availability of on-the-job training opportunities and emphasize the need to optimize it. The development of a 3D multi-user mechanical training simulator for this scenario thus aims to provide trainees and trainers with more opportunities to conduct training, with the goal of allowing trainees to reach on-the-job training better prepared and thus to optimize the effectiveness of the resource-intensive training occasions with physical engines. This is a joint effort of the Portuguese Air Force and the University of Trs-os-Montes e Alto Douro (UTAD). From a software engineering perspective, there is also a goal of lessening the resource requirements of simulation development, by conducting it in a readily available virtual world platform (OpenSimulator) but implementing the control code and decision-making logic at an external system. The rationale for this architectural choice was to focus the software development effort on the simulation behavior in a way that could be rendered independent from the graphical platform of user interaction [2]. 2. Background F-16 aircraft of the Portuguese Air Force employ Pratt & Whitney F100-PW-220/220E engines, with large number of mechanical maintenance procedures. The team at UTAD conducted meetings with the training team and experts at air base Nr. 5, to ascertain the most relevant procedures for technicians that are initiating their training for this specific engine. Through these meetings, the procedures for installation of the engine inside the F-16 aircraft fuselage were selected as a first simulation target. This installation involves a series of steps for properly installing and connecting the engine, which need to be done not only effectively but also safely. We have collected data on this process by combining several sources: we reviewed the TOs [1] and taped and photographed the engine installation process from various perspectives. We then decoded this data, describing it in terms of a naturallanguage script, and creating UML diagrams of its steps, in regular contact with Air Force trainers to clear out doubts and details. Briefly, these steps involve preparing the engine for transportation towards the aircraft fuselage, transporting it and preparing it for insertion, its insertion into the aircraft fuselage, establishing the engine connections, and testing the installation. The level of detail of the simulation of each task was also determined in cooperation with the trainers at Air Base Nr. 5. For creating the simulation, given that this process moved on with minimal funding, it would be necessary for it to be modular, so that development could be incremental, in small steps over time, likely involving different people in each academic year. In the meetings with the training team at the air base, context-specific requirements were established. First, the main usage focus is for trainees in a training room at the air base, but with flexibility for later use from home, or with trainees at other military locations. This led us to consider virtual worlds as a development platform, to lessen the resource requirements of simulation development and benefit from pre-existing networking and multiuser features of these platforms. The rationale is that the virtual world provides a set of basic features such as content rendering, user login and interaction, user messaging, and object physics, among other aspects. This enabled development to focus on the behavioral elements of simulation development. In order to leverage this scenario for this purpose, we set to plan the prototype development by using OpenSimulator, due to its ability to enable communication with external systems without requiring changes to the code of the virtual world platform. It shares its client-server protocol with Second Life and therefore we could also benefit from the full set of coding resources and community that develop scripts and client applications for both platforms. OpenSimulator specifically is used by many different groups and for different purposes, and as with many virtual worlds it supports collaboration, including awareness of the presence of other users and communication, immersive interaction, and a credible 3D representation. The role of OpenSimulator in the simulation community has been defended for various scenarios, excluding pure science and target user groups of simulation experts [11]. We have also been exploiting the development of this scenario to create and further a simulation control architecture which attempts to have more The use of virtual world platforms as the environment for developing and deploying training simulations is frequent, in fields as diverse as emergency response [3], business management [4], medical and health scenarios [5], and security forces [6]. Military training scenarios have also employed virtual worlds as their development platform, such as the OLIVE platform [7], and this use follows in the stead of a long history of gaming and gaming technologies in military training [8]. Given that training is critical for the success of military operations, for technical operations such as aircraft maintenance as for tactical operations and combat. In this sense, virtual world

250

Andr Pinheiro et al. / Procedia Computer Science 15 (2012) 248 255

environments with multi-user abilities allow personnel to interact in a simulated face-to-face environment. There are several examples of serious games used by multiple users for training combat and tactical activities [9][10], but the field of multi-user technical training simulations is still in the beginning, but some examples exist, such as the work done at the University of Pennsylvania [13]. 3. Prototype aspects To develop the system prototype, we expedited the 3D environment modeling using the built-in tools of OpenSimulator/Second Life client viewers, and employed QAvimator to recreate in 3D the movements of the technicians. The goal in this prototype stage was not one of photorealistic visuals, but simply to be credible for testing and development. Following the overall goal of separation of concerns between simulation rendering and software control/decision-making, we implemented control and decision-making as an autonomous system, available to the OpenSimulator platform as a Web Service. This Web Service receives reports of all interactions taking place in the 3D environment, decides how the system must react and responds accordingly, with commands specifying the intended behavior of the simulation (see Fonseca et al. for more architecture details [2]). The decision-making was implemented in the form of an hierarchical state machine. In order to separate the control system from the specificities of OpenSimulator client-server communication, we developed simple text-based protocols for reporting of events by the 3D environment to the Web Service, and for the Web Service response to OpenSimulator with commands. Using these protocols, we managed to use a single script that is running concurrently in all virtual world objects. This script, developed in LSL (Linden Scripting Language, commonly used for scripting in Second Life and OpenSimulator worlds) acts as an interfacing module, to report events to the Web Service, and process the Web Service response. The commands in the response may be intended for the object originating the event, or for other objects, in which case the script broadcasts or relays the commands and parameters, depending on what the Web Service specified in its response. As a consequence, new objects can be deployed in the simulator without new OpenSimulator-specific code having to be developed: since the script is identical for all objects, only the decision-making component needs to be aware of the new objects. We emphasize that the 3D environment does not conduct any decision-making: it is only responsible for commands issued by the Web Service, in response to these reports. In this sense, it is an embryonic implementation of the Model-View-Controller architectural style [12] for virtual worlds. This holds the potential to render decisionmaking independent from the 3D environment: already in its current form we can replace the decision-making algorithm entirely, without having to change a single line of code in the 3D environment. In a parallel effort, we have used the same 3D models and scenario to implement the Partial-Order-Planning algorithm as a reasoning model to replace some of the human-controlled avatars by an intelligent software agent. The approach we used enables us to now combine these decision-making approaches without having to change the 3D scripts. Currently, the 3D script is the one aspect which still binds the system to a specific virtual world technology. It is reporting OpenSimulator/Second Life events and its copies are meant to be running independently. But the script is detached from the decision-making process. Thus, we have the ambition of making this approach evolve in order to render it viable for other virtual world platforms. Ideally, by developing a new event-reporting module and a new translation mechanism of decision-making commands into the specific requirements of the new technology one would like to be able to benefit from existing simulations with much lessened development efforts and resources. The current decision-making system, while not the focus of our development concerns, is presented herein for clarity. It works as follows: when it receives an event from OpenSimulator, it first queries a data store with all current data on simulation state. Then, based on this information and the date of the OpenSimulator-originating event, an hierarchical state machine algorithm determines the adequate response. For example, if a technician is in a sition is translated into virtual-world specific commands, which are provided to the originating script for execution. 4. Sample simulator tasks The installation process of a Pratt & Whitney F100 engine in an F-16 aircraft is quite extensive and complex,

Andr Pinheiro et al. / Procedia Computer Science 15 (2012) 248 255

251

requiring three technicians to do the various procedures. Plus, a specific role in the process is that of process checker, which may lead to a fourth person being involved, should none of the three required technicians have the credentials to perform this role. All necessary procedures are specified in a document known as the Job Guide, which refers to all the Technical Orders containing the necessary information. Installation of the engine inside the aircraft fuselage, the first procedure being implemented in the prototype, is a single process, but it is typically subdivided into four jobs, known as PT1, PT2, PT3, and PT4. Currently the prototype is implemented to support simulation throughout the PT1 job, which is a precondition for executing jobs PT2, PT3, and PT4. But more important, it enables testing of the cooperation situations of the process, because this PT1 job requires the involvement of all technicians, as it entails several tasks that cannot be performed by a single individual. One such task is the raising of the engine to align it with the empty hull of the aircraft fuselage, which requires all 3 technicians: two on the left side, another on the right side, as shown in Fig. 1. When the process checker issues the command (i.e., this may be the fourth element in the team), the three technicians will operate screw driving machines in concert to lift the motor simultaneously and coordinately, using speech as a means of synchronization, to avoid tilting the engine excessively while raising it.

Fig. 1 - Mechanics raising the engine at Air Base Nr. 5

The multi-user virtual world platform OpenSimulator enables three or four trainees to practice the synchronization in this task in a similar way, as long as voice chatting is available. In the current prototype, once all preconditions are adequate (such as having steadied the bearing cart and having adequate tools in hand and fitting for the tools in place), the engine lifting can take place. Once it is initiated, we elected to implement it in the following way: the simulator will take control of the arrow keys and change their function. During the lifting process, up and down arrow keys will respond as if operating the direction of screw driving machines or the turning of lifting wrenches (Fig. 2a).

Fig. 2

Simulator aspects: (a) Arrow in virtual space; (b) Mechanics lifting the engine in virtual space

The three technicians need to press their respective keys in the correct direction at roughly the same time (using voice chatting to synchronize their actions, as in the physical world) in order to lift or lower the engine. We defined

252

Andr Pinheiro et al. / Procedia Computer Science 15 (2012) 248 255

"roughly" as a 2-second interval. Currently, this is a precondition for lifting/lowering to occur, but in the future we intend to expand the simulator behavior to cover error conditions and accidents caused by wrong operations. Fig. 2b shows the avatars of trainees during the lifting of the engine in the virtual space. 5. User testing In order to evaluate the simulator prototype, to inform its subsequent development, a 2-day stay at Air Base Nr. 5 was planned and conducted, for conducting tests with prospective users. An early version of the prototype had been demonstrated earlier to the air base command, but no actual user tests. With this 2-day stay the team aimed to make the prototype available to technicians that are also trainers, in order to identify their satisfaction as users, and their expectations regarding subsequent development particularly in areas such as technical details and pedagogical affordances, user interface choices, and interaction methods. We also wanted to ascertain whether the technological context at the air base (computer hardware and network) supported the execution requirements of the prototype. Since the air base is located 260 km away from UTAD, the test preparation was done remotely, via e-mail exchanges. Two test sessions were planned, which included 3 trainers each, chosen on the basis of being potential future users of the simulator in the context of training sessions. These trainers were the people in charge of training new technicians as they are assigned to the air base, for operation on F-16 aircraft engines. I.e., they were the air base most qualified technicians regarding the mechanical maintenance procedures for F-16 engines. The 2 sessions were similar. Both took place in a side room of the main engine maintenance hangar (Fig. 3). This room had 2 group tables with chairs, a projector and a white board. 4 computers were used for tests, 3 for the users and 1 as an OpenSimulator server. The test preparation consisted in gathering and assembling all necessary supporting materials (computers, networking, cameras and microphones for test recording, software installation, and distribution of consent forms for data collection, questionnaires for characterization of user profiles, and paper guide).

Fig. 3 - Room arrangement during the test

The first day of our stay was employed in the room preparation, and installation of the software on the air base computers. The actual training sessions took place on day 2, each lasting about 1 hour 15 minutes. The sessions consisted of a brief introduction with a project presentation, an explanation of the motive for the team being present, and a presentation of the work plan for the session. All trainers read and signed consent forms regarding data collection for this test, and filled in the characterization questionnaire. During the simulation session proper, the whole dynamic was being projected onto a screen, from the perspective of the computer whose user (a development team member) played the role of supervisor for the engine installation simulation. The session began with an activity for acquainting users with the platform, aimed at the acquisition of basic skills for moving in the virtual world and interacting with objects. This acquaintance activity had 4 steps. The group of avatars started by meeting in a location away from the virtual hangar. At each step there was a virtual poster with a task for users to accomplish. The first was to enter avatar flight mode and move near the aircraft parked outside the hangar (Fig. 4a). Then they had to walk into the hangar, up to a table with objects on it (Fig. 4b). At this table they had to pick the objects, saving them to their o carry the objects to another table, near the first one (Fig. 5). Here they had to place the objects on this second table, i.e., taking them from the inventory. Finally, the acquaintance activity

Andr Pinheiro et al. / Procedia Computer Science 15 (2012) 248 255

253

was completed by asking them to return to the aircraft parking location, and from there to the engine, to initiate the simulation of engine installation. During the engine maintenance simulation, trainers were asked to employ the think aloud protocol throughout the process, communicating their thoughts and feelings. After completing the simulation process, a final group interview took place, recorded for later analysis. The interview attempted to measure users' satisfaction towards the system, and collect suggestions and recommendations for improvement.

Fig. 4 - Setting activity: (a) first station; (b) second station

Fig. 5 - Setting activity: third and fourth stations

6. Results

6.1. Trainers Profile By analysis of questionnaires, All trainers taking part in this test were male, with an average age of 42. They reported being regular computer users, using computers more than once per day. Their main uses of computing technology are searching for information (100%), work completion (83%), and socialization (83%). Regarding the use of virtual worlds, none of the trainers had used them before, nor were they aware of them. 6.2. Problems and limitations found

6.2.1. With the local hardware The air base assigned for this test: four laptops, a network switch, and network cables. We found out that this equipment was not entirely suitable: to run this prototype, we needed a network gateway, a service that was not provided by the provided network switch. So we proceeded with testing employing a small router we had planned as an alternative. The laptops had adequate memory and processing powers, but some limitations in terms of graphic cards, which were able to run the client software (Second Life viewer) but not in the best conditions. One was The computer being used as server also revealed itself with insufficient performance, and this exposed some deficiencies in data consistency between the Web Service and the 3D environment.

254

Andr Pinheiro et al. / Procedia Computer Science 15 (2012) 248 255

6.2.2. With the simulator prototype From analysis of interviews, respondents (trainers) mentioned some aspects for improvement and correction. For instance, avatar positioning was not always correct for all procedures. This is an important component of training, since inadequate positioning may expose a technician to unnecessary danger or inability to perform adequately. Regarding the usefulness of the simulator, respondents found it useful, an asset in support of practice. But they expressed the need to take into consideration not only the actual procedures, but also the prior safety inspection checklist. For ins

(Respondent 2, March 2nd, 2012). 6.2.3. Other aspects mentioned in the interviews From a user perspective, respondents of both groups classified the level of knowledge and preparation required to deploy the 3D virtual world as acceptable, although depending on actual computer and software capabilities. Regarding the acquaintance activity with the virtual world, respondents felt it facilitated the contact, with nd , 2012). They found the time for the acquaintance activity to Regarding the use of the actual simulator, the usefulness of having an introduction was reported, as was the importance of a checklist with common errors and required tasks to accomplish in the simulator. Trainers considered that support by training staff and colleagues will be important at an early phase, but will eventually no longer be As for the pedagogical content of the simulator, trainers considered that it did allow training of necessary skills for installation of F-16 engines, but noted that it is necessary to increase the level of detail. E.g., with statements n the ground, to check materials, you could watch the desired goals, since it allowed the practice of these aspects. Overall, they found that that the simulator could benefit the training of the installation of F-16 engines, reporting staff arrives, they learn, they do stuff, bu On the subject of virtual worlds, they considered them an alternative to the physical training of tasks, and an Regarding the graphical interface, trainers pointed out that some engine parts have to be more realistic, to the to identify the objects, Respondents 2, ibid.). They pointed out that interaction with the system is adequate and sufficient, yet that there is the need to get used to the conditions that the system offers. Finally, some suggestions were provided regarding technical training, including on the issue of safety awareness, and details such as ensuring the use of a toolbox in certain circumstances, or identification of occasional errors in the installation sequence of tasks. Interface cueing requests such as displaying an object description or name upon it on mouse over, were also recorded. 7. Conclusions and future work The test confirmed that the selected technology is feasible for use by the actual trainers involved in technician training at the air base. The nature of most omissions in terms of simulation aspects revealed that the procedure analysis needs to be fine-tuned, taking into account tactical know-how and systemic aspects (such as the security component of certain avatar positions or the importance of using specific accessories such as the toolbox for better team coordination). Expanding the simulator to support mixed teams of human-controlled avatars and artificial intelligence agents, to support training even if only some of the human trainees are available, is another promising line of work, which other teams have pursued [14], and which we have also started to explore [15].

Andr Pinheiro et al. / Procedia Computer Science 15 (2012) 248 255

255

The prototype is being further developed and the architectural goals of independence between virtual world platform and decision-making/control logic has shown its feasibility. We hope not only to complete the simulator, but to be able to pursue further the development of this software engineering approach, in view of faster, and less consuming simulation development. Acknowledgements This work is financed by the ERDF European Regional Development Fund through the COMPETE Programme (operational programme for competitiveness) and by National Funds through the FCT Fundao para a Cincia e a Tecnologia (Portuguese Foundation for Science and Technology) within project FCOMP - 01-0124-FEDER022701. This project has been partially funded with support from the European Commission. This article reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein. References
Lockheed-Martin-Corp., TO 1F-16AM-2-70JG-10-21 - Organizational Maintenance - Engine Removal and Installation Model F100-PW220/220E USAF/EPAF Series F-16/B Mid-Life Update aircraft, Technical Manual Job Guide. Bethesda, MD, USA: Lockheed Martin Corporation (2009). [2] B. Fonseca, H. Paredes, J. Rafael, L. Morgado, and P. Martins, Paulo, A Software Architecture for Collaborative Training in Virtual Worlds: F-16 airplane engine maintenance. In Adriana S. Vivacqua, Carl Gutwin and Marcos R.S. Borges (Eds.) "Collaboration and Technology: 17th International Conference, CRIWG 2011, Paraty, Brazil, October 2-7, 2011. Proceedings", 102-109. Berlin: Springer (2011). [3] Y.F.Chen, G. Rebolledo-Mendez, F. Liarokapis, S. de Freitas, E. Parker, The use of virtual world platforms for supporting an emergency response training exercise. In Procedings of the 13th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games , Wolverhampton, UK, 47-55 (2008). [4] C. Rodrigues, D. Coelho, L. Morgado, J. Varajo, A. Haidimoschi, G. Doppler, H. Koivusalo, P. Jokinen, G. Velegrakis, C. Sancin, and V. Carmenini, Virtual Learning for the management of successful SMEs in Europe. In Morgado, Leonel; Zagalo, Nelson; Boa-Ventura, Ana (Eds.) "Proceedings of the SLACTIONS 2009 International Conference - Life, imagination, and word using metaverse platforms", pp. 165170, ISBN 978-972-669-924-8. Vila Real, Portugal: Universidade de Trs-os-Montes e Alto Douro (2009). [5] M. Boulos, L. Hetherington, and S. Wheeler, Second Life: an overview of the potential of 3-D virtual worlds in medical and health education, Health Information & Libraries Journal, 24 (4), 233-245 (2007). [6] K. Hudson, K. deGast-Kennedy, Canadian Border Simulation at Loyalist College, Journal of Virtual Worlds Research, 2 (1), 3-11 (2009). [7] S. de Freitas, Serious virtual worlds. A scoping study. Bristol, UK: JISC (2008). [8] R. Smith, The long history of gaming in military training. Simulation & Gaming, 41 (1), 6-19 (2010). [9] K.A. Orvis, J.C. Moore, J. Belanich, J.S. Murphy, and D.B. Horn, Are soldiers gamers? videogame usage among soldiers and implications for the effective use of serious videogames for military training. Military Psychology 22 (2), 143-157 (2010). [10] T.M. Sotomayor, Teaching tactical combat casualty care using the TC3 sim gamebased simulation: a study to measure training effectiveness. Studies in health technology and informatics, 154, 176 179 (2010). [11] P.A. Fishwick, An introduction to OpenSimulator and virtual environment agent-based M&S applications. In Winter Simulation Conference (WSC '09), 177-183 (2009). [12] G. Krasner and S. Pope, A cookbook for using the model-view controller user interface paradigm in Smalltalk-80, Journal of ObjectOriented Programming, 1 (3), 26-49 (1988). [13] C. Stocker, B. Sunshine-Hill, J. Drake, I. Perera, J. Kider and N. Badler, CRAM it! A comparison of virtual, live-action and written training systems for preparing personnel to work in hazardous environments. In Hirose, Michitaka; Lok, Benjamin; Majumder, Aditi; and Schmalstieg, Dieter (Eds.), "IEEE Virtual Reality 2011 Singapore, March 19-23, 2011, Proceedings", pp. 95-102, ISBN 978-1-4577-0037-8. Danvers, MA: IEEE (2011). [14] N. Badler, C. Erignac, and Y. Liu, Virtual humans for validating maintenance procedures, Communications of the ACM, 45 (7), 56-63 (2002). [15] A. Vilela, A. Marques, H. Costa, J. Rafael, R. Prada, and L. Morgado, Aplicao de avatares autnomos para desempenhar o papel de de la 7 Conferencia Ibrica de Sistemas y Tecnologas de , Madrid, Spain (2012). [1]