Simulator validation – a new methodological approach applied to motorcycle riding simulators

Type of the Paper: Extended Abstract

Simulator validation – a new methodological approach applied to motorcycle riding simulators

Sebastian Will¹*, Thomas Hammer¹, Raphael Pleß¹, Nora Leona Merkel¹, and Alexandra Neukum¹

¹ Würzburger Institut für Verkehrswissenschaften (WIVW GmbH); will@wivw.de, https://orcid.org/0000-0003-0098-6212; hammer@wivw.de; pless@wivw.de; merkel@wivw.de, https://orcid.org/0000-0002-4865-368X; neukum@wivw.de

*corresponding author.

Name of Editor: Jason Moore

Submitted: 27/02/2023

Accepted: 12/04/2023

Published: 26/04/2023

Citation: Will, S., Hammer, T., Pleß, R., Merkel, N. & Neukum, A. (2023). Simulator validation – a new methodological approach applied to motorcycle riding simulators. The Evolving Scholar - BMD 2023, 5th Edition.

This work is licensed under a Creative Commons Attribution License (CC-BY).

Abstract:

Whenever driving simulators are used in research and development, to a certain extent the generalizability of the gained results is subject to discussion. Typically, a simulator gets validated in a rather effortful and complex process in order to prove the adequacy of the use of this specific simulator as research tool for a given research question. Since decades, there is plenty of research regarding the methods to validate simulators mainly from the automotive domain (e.g., Blaauw, 1982, Blana, 1996). Traditionally, there is a differentiation between a simulator’s physical validity and its behavioral validity. Whilst the first focusses on the simulator’s behavior and the presence of specific cues and operating elements, the latter focusses on the driver’s perception and consequently behavior. Furthermore, the degree of accordance between vehicle and simulator forms a category of validity, namely, absolute and relative validity. Whilst absolute validity describes an absolute numerical accordance of measurable dimensions between vehicle and simulator (e.g., certain forces, accelerations), relative validity describes a correlational accordance. Independent of the addressed dimension, simulator validation is a highly complex process, which is specific to the respective research question for which the simulator gets validated. Regarding single-track vehicle simulator concepts for which there is less experience from previous research (e.g., Cossalter, Lot, Massaro & Sartori, 2011), a rather broad validation procedure could be a useful tool in order to assess a simulator’s overall characteristics and therefore to assess its potential fields of application on a wider basis. This paper presents such a methodological validation approach applied to motorcycle riding simulators. The main assumption of the method is that complex riding tasks can be divided into smaller units that allow for discrimination of specific rider input characteristics, the so-called minimal-scenarios. These minimal-scenarios are riding tasks such as ‘starting from standstill’ or ‘initiating a curve at constant velocity’. Furthermore, it is assumed that minimal-scenarios can be reorganized to more complex riding tasks. This is intended to describe the variety of potential applications with a necessary minimum of elementary tasks in order to reduce the validation effort (Hammer, Pleß, Will, Neukum & Merkel, 2021).

Figure 1. Measurement motorcycle (left), dynamic motorcycle riding simulator DESMORI (center) and static motorcycle riding simulator (right) used for the validation study.

In order to investigate the applicability of the developed validation concept, a series of experiments has been conducted. The data shown here comes from a study with N = 15 motorcyclists (aged m = 37 years, SD = 14), which compared motorcycling with a real vehicle on a test track and the riders’ performance in a dynamic and a static motorcycle riding simulator (see Figure 1).

Figure 2. Roll angles and velocities for the avoidance maneuver in the environments test motorcycle (blue), dynamic motorcycle riding simulator (red) and static motorcycle riding simulator (yellow). The solid line indicates an avoidance maneuver to the left, the dotted line to the right and the dashed line shows the control maneuver going straight.

In all three test environments the motorcyclists were instructed to ride on an oval-shaped test course with a constant velocity of 35 km/h. A visual signal in the dashboard indicated at short notice whether to perform an avoidance maneuver to the left, to the right or whether to continue the oval-shaped course without avoidance maneuver. The different trajectories were marked using gates with traffic cones. Figure 2 shows exemplary vehicle dynamics data from this test sequence ‘avoidance maneuver’, which consisted of three previously defined minimal-scenarios ‘constant riding’, ‘entering a turn (v = const.)’ and ‘exiting a turn (v = const.)’. Across all test environments, people manage to comparably follow the target speed instruction. The roll angle over time shows a higher accordance between real riding and the dynamic motorcycle riding simulator, while the implemented vehicle dynamics model of the static simulator is not capable of replicating real effects. Based on the specific research question and the resulting relevance of physical or behavioral validity, the tested sequence of minimal-scenarios can support the assumed simulator’s validity for different research questions. For instance, the example given above aims at validating an avoidance maneuver existing of three different minimal-scenarios. If the results for the relevant concept of validity are positive, there is no need to conduct separate validation studies for different research questions involving the same relevant minimal-scenarios. In this case, an investigation of a warning assistance system, which aims at triggering an avoidance maneuver could be investigated likewise to a hazard perception training, which includes avoiding a suddenly appearing threat, on that same simulator.

In summary, the presented method does not try to substitute established methodologies in the field of driving simulator validation. The proposed approach shall provide a sound method for a justified global assessment of a simulator’s potential fields of application. This is done with a defined set of minimal-scenarios to which the established validation concepts shall be applied. This method was developed to be applied to single-track vehicle simulators as these simulators are in a rather early stage compared to well-established passenger car simulators and a wider overview about the simulator’s validity could be more helpful than a statement about the simulator’s validity for one specific research question. Yet, it is not limited to the field of single-track vehicles and may deliver interesting insights in potential down- and upsides of a simulator concept across all modes of transport.

Acknowledgements

This research was funded by the German Federal Highway Research Institute (Bundesanstalt für Straßenwesen, BASt) with the grant agreement number FE 82.0700/2017.

References

Blaauw, G. J. (1982). Driving experience and task demands in simulator and instrumented car: A validation study. Human Factors, 24 (4), 473-486, doi: https://doi.org/10.1177/001872088202400408.

Blana, E. (2023, February 17). Driving Simulator Validation Studies: A Literature Review. Institute of Transport Studies, University of Leeds, Working Paper 480. http://eprints.whiterose.ac.uk/2111

Cossalter, V., Lot, R., Massaro, M., & Sartori, R. (2011). Development and validation of an advanced motorcycle riding simulator. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 225, 705-720. doi:10.1177/0954407010396006

Hammer, T., Pleß, R., Will, S., Neukum, A., & Merkel, N. L. (2021). Anwendungsmöglichkeiten von Motorradsimulatoren (Bundesanstalt für Straßenwesen Ed. Vol. M323). Carl Schünemann Verlag, Bremen.