Simulator validation – a new methodological approach applied to motorcycle riding simulators

Type of the Paper: Conference Paper

Simulator validation – a new methodological approach applied to motorcycle riding simulators

Sebastian Will¹*, Thomas Hammer¹, Raphael Pleß¹, Nora Leona Merkel¹, and Alexandra Neukum¹

¹ Würzburger Institut für Verkehrswissenschaften (WIVW GmbH); will@wivw.de, https://orcid.org/0000-0003-0098-6212; hammer@wivw.de; pless@wivw.de, https://orcid.org/0009-0006-5984-6723; merkel@wivw.de, https://orcid.org/0000-0002-4865-368X; neukum@wivw.de

*corresponding author

Name of Editor: Jason Moore

Submitted: 15/09/2023

Accepted: 21/09/2023

Published: 26/09/2023

Citation: Will, S., Hammer, T., Pleß, R., Merkel, N. & Neukum, A. (2023). Simulator validation – a new methodological approach applied to motorcycle riding simulators. The Evolving Scholar - BMD 2023, 5th Edition.

This work is licensed under a Creative Commons Attribution License (CC-BY).

Abstract:

Whenever driving simulators are used in research and development, to a certain extent the generalizability of the gained results is subject to discussion. Typically, a simulator gets validated in a rather effortful and complex process in order to prove the adequacy of the use of this specific simulator as research tool for a given research question. Since decades, there is plenty of research regarding the methods to validate simulators mainly from the automotive domain (e.g., Blaauw, 1982; Blana, 1996). Traditionally, there is a differentiation between a simulator’s physical validity and its behavioral validity. Whilst the first focusses on the simulator’s behavior and the presence of specific cues and operating elements, the latter focusses on the driver’s perception and consequently behavior. Furthermore, the degree of accordance between vehicle and simulator forms a category of validity, namely, absolute, and relative validity. Whilst absolute validity describes an absolute numerical accordance of measurable dimensions between vehicle and simulator (e.g., certain forces, accelerations), relative validity describes a correlational accordance. Independent of the addressed dimension, simulator validation is a highly complex process, which is specific to the respective research question for which the simulator gets validated (e.g., training race riders vs. assessing distraction caused by human-machine interfaces, HMI). Regarding single-track vehicle simulator concepts for which there is less experience from previous research (e.g., Cossalter, Lot, Massaro, & Sartori, 2011; Grottoli, Mulder, & Happee, 2022), a rather broad validation procedure could be a useful tool in order to assess a simulator’s overall characteristics and therefore to assess its potential fields of application on a wider basis. This paper presents such a methodological validation approach applied to motorcycle riding simulators. The main assumption of the method is that complex riding tasks can be divided into smaller units that allow for discrimination of specific rider input characteristics, the so-called minimal scenarios. These minimal scenarios are riding tasks such as ‘starting from standstill’ or ‘initiating a curve at constant velocity’. Furthermore, it is assumed that minimal scenarios can be reorganized to more complex riding tasks. This is intended to describe the variety of potential applications with a necessary minimum of elementary tasks in order to reduce the validation effort for a global assessment of the simulator’s capabilities (Hammer, Pleß, Will, Neukum, & Merkel, 2021). This more generic result can also be regarded as a limitation. The proposed empirical evidence from participant studies on a static, a dynamic motorcycle riding simulator as well as a reference ride on a real motorcycle suggests that the validation approach can be beneficial.

Keywords: Motorcycle, Powered Two-wheeler, Simulator, Methods, Validation

Introduction

Driving simulators play an essential role for the investigation of safety relevant research topics in the passenger car sector. Ranging from usability and user experience research to the investigation of highly automated driving, simulators support the fast development in the passenger car domain. In recent years, the motorcycle sector has undergone a rapid change of technical developments as well. Yet, motorcycle simulators are in a rather early stage in this regard which poses the challenge to gain knowledge concerning their validity for specific use cases and therefore their applicability for different research questions at hand. Motorcycle simulators, especially those with a complex, high-fidelity setup are less common than passenger car simulators. Therefore, the overall experience regarding their application is at a rather early stage.

Especially the inherent instability of single-track vehicles needs to be considered. This major difference makes the simple transfer of knowledge of applications from the passenger car sector to the motorcycle sector almost impossible. In general, motorcycle simulators seem appropriate as a tool for research and development (R&D) as well as training purposes. Same as in the passenger car sector. Yet, this assumption needs to be proven before results gained on a simulator may be generalized to real riding. This process of simulator validation is typically limited to the specific field of application and therefore a complex and expensive process which implies the need for a validation study for every single use case. For instance, a simulator without motion base might be perfectly applicable to assess the effects caused by the interaction with a human-machine interface (HMI) while driving, if the workload caused by the driving task is comparable between simulator and reality. Yet, this specific static simulator might not be appropriate to investigate the effects caused by an autonomous emergency braking as the participant does e.g., not feel the resulting vestibular cues of braking.

Another important aspect lies within the selection of the investigated parameters for validity assessment. A validation study aiming at distraction assessment will observe other parameters than a validation study aiming at the application of a suspension setup. Since the 1970s validity concepts that are specific for the field of driving simulation have been developed. Traditionally, there is a differentiation between a simulator’s physical validity and a behavioral validity. Whilst the first focusses on the simulator’s behavior and the presence of specific cues and operating elements, the latter focusses on the driver’s behavior. Yet, the idea of applying a more holistic and multidimensional validation approach is almost as old as the field of research itself. For instance, Blaauw (1982) proposed that simulator validation should involve objective riding parameters, subjective performance indicators, and rider workload.

Furthermore, the degree of accordance between vehicle and simulator forms a category of validity, namely, absolute, and relative validity. Whilst absolute validity describes a numerical accordance between vehicle and simulator, relative validity describes a correlational accordance (Allen & O'Hanlon, 1979; Blaauw, 1982; Blana, 1996). As absolute validity for all components of a simulator and all fields of application will probably remain unreachable, a focus on relative validity is recommended by different researchers (Blaauw, 1982; Caird & Horrey, 2011; Godley, Triggs, & Fildes, 2002). Depending on the intended purpose of simulator use, the focus on physical validity or behavioral validity might change. To take up the above-mentioned example again, a simulator that is used for suspension assessment is going to focus more on physical validity. Yet, it must be noted that even simulators with a complex motion system are always limited and cannot replicate all potentially relevant and highly dynamic scenarios. The simulator in use for distraction assessment will probably be validated regarding behavioral validity. For the latter, the driver or rider workload is also an important aspect to be considered (Espié, Gauriat, & Duraz, 2005). For human factors research questions, the workload caused by the primary riding task is of highest importance. This workload should be as close to normal driving on public roads as possible, if e.g., the effects of a warning shall be measured while driving. Any overstraining simulator control task may bias the observed reaction times as response to the warning.

The technical term of simulator validation contains different facets which will be described in the following sections. Firstly, a simulator’s validity cannot be attributed to its components in an additive way. The question that must rather be posed is: which components are necessary to answer the respective research question (Caird & Horrey, 2011). Thus, providing specific sensory cues, such as certain vestibular feedback, can be seen as a necessary but not sufficient condition for a simulator’s validity for a specific use case. An available but distorted presentation of a stimulus might even cause more unnatural behavior which in turn decreases validity (e.g., if a poorly implemented motion cueing feels unnatural while braking, participants may – voluntarily or not – avoid (stronger) braking). Furthermore, a change in a simulator’s setup might change rider’s perception regarding the simulator behavior fundamentally. As an example, a simulator’s visualization system could be replaced from a front projection to a head-mounted display. By changing the presentation of the visualization also the rider’s reference to his spatial orientation in the room changes (as nothing is visible except the presentation in the head-mounted display). Additionally, the perception of simulator motions (if available) might change by the alteration of visual presentation, as participants’ expectations towards motion might depend on visualization and vice versa. Therefore, every significant change in a simulator’s setup would need a repetition of the previously conducted validation studies.

Thus, established validation approaches do hardly seem feasible for simulator setups that are still in development and therefore underlie regular changes. In order to cope with this challenge, a new validation method was developed that shifts the focus from detailed validation studies to a broader assessment of a simulator for a predefined set of potential applications.

Summarized, simulator validation is a highly effortful and complex process, which is specific to the respective use case that is validated. Especially regarding innovative simulator concepts for which there is less experience from previous research, a rather broad validation procedure could be a useful tool in order to assess a simulator’s overall characteristics and therefore to assess its potential fields of application on a broad basis. The following paper presents a method that tries to assess simulator validity on a holistic and less detailed level than conventional validation methods do.

Methods

General methodology

The main assumption of the method is that complex riding tasks can be divided into smaller units that allow for discrimination of specific rider input characteristics, the so-called minimal scenarios. These minimal scenarios are riding tasks that are conducted only serially such as ‘starting from standstill’ or ‘initiating a curve at constant velocity’. In order to identify relevant minimal scenarios for the respective simulator that should be investigated, the potential fields of application, for which a simulator is intended to be used, must be defined. Usually, a simulator is not suitable for all fields of application in the same way (e.g., a simple, static simulator which was built for training hazard perception can probably not be used to assess different suspension setups). For the general verification of this approach, the German system for accident classification was analyzed to retrieve a list of practically important rider behaviors, which should help avoiding the different accident types. This list was transferred into a list of minimal scenarios that fulfill the assumption of seriality and that reflect the most typical riding scenarios.

With that list at hand, a verification study with N = 6 experts (professional trainers, motorcycle researchers etc.) that understand the underlying vehicle dynamics principles was conducted. In this study, the different minimal scenarios were tested as isolated as possible (Figure 1) with varying levels of dynamics (i.e., different predefined speeds, different distances to achieve a certain speed etc.) and in all three test environments (static simulator, dynamic simulator, real motorcycle). The outcome was a matrix of objective and subjective parameters characterizing the different test environments.

Figure 1. List of identified minimal scenarios with the selection made to be tested in the expert study.

In a next step, a participant study with N = 15 non-professional riders was conducted. The aim was to understand whether more natural and complex combinations of the different minimal scenarios still reflect the previously identified characteristics of the test environment. Further, this is intended to describe the variety of potential applications with a necessary minimum of elementary tasks in order to reduce the validation effort for a global assessment of the simulator’s capabilities (Hammer et al., 2021).

Test environments

In order to investigate the applicability of the developed validation concept, a series of experiments has been conducted involving a measurement motorcycle to deliver some kind of ground truth and two motorcycle riding simulators.

Measurement motorcycle

A KTM 790 Duke was used as measurement motorcycle within the participant study (see Figure 2 left). The vehicle has a 799 cm³ two-cylinder in-line engine with an engine power of 77 kW (103 HP). The series motorcycle is already equipped with state-of-the-art sensors and assistance systems, as for example an IMU (inertial measurement unit) and motorcycle stability control. Due to the rather low sitting height, the upright seated position, and the low weight of 187 kg, the vehicle is suitable for a wide range of participants. Additionally, the implemented measurement technology allows direct access to the onboard sensors which allows the recording of relevant riding parameters such as roll angle, brake pressure and velocity. The entire measurement technology was placed within a custom aluminum side case which is mounted at the left-hand side of the motorcycle. Most relevant parameters are supplied via CAN-Bus and recorded with an Intel NUC^© and SILAB^® as data logging software. The recording rate is at 60 Hz. Additionally, position data is recorded with a Navilock USB receiver with a sampling rate of 5 Hz. For acquisition and output of analog signals an Arduino^© was installed. The power supply of the measuring technology is provided by a second battery with charging unit.

Motion-based dynamic motorcycle riding simulator DESMORI

The DESMORI dynamic motorcycle riding simulator at WIVW (see Figure 2 center) with the driving simulation software SILAB was used as high-fidelity simulator (2018 setup). For the studies a vi-grade BikeRealTime multi body simulation model was used. A BMW F 800 is used as mockup, which is mounted on a 6-dof motion platform. All rider controls such as throttle twist grip, front and rear brakes, 6-speed manual gear box including clutch operation are available. A curved projection screen with 4.5 m diameter and 2.9 m height enables 220° horizontal field-of-view. The two rear-mirrors are realized by 7-inch TFT-displays while the dashboard is displayed on a 10-inch TFT-touchscreen. The sound is provided via in-helmet speakers. A so-called g-vest provides forces to the rider torso simulation aerodynamical drag and acceleration. Steering torque is provided by an electric motor providing up to 80 Nm maximum torque.

Static motorcycle riding simulator

Opposed to the dynamic motorcycle riding simulator, the static motorcycle riding simulator of WIVW was equipped with a simplified single-track vehicle model with positive steering (Figure 2 right). Visual cues are provided by three 55” LCD screens offering 180° horizontal field of view. The instrument cluster is displayed on a 10” LCD screen and the two rear-mirrors via 7-inch TFT-screens. A KTM 1290 Super Adventure R was installed as mockup. The motorcycle simulator uses an automatic gear box. All relevant controls such as throttle twist grip, front and rear brake levers are implemented. As within the dynamic motorcycle riding simulator the simulation was provided by SILAB simulation software. Data including all rider inputs was recorded at a frequency of 100 Hz. Acoustic cues come from in-helmet speakers. Haptic feedback on steering torque is provided by an electric motor that delivers a maximum torque of 50 Nm.

Figure 2. Measurement motorcycle in the participant study (left), dynamic motorcycle riding simulator DESMORI (center) and static motorcycle riding simulator (right) used for the validation study.

Procedure

The study was conducted using a within-subjects design. This means that all participants completed all tasks in all three test environments. The real riding investigation exemplarily described in this paper was performed on a closed test-track. In the beginning, all riders had time to familiarize with the measurement motorcycle as to their own needs. Same on both simulators, where all participants went through a training session in order to familiarize with the simulated riding. Every appointment started with a completion of an informed consent document and explanation of the study purpose. The participants completed different test rides as described more detailed in the report from Hammer et al. (2021). Questionnaires were filled in after each ride and a longer inquiry took place at the end of each appointment. For the test sequence described in this paper, the motorcyclists were instructed to ride on an oval-shaped test course with a constant velocity of 35 km/h. A visual signal in the dashboard indicated at short notice whether to perform an avoidance maneuver to the left, to the right (3 m lateral displacement within 15 m) or whether to continue the oval-shaped course without avoidance maneuver. The different trajectories were marked using colored gates with traffic cones. All tests were done clockwise and counter-clockwise.

Participant panels

For the expert study, N = 6 motorcyclists (aged m = 26.2 years, SD = 1.95; all male) with professional knowledge in the field of motorcycle dynamics participated. The average mileage ridden in the last 12 months was M = 7 667 km (min = 2 000 km; max = 20 000; SD = 6 574 km). In the participant study, N = 15 motorcyclists (aged m = 37 years, SD = 14; n_female = 1) with a valid license class A (motorcycle unlimited) were observed. The average mileage ridden in the last 12 months was M = 7 300 km (min = 1 000 km; max = 25 000; SD = 7 496 km).

Results

Qualitative rider feedback

The riders rated the dynamic simulator as more realistic than the static motorcycle simulator in terms of steering and overall impression. The interaction with the dynamic simulator was also rated as being more natural. The stabilization of the dynamic simulator approaches the stabilization of the real motorcycle with increasing speed. Hence, the cornering stability is evaluated as too low for the dynamic motorcycle simulator. In the static simulation the stabilization is evaluated as too high. Visual and haptic/proprioceptive feedback of the vehicle dynamics is evaluated as too weak in both simulation environments. More of these subjective ratings were captured with questionnaires, e.g., on presence (Will, 2017), which provide important insights on the simulator acceptance, for instance.

Subjective questionnaire data

As an example, on how to deal with subjective measures, Figure 3 shows the subjectively experienced levels of workload while performing the different minimal scenarios in the expert study as a function of the test environment. The general pattern comparing the minimal scenarios seems stable across test environments. The demand posed by the minimal scenarios is on average allocated in the lower half of the workload scale. An offset towards higher perceived workload can be seen in the dynamic motorcycle riding simulator. A certain variation regarding the ratings between riders can be observed across all minimal scenarios and test environments.

Figure 3. Subjective workload ratings for the different investigated minimal scenarios (MSC) as a function of test environment. Average mean values and standard deviations are displayed.

Objective vehicle dynamics data

An example dealing with objective metrics is given in Figure 4. This graph shows exemplary vehicle dynamics data from the test sequence ‘avoidance maneuver’, which consisted of three previously defined minimal scenarios ‘constant riding’, ‘entering a turn (v = const.)’ and ‘exiting a turn (v = const.)’. Across all test environments, people manage to comparably follow the target speed instruction, which can be seen in the lower half of the figure. The roll angle over time shows a higher accordance between real riding and the dynamic motorcycle riding simulator, while the implemented vehicle dynamics model of the static simulator is not capable of replicating these effects. On a qualitative basis, it is obvious that the completion of the given minimal scenarios combined to an avoidance maneuver requires a roll angle sequence to the left and right – or vice versa – on the real motorcycle as well as the dynamic simulator. So, for the solid line in Figure 4, which shows an avoidance maneuver to the left, a roll angle to the left can be observed before the vehicle leans to the right to pass through the laterally offset gates (Gate₂ and Gate₃). In absolute numbers the median roll angle reaches between 7° and 10° maximum roll angle between the gates. In both environments, a certain variation can be seen between different runs and riders (shaded area). The dynamics data of the static motorcycle simulator differs from that pattern. Further the absolute roll angle medians reach 1° to 4° maximum and there is almost no variance in between runs and riders.

Figure 4. Roll angles and velocities for the avoidance maneuver in the environments test motorcycle (blue), dynamic motorcycle riding simulator (red) and static motorcycle riding simulator (yellow). The solid line indicates an avoidance maneuver to the left, the dotted line to the right and the dashed line shows the control maneuver going straight. Negative roll angles imply a lean angle to the right.

Conclusion

The qualitative feedback provides important information on which holistic impression the simulator creates. It can guide the direction of simulator optimization and provide first insights in potential fields of application for the given simulator. More directly related to the validation process itself is the quantitative subjective feedback. For instance, the different minimal scenarios vary as to their demand towards the rider. As an example, riding constantly requires less input and control than accelerating out of a turn. Therefore, it was expected to find this differentiation between the minimal scenarios in the subjectively experienced workload. This was true and the pattern between minimal scenarios was stable across all test environments, which is a good indicator for relative validity in this domain. Yet, the absolute level of perceived workload was constantly higher in the dynamic motorcycle riding simulator as compared to a good fit between static simulator and real motorcycle. This implies that riding the given dynamic simulator setup requires more resources than riding the static simulator or the real motorcycle. For research questions with a focus on rider workload (e.g., dual-task paradigm to investigate the distraction caused by the interaction with an HMI while riding), the static simulator might be a better fit as the amount of resources to deal with the secondary task is comparable between static simulator and real riding across all minimal scenarios. This would follow the arguments of Espié et al. (2005).

Opposed to the workload results the objective data has shown a more adequate representation of motorcycle dynamics for the dynamic motorcycle simulator compared to the static motorcycle simulator across the three minimal scenarios. This was shown both for steering characteristics as well as roll angle representation. Both, the measurement motorcycle, and the dynamic motorcycle simulator have shown variance in riders’ behavior while the results for the static motorcycle simulator are rather homogeneous. Yet, the simulator properties might emphasize different strategies. For instance, the MSC entering a turn should start when passing gate₁. While the main roll angle increase is observed after passing the gate with the real motorcycle, participants on the dynamic motorcycle riding simulator seem to initiate entering the turn earlier and pass gate₁ already at a certain lean angle. This might, e.g., be because riders try to avoid higher roll rates in the dynamic simulator. The initiation of MSC exiting a turn (changing lean angle from one side to the other) seems to take place similarly and with a comparable roll angle progression on the real motorcycle and the dynamic simulator. Thus, besides the higher workload levels the dynamic motorcycle simulator appears to be more adequate to investigate research questions which involve a detailed representation of vehicle dynamics and the rider input provoking these dynamics as compared to the static simulator.

Based on the specific research question and the resulting relevance of physical or behavioral validity, the tested sequence of minimal scenarios can support the assumed simulator’s validity for different research questions. For instance, the example given above aims at validating an avoidance maneuver existing of three different minimal scenarios. If the results for the relevant concept of validity are positive, there is no need to conduct separate validation studies for different research questions involving the same relevant minimal scenarios. In this case, an investigation of a warning assistance system, which aims at triggering an avoidance maneuver could be investigated likewise to a hazard perception training, which includes avoiding a suddenly appearing threat, on that same simulator.

Still, what remains an issue of discussion is the definition of thresholds. How precisely must the observed parameters in the simulator and a real vehicle resemble each other to create a meaningful fit? Inferential test statistics are obviously a way to define this. Hence, depending on the data set this may not necessarily mark a meaningful correspondence. Further, the decision to start the validation process coming from accident scenarios resulted from the fact that the specific simulators under investigation were mainly designed to deal with research questions related to motorcycle safety. Also, the definition of the relevant range of vehicle dynamics resulted from that origin. Different minimal scenarios or at least vehicle dynamics ranges may result, if a simulator is meant to be used for other purposes, such as race rider training. Even if there remain limitations, certain degrees of freedom for the design of simulator validation studies and their interpretation, the benefits of using a validated simulator for a given research question may outweigh the disadvantages and challenges. Figure 5 proposes a decision tree that could guide towards one or the other test environment based on the specific research question, which follows Caird’s (2011) idea of individual simulator setups based on the research question at hand.

In summary, the presented method does not try to substitute established methodologies in the field of driving simulator validation. Further, the proposed concept of validating minimal scenarios that can be combined to different meaningful and more complex maneuvers needs more investigation. The proposed approach shall provide a method for a justified global assessment of a simulator’s potential fields of application including objective dynamics data as well as subjective assessments. Additionally, the method shall help identifying potentials for optimizing a certain simulator setup. This is done with a defined set of minimal scenarios to which the established validation concepts shall be applied.

Figure 5. Decision tree for the selection of a research environment based on boundary conditions (Hammer et al., 2021, p.10).

This method was developed to be applied to single-track vehicle simulators as these simulators are in a rather early stage compared to well-established passenger car simulators and a wider overview about the simulator’s validity could be more helpful in the beginning than a statement about the simulator’s validity for one specific research question. Yet, the approach is not limited to the field of single-track vehicles and may deliver interesting insights in potential down- and upsides of a simulator concept across all modes of transport.

Acknowledgements

This research was funded by the German Federal Highway Research Institute (Bundesanstalt für Straßenwesen, BASt) with the grant agreement number FE 82.0700/2017.

References

Allen, R., & O'Hanlon, J. (1979). Effects of roadway delineation and visibility conditions on driver steering performance. Transportation research record, 739, 5-8.

Blaauw, G. J. (1982). Driving experience and task demands in simulator and instrumented car: a validation study. Human FActors, 24(4), 473-486.

Blana, E. (1996). Driving Simulator Validation Studies: A Literature Review.

Caird, J. K., & Horrey, W. J. (2011). Twelve practical and useful questions about driving simulation. Handbook of driving simulation for engineering, medicine, and psychology, 5.1-5.16.

Cossalter, V., Lot, R., Massaro, M., & Sartori, R. (2011). Development and validation of an advanced motorcycle riding simulator. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 225, 705-720. doi:10.1177/0954407010396006

Espié, S., Gauriat, P., & Duraz, M. (2005). Driving simulators validation: The issue of transferability of results acquired on simulator. Paper presented at the Driving Simulation Conference North-America (DSC-NA 2005), Orlondo, FL.

Godley, S. T., Triggs, T. J., & Fildes, B. N. (2002). Driving simulator validation for speed research. Accident Analysis & Prevention, 34(5), 589-600.

Grottoli, M., Mulder, M., & Happee, R. (2022). Motorcycle simulator subjective and objective validation for low speed maneuvering. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 1-16. doi:10.1177/09544070221110

Hammer, T., Pleß, R., Will, S., Neukum, A., & Merkel, N. L. (2021). Anwendungsmöglichkeiten von Motorradsimulatoren (Bundesanstalt für Straßenwesen Ed. Vol. M323). Bremen: Carl Schünemann Verlag,.

Will, S. (2017). Development of a presence model for driving simulators based on speed perception in a motorcycle riding simulator. (PhD thesis), University of Wuerzburg, Wuerzburg.