- Original Paper
- Open Access
Objective and subjective evaluation of an advanced motorcycle riding simulator
European Transport Research Review volume 2, pages223–233 (2010)
This paper outlines the characteristics of a top-of-the range motorcycle simulator designed and built at the University of Padua over a period of several years last years; it consists of a motorcycle mock-up with functional throttle, brakes, clutch and gearlever mounted on a five ‘degrees of freedom’ platform, a real-time multibody model of the motorcycle and an audio and visual systems. The applications of the simulator are to test devices such as ABS, traction control and other ARAS in a controlled, safe environment, to study riders’ behaviour and to train them. The aim is to find a procedure to validate the behaviour of a Motorcycle Riding Simulator with a real PTWs.
An innovative procedure for the objective and subjective validation of motorcycle simulators has been developed and implemented, in order to be able to apply the results obtained on the simulator to the real world.
The evaluation of objective and subjective data collected shows that the proposed simulator is adequate for handling tests. The proposed method is suitable to be extended to vehicle simulator in general.
The development work done by University of Padova provides an innovative and reliable tool for the validation of a motorcycle riding simulator.
Nowadays, powered two-wheeler vehicles (PTW) are widely used not only for pleasure, but also for increasing mobility in the crowded urban and sub-urban roads of many European towns. For several reasons, PTW dynamics and safety have not been investigated as much as with four-wheeled vehicles, despite the fact that riders are among the most vulnerable road users. Therefore the development of devices aimed at improving the comfort and safety of PTWs, as well as investigating the behavioural factors that contribute to crashes, are important areas for research. Moreover, the roll degree of freedom which makes PTWs quick and prompt on urban roads and diverting on the rural track has some safety implications which require new riders to receive proper training. Unfortunately it is not easy to train new riders in dangerous situations, such as riding on a slippery road or emergency braking. From this point of view riding simulators may help both as a training tool and in the development of innovative devices aimed at improving rider safety.
It is worth noting that motorcycle riding simulators are not as widespread as aircraft and car driving simulators, and therefore the current selection is not very rich. Honda started to develop a series of motorcycle simulators in 1988; its first prototype consisted of a 5 DOF mock-up (lateral, yaw, roll, pitch and steer motions on a swinging system for the longitudinal acceleration restitution) and was based on a linear 4 DOF motorcycle dynamics model. In 1996, as a consequence of the change of the Japanese Road Traffic Act which required the use of simulators in riding schools lessons, Honda put a mass-produced model on the market. This second prototype had a simplified 3 DOF mock-up (roll, pitch and steer motions) and it was based on a properly tuned empirical motorcycle model. In 2002, Honda developed a third prototype which consisted of a 6 DOF plan manipulator for the mock-up motion, a head mounted display for visual projection, a 4 DOF model for the lateral motorcycle dynamics and a 1 DOF model for the longitudinal dynamics [1, 2]. The Department of Innovation in Mechanics and Management (DIMEG) of Padua University began the development of a riding simulator in 2000 and presented the first prototype in 2003 . In 2003, PERCRO laboratory also presented its riding simulator with a real scooter mock-up mounted on a steward platform , and in 2007 INRETS presented a riding simulator based on a 5 DOF platform and a linear 5 DOF motorcycle mathematical model .
The DIMEG motorcycle simulator has been developed to test devices such as ABS, traction control and other ARAS in a controlled, safe environment, to study riders’ behaviour and to train them. It is possible to reproduce and consequently analyse the most critical and risky situations that a normal rider will find every day on all roads. However, in order to apply the results obtained by the studies on the riding simulator to a real motorcycle it is necessary that the behaviour of both simulator and motorcycle are the same. Since there are very few studies focussing on motorcycle simulators and in particular on their validation, this paper proposes an innovative procedure for both objective and subjective validation and reports the results of its application to the DIMEG simulator. Fine tuning and validation activities were performed inside the 2BeSafe project in the Seventh European Framework Programme (theme 7—sustainable surface transport), and commenced in January 2009. 2BeSafe is a collaborative research project and its objective is to conduct behavioural and ergonomic research in order to develop countermeasures for enhancing powered two-wheeler (PTW) riders’ safety, including research into crash causes and human errors and the world’s first naturalistic riding study involving instrumented PTWs.
This paper first describes the DIMEG simulator, then explains the proposed validation methodology and illustrates the data collected during objective and subjective evaluation.
Description of the riding simulator
A simulator is a complex system that aims to reproduce a real environment in a restricted and controlled area where it is possible to simulate any actions under totally safe conditions. The motorcycle riding simulator shown in Fig. 1 is a top of the range one and has been designed and developed in its entirety at DIMEG. On the simulator, the rider sits on a motorcycle mock-up and operates the throttle position, brakes, clutch and gearshift lever like on a real bike. Moreover, the handlebar and footpads are sensorized.
The rider’s control actions are transferred to the real-time multibody model of the motorcycle which has a 14 ‘degrees of freedom’ model, includes a realistic model of the suspension, clutch, engine, tires and a 3-D road, and has been optimised for real-time performance. The simulated dynamics are then filtered by the washout and converted into references for the motion and visual cues. Motion cues are generated by the servomotors that drive 5 axes of the mock motorcycle; the roll, pitch, yaw and steer rotations plus the lateral displacement. The different subsystems are described in detail below.
The rider rides a motorcycle mock-up equipped with all of the commands available on a real bike. In particular, the rider’s actions are monitored by measuring the steering torque, leaning of the body, throttle position, front brake lever and rear brake pedal pressures, clutch position and gearshift lever position.
Figure 2 shows the motion cues of a motorcycle mock-up whose serial kinematic chain is composed of 4 mobile frames plus a fixed one to reproduce the motion of the vehicle in terms of lateral displacement, yaw, roll, pitch and steer rotations. The first mobile member has the yaw and lateral displacement degrees of freedom, which are actuated by two servomotors equipped with ball screws; in sequence there are the roll, the pitch and steer degrees of freedom, as summarised in Figs. 6, 7 and 8.
The simulator includes an audio-visual system; in particular, the scenario is projected onto three widescreens measuring 1.5 × 2 m2 placed in front of the rider. The 5.1 surround sound system reproduces engine sound previously recorded on a real motorcycle for a range of engine rpm.
Real-time multibody model
In order to achieve a good, realistic correspondence between a real and the simulated motorcycle a detailed multibody model was developed. The motorcycle model is composed of a whole motorcycle; its front frame, wheels and the rider’s upper body (see Fig. 3).
The mathematical model is non linear and has 14 DOF (see Fig. 3), corresponding to the position and orientation of the chassis, the steering angle, the front and rear suspension travels, the front and rear wheel rotations, the engine spin rate, the front fork bending deflection and the sprocket absorber deflection. There are 7 motorcycle inputs: steering torque, throttle position, rear and front brake torques, gears selected, clutch position and the foot pegs effort (as an indirect measure of the rider’s torso motion). Suspensions and tires are modelled in detail, as well as the clutch, the gearbox and the engine. More details are given in references [9, 10].
Since it is physically impossible to reproduce accelerations as they are in real life using the simulator, a washout filter is used, which aims to recreate riding accelerations and angular velocities by using the acceleration and the angular velocity of the simulator (inertial effect) and the acceleration of gravity (gravitational effect). Simulated accelerations are first separated by filters into their spectral components. The components at low frequencies are generated using the gravitational effect, slowly tilting the simulator, while the components at high frequencies are reproduced by moving the simulator faster (with the electrical engine) and exploiting the effects generated by the inertial motion. As shown in Fig. 4, this implemented washout is made up of two parts: the first filter (low pass filter), after an initial gauge, which removes high frequency components of input variables and includes a matrix that combines the various inputs in a linear combination, and, after that, a second filter which provides the output for the platform.
It has been found that moving the simulator like a real motorcycle to the greatest possible extent does not give the best riding feeling, so gains and other adjustable parameters of the washout filter have been tuned using a trial-and-error procedure based on the subjective evaluation of feelings. Appropriate tuning leaded to a different washout for the visual and motion screens; as an example, while cornering, the roll angle is divided into two parts: the biggest one is used to tilt the virtual horizon on the screen, while a smaller part is used to give a motion cue by rolling the mock-up motorcycle. This solution is particularly useful while using the new visual system composed of three widescreens and a large field of view (FOV).
Motorcycle riding simulators are more recent than car and truck simulators, so they still need to be tuned to make them suitable for use in studies into rider behaviour. The challenge is to find an optimal compromise in the rendering of the simulator which allows the riders to feel as if they are riding an actual PTW and at the same time allows them to succeed in mastering the PTW as easily as they can master an actual one (at least for normal riding situations). To attain this goal, the following procedure has been iteratively applied: the first step is the fine-tuning of the motion, sound and visual rendering devices, which has been done experimentally using a small group of highly skilled riders; the second step is the simulator validation, conducted by comparing the behaviours, performances and self-reported impressions of a wider group of riders of different ages, experience and skill. The validation, which of course is the most important aspect, is based on two complementary concepts: the objective validation, where some objective, carefully selected parameters are compared between motorcycle and simulator test sessions and the subjective validation, where the riding feeling is evaluated by means of the subjective rating of test subjects.
The aim of the validation process may be better understood by looking at Fig. 5. On the top, it shows the interaction between a rider and a motorcycle: the rider controls the motorcycle (by moving the handlebar, by actuating the throttle or by braking etc.), the result is the actual motion of the vehicle and then feedback is given to the rider in terms of motion, sounds and visual cues. On the bottom, it shows the interaction between a rider and a virtual motorcycle (i.e. a riding simulator) with the same kind of human-machine interaction as in real conditions (Figs. 6 and 7).
Since it is physically impossible to reproduce accelerations as they are in real life using the simulator, it is fundamental to use a washout filter and properly tune it. The washout tuning has been carried out by team members who are also expert riders and engineers involved in motorcycle dynamics. Tuning was performed with particular attention to:
the perception of the speed;
the braking feeling and the feeling while riding on bumpy roads;
the feeling during transient cornering;
the vehicle responsiveness during lane changes, overtaking and obstacle avoidance manoeuvres;
the riding experience at low speed.
Besides the identification of the most suitable washout filter parameters, the tuning phase demonstrated that foot pegs control is very important for the improvement of rider feeling in transient motion and that the projection system using 3 widescreens greatly improved speed perception, even if it did increase simulator sickness.
After the completion of tuning, a final validation was conducted using a sample group of riders of different ages and levels of experience and skill. This was done by considering both objective and subjective data, as explained in detail in the next sections.
The objective evaluation consists of a comparison between the behaviour of the real and virtual motorcycles during the same riding actions. Despite the fact that there are many riding conditions with several uncontrolled parameters, the literature concerning the objective evaluation of motorcycle handling characteristics [7–27] helped us to focus on selected manoeuvres that are representative of the more general vehicle behaviour. In particular, the following three typical manoeuvres have been selected for the evaluation of the riding experience:
Slalom (three different cone distances);
Lane change (two different lane geometries);
Steady turning (three radii);
The above manoeuvres are also part of the set of manoeuvres commonly used by motorcycle manufacturers to develop their own vehicles. Tests were carried out by two skilled riders. The motorcycle used for the tests was equipped with a special handlebar with steering torque and steering angle sensor, foot pegs with load cells, GPS and an inertial measurement unit with accelerometers and gyrometers.
The slalom test was performed with three different distances between the cones on a straight line at established speeds.
For the sake of clarity, the results of both the simulator and motorcycle tests are presented in Table 1, indicating the most relevant parameters. Since it is practically impossible to reproduce exactly the same manoeuvre, first on the motorcycle and then on the simulator, the comparison between real and simulated manoeuvres is more meaningful when based on the ratio Gx/τ between the roll rate (which represents the vehicle behaviour) and the steering torque (which represents the rider action). Moreover, the ratio Gx/Gz between the roll and yaw rates and the phase lag between the Gx phase and the steering torque gives additional information. As the cone distance increases from 14 m to 21 m the magnitude ratio decreases, whereas there are only small changes in the phase difference. This can be observed from the values in Table 1.
Lane change manoeuvres may be classified by means of the width and length of the trajectory and vehicle speed, as shown in Fig. 8. In this case, tests have been performed using a lane width of 3 m and lengths of 14 and 21 m at speed range between 50 and 75 km/h. The manoeuvre can start from the right side and finish on the left side of the cones (right to left lane change) or the reverse. In addition, in this situation the speed should be kept constant as much as possible.
In the lane change manoeuvre the rider exerts some controlling action (torque) causing the vehicle to roll and yaw. The ratio of the peak-to-peak magnitude of steering torque to the peak-to-peak roll rate is a good indicator of a motorcycle’s manoeuvrability. Normalising this quantity by velocity we obtain the lane change roll (LCR) index, where the subscript p-p indicates peak-to-peak values [6, 12, 15, 16, 25]:
This index represents the effort required on the part of the rider in the form of steering torque to obtain a desired vehicle response in roll rate and can be used to contrast the behaviour of different classes of motorcycle: touring, sport, cruiser etc. Results of the simulator and motorcycle tests are compared in Table 2 where the range of the values for each parameter during the test is presented. This has been done in order to calculate the lane change roll index explained above and the data have been collected in Table 2 under the name MDRG Index. There are some differences in the index value, more so in the 3 × 14 m test than the 3 × 21 m test, but both indices are comparable with the typical value of the LCR index.
Finally, the steady state circular test (riding at a constant speed on a circular path) was conducted with different turning radii. For each test, the acceleration index [12, 15, 16, 26, 27] has been calculated, which is a manoeuvrability index that links the riding parameters as follows:
Where τ is the steering torque, v is the average speed [m/s] and R c is the cornering radius. According to the literature , the relationship between driver action and vehicle response can be quantified using the ratio between the steering torque and lateral acceleration, as shown above. The acceleration index is mainly negative (i.e. negative steering torque, outwards of the curve) and characteristically, for a given radius, it transitions from negative to positive (i.e. positive steering torque, towards the curve) as speed increases. Negative applied steering torque is preferable because in this situation the motorcycle’s tuning behaviour tends to be stable. Simulator and motorcycle test results are collected in Table 3. For each parameter, the average has been calculated to obtain the acceleration index explained above. In Table 3, the value of the negative radii corresponds to a counter-clockwise manoeuvre. The magnitude is always comparable with the typical value of the acceleration index calculated on different motorcycles [26, 27].
The aim of subjective evaluation is the enhancement of riding sensations in terms of visuals, acoustics and motion cues. It is worth highlighting that each different kind of cue has different physical and technological limitations; in particular, for visual cues there are limitations due to the need to stay true to the scenario being represented, as well as technological limitations in terms of resolution and the brightness capabilities of the visual devices. For acoustic cues there are technological limitations in the reproduction of the sound and noises of the environment; for motion cues there are both technological and (more problematic) physical limitations; indeed, the reproduction of acceleration is partial in amplitude and duration since the travel of the motorcycle mock-up is limited. Further limitations on the acceleration frequency bandwidth depend on the power of the simulator motor.
The riding sensations of the test riders have been collected by means of a questionnaire, which includes both technical questions and questions about perception and cognitive processes. The questionnaire was developed with the aid of two skilled riders who are also experts in motorcycle dynamics. The questionnaire (shown in Appendix 1) focuses on different aspects and situations including speed perception, the feeling accompanying braking and acceleration, the feelings of cornering and overtaking and obstacle avoidance. Moreover, for each situation is rated the fidelity of the simulator response to the rider input, the motion cues (in particular roll motion feeling and longitudinal acceleration feeling), and the audio/visual cues. The final validation was conducted on a wider user group of 20 subjects, aged between 20 and 60 years old, with different levels of riding experience but a minimum of 2,000 km per year. They had all held a valid riding license for at least 2 years, and were accompanied by a highly experienced rider (to avoid special biases induced by inexperience, problems with learning and becoming familiar with the equipment etc.). The test protocol is reported in Appendix 2.
The validation results collected are shown in Fig. 9 by using radar charts for the different evaluation points. It is worth noting that these results are consistent with the judgments that expert riders made during simulator tuning. Moreover, a couple of alternative configurations of the simulator were also tested and the score was lower, as foreseen during tuning.
The paper has presented the main features of the motorcycle riding simulator developed by the University of Padova, proposed a method for the objective and subjective evaluation of simulator riding feeling and presented the results of the validation of the DIMEG simulator conducted using a group of 20 riders of different levels of experience. The tests outlined have been done inside the 2BeSafe project in the Seventh European Framework Programme (theme 7—sustainable surface transport), and commenced in January 2009. 2BeSafe is a collaborative research project and its objective is to conduct behavioural and ergonomic research in order to develop countermeasures for enhancing the safety of powered two wheeler (PTW) riders. This has included research into crash causes and human errors, and the world’s first naturalistic riding study involving instrumented PTWs.
Objective validation demonstrated that the DIMEG motorcycle simulator reproduces with good approximation the physics of a real motorcycle. Subjective validation showed that, at the end of a trial and error tuning procedure, the audio visual experience and feelings of movement perceived by the rider had been remarkable increased. In particular, the riding experience has been improved by the installation of the visual system using three widescreens and the introduction of foot peg control.
Chiyoda S, Yoshimoto K, Kawasaki D, Murakami Y, Sugimoto T (2000) Development of a motorcycle simulator using parallel manipulator and head mounted display. Driving Simulator Conference, DSC 2000, Paris, France, September 6–7
Miyamaru Y, Yamasaky G, Aoky K (2002) Development of motorcycle riding simulator. JSAE Rev 23:121–126
Cossalter V, Lot R, Doria A (2003) Sviluppo di un simulatore di guida motociclistico. Proc. of the 16th AIMETA Congress of Theoretical and Applied Mechanics, Ferrara, Italy, September 9–12 2003
Ferrazzin D, Barbagli F, Avizzano C, Pietro G, Bergamasco M (2003) Designing new commercial motorcycles through a highly reconfigurable virtual reality-based simulator. J Adv Robot 17(4):293–318
Nehaoua L, Hima S, Arioui H, Seguy N, Éspié S (2007) Design and modelling of a new motorcycle riding simulator. Proc. of the American Control Conference, New York City, USA, July 11–13 2007
Cossalter V, Lot R, Sartori R, Massaro R. A motorcycle riding simulator for the improvement of the rider safety. FISITA F 2008-11-015
Cossalter V, Lot R, Doria A, Maso M (2006) A motorcycle riding simulator for assessing the riding ability and for testing rider assistance systems. 9th Driving Simulation Conference, Paris, France, October 4–6 2006
Cossalter V, Doria A, Lot R (2004) Development and validation of a motorcycle riding simulator. FISITA, Barcelona
Cossalter V, Lot R (2002) A motorcycle multi-body model for real time simulations based on the natural coordinates approach. Veh Syst Dyn 37(6):423–448
Cossalter V, Lot R, Maggio F (2003) A multibody code for motorcycle handling and stability analysis with validation and examples of application. Small Engine Technology Conference & Exhibition, Madison, WI, USA, September 2003, SAE 2003-32-0035/20034335
Koch J (1978) Experimentelle und analytische untersuchungen des motorrad-fahrer systems. Dissertation, Berlin
Zellner J, Weir D (1978) Development of handling test procedures for motorcycles. Warrendale, PA, SAE 780313
Weir DH, Zellner JW (1979) Motorcycle handling - volume I: summary report. U.S. Department of Transportation, 001-05, DOT HS-804 190 MISC
Weir DH, Zellner JW, Teper GL (1978) Motorcycle handling - volume II: technical report, U.S. Department of Transportation, No. 1086-1
Rice RS (1978) Rider skill influences on motorcycle manoeuvring. Warrendale, PA, SAE 780312
Kuroiwa O, Baba M, Nakata N (1995) Study of motorcycle handling characteristics and rider feeling during lane change. Warrendale, PA, SAE 950200
Bunz D, Klasen M, Schaffler A (2004) Determination of parameters for objective evaluation of motorcycles driving dynamics behaviour. 5th International Motorcycle Conference, Munich, Germany, Sept 2004
Varat M, Husher S, Shuman K, Kerkhoff J (2004) Rider inputs and powered two wheeler response for pre-crash manoeuvres. Proceedings of the 2004 International Motorcycle Safety Conference (Institut Fur Zweiradsicherheit e.v., Munich, Germany)
Cossalter V (2002) Motorcycle dynamics. Race Dynamics, Greendale
Sharp R (1997) Design for good motorcycle handling qualities. Warrendale, PA, SAE972124
Cossalter V, Da Lio M, Lot R, Fabbri L (1999) A general method for the evaluation of vehicle manoeuvrability with special emphasis on motorcycles. Veh Syst Dyn 31(2):113–135
Roe GE, Thorpe TE (1980) Improvements to the stability, handling, and braking of high-performance motorcycles. Motorcycle Safety Foundation Int. Conference, Washington Proceedings, 565–597
Schweers TF, Remde D (1993) Objective assessment of motorcycle manoeuvrability. Warrendale, PA, SAE 950200
Cossalter V, Lot R, Doria A, Maso M (2006) A motorcycle riding simulator for assessing the riding ability and for testing rider assistance systems. 9th Driving Simulation Conference, Paris, France, October 4-6 2006
Sadauckas J, Cossalter V (2006) Elaboration and quantitative assessment of manoeuvrability for motorcycle lane change. Vehicle System Dynamics: International Journal of Vehicle Mechanics and Mobility 44(12):903–920 ISSN 0042-3114
Cossalter V, Lot R, Peretto M (2007) Motorcycles steady turning. Journal of Automobile Engineering 221(Part D):1343–1356
Cossalter V, Doria A, Lot R (1999) Steady turning of two wheeled vehicles. Veh Syst Dyn 31(3):157–181
This project has been partially supported by 2BeSafe, grant agreement number 218703.
Appendix 1: Rider feeling questionnaire
|Subjective evaluation of simulator riding feeling|
My judgments of the Simulator Riding Feeling are... (tick one box in every line)
Appendix 2: Validation test protocol
This section describes the test protocol adopted during simulator validation in the 2BeSafe project. Before the test, the experimenter informs the participant on the framework of the project and of the aim of the experiment. In particular, the participant is informed that it is a test of the simulator and not a test of the rider. The test starts with the warm-up phase, followed by the evaluation one. The warm up phase aims to make the participant confident with the simulator commands: handlebar, brakes and clutch. During this phase the experimenter will ask the participant to perform some simple, specific task on a (virtual) free space. The warm up duration is approximately 8–10 min. The evaluation phase duration is approximately 15–20 min of driving in rural and urban environments. Each participant fills in the participant profile questionnaire. During the test the experimenter continuously assists the participant.
Warm up protocol
The experimenter demonstrates to the participant the simulator controls: handlebar and foot pegs, throttle, clutch and front and rear brake.
The participant gets on the simulator and starts riding in a (virtual) open space following the experimenter’s instructions:
Switch on the engine;
Accelerate and decelerate using the throttle;
Pull the clutch lever, enter 1st gear, release the clutch lever in the range between 3,500 and 4,000 rpm; start and maintain a constant speed between 60 and 70 km/h. If necessary and possible, enter 2nd gear using the clutch as explained above;
On the straight road, push the handlebar with the right hand (i.e. apply a counter-clockwise steering torque): the motorcycle will turn right. Release the handlebar: the motorcycle will come back to the straight motion;
Still on the straight road, push the handlebar with the left hand (i.e. apply a clockwise steering torque): the motorcycle will turn left. Release the handlebar: the motorcycle will come back to the straight motion;
Press the right foot peg: the motorcycle will turn right. Depress the foot peg: the motorcycle will come back to a straight motion;
Load the left foot peg: the motorcycle will turn left. Unload the foot peg: the motorcycle will come back to a straight motion;
Repeat and combine the tests above until you feel confident with the simulator controls;
On the straight road, at about 70 km/h, pull the front brake lever slowly; after that, repeat the same test acting more vigorously on the lever to feel the different behaviour of the pitch during braking;
Repeat the previous test using the rear brake on the straight road; after this combine the two brake actions;
On the straight road, accelerate until you enter 6th gear and after that brake using both brakes and the brake engine. Appreciate the varying behaviour in different gears during braking;
Repeat the previous test on a corner and feel the loss of adherence at the rear in lower gears.
After these initial rules, the test can start for the validation of the simulator following the evaluation protocol.
The first ride is done in the rural naturalistic scenario i.e. the participant has to ride in a (virtual) rural environment for 8–10 min, with low traffic conditions; the experimenter must remember to abide by the traffic rules and not to jeopardise him/herself or the other road users during the ride. Moreover, if the tester has any problems or feels sick during the ride he/she can stop the test immediately.
The second ride is done in the urban scenario i.e. the participant has to ride in a (virtual) urban environment for 8–10 min, with normal traffic conditions. In addition, in this case the experimenter must remember to abide by the traffic rules and not to jeopardize him/herself or the other road users during the ride. Moreover, if the tester has any problems or feels sick during the ride he/she can stop the test immediately.
After these two rides, the participant has to fill in the rider feeling questionnaire.
The previous 3 steps have to be repeated for each different simulator setup being tested as described above.