Area-wide real-world test scenarios of poor visibility for safe development of automated vehicles

Automated research vehicles increasingly show higher levels of automation than present series-production vehicles [1, 2]. Even when using highly automated functions requires reliable reactions of the technology, because the driver only temporarily has to control the vehicle having a safe and collision-free journey [3, 4].

The safety significance to develop a safe system for real-world traffic that reacts safely under all weather conditions is evidently based on the first fatal crash that occurred in Florida 2016 on May 7 while driving with a vehicle in the so-called “Auto pilot” mode. According to the accident report, the driver of a passenger car died in a collision with a tractor trailer:

“Vehicle 01 (V01) was travelling westbound on US-27… proceeded to make a left turn … V02’s roof struck the underside of V01’s trailer … Driver 02 … was pronounced deceased …” [5].

Tesla Motors, the manufacturer of the car, subsequently acknowledged that the car was in “Autopilot” mode. The system failed to recognize a white object (the trailer) against a brightly lit sky (limited visibility), and therefore did not activate emergency braking. Meanwhile the driver was watching a movie when the accident occurred.

A second accident involving a vehicle which reacted inappropriately in automated mode happened on March 18, 2018 in Tempe, Arizona. A 49-year-old pedestrian, who was crossing the street pushing a bicycle, died after she was hit by an automated Uber vehicle. She was crossing the road outside a pedestrian crossing. According to the police, the vehicle was driving in automated mode. A safety driver was sitting behind the wheel. The pedestrian had stepped out of a shadow (limited visibility) on the roadway while the safety driver was watching a video, stated Tempe Police Departement. The National Highway Traffic Safety Administration (NHTSA) also investigated this fatal crash [3]. One assumption is that the sensors on the self-driving Uber car may have indeed detected objects, but evidently the interpretation software decided that no reaction was required (incorrect interpretation). Current software algorithms in the robot car must evaluate “false positive” objects on the road surface – such as plastic bags floating across the road – as harmless to prevent unintended emergency braking. In this case plastic bags hung on the handlebars of the pushed bike. Many years of main author professional experience in consulting automotive development processes show that the minimization of these exemplary above mentioned risks must be considered primarily [4]. Software algorithms must be developed on the basis of ethical and legal aspects. Safety-increasing measures ensure functional safety aspects of electrical and electronic functions. Automobile manufacturers must also take into account limitations how machines perceive, process and react adequately to their surroundings so that automated vehicles will conduct a conflict and collision-free journey [6]. In addition extended concepts for human machine interaction of highly automated functions are arising at takeover situations [7]. This will ensure continuous automation of all driving tasks to maintain vehicle control [4, 8]. Vehicles supported by partially or fully automated systems must – at the very minimum – fulfil the driving abilities of an alert vehicle driver, before considering series development. The measures necessary for ensuring a correspondingly high functional reliability extend from the development stage to the entire life cycle of automated vehicles, and especially its electronic components.

Vehicle manufacturers or suppliers perform various methods of risk management to increase automated vehicle reliability [9]. Amongst other measures (see Fig. 1) risk management takes real world scenarios based on accident data into account. However, until now mainly random samples of accident investigation have been implemented by different organizations [4, 10].

The current best-known methods for evaluating safety related systems and automated systems are dynamic forward calculations based on real pre-crash scenarios of traffic accidents [11]. These calculations are carried out by using various tools, such as rateEFFECT [12] or PreScan [13]. One of the biggest simulation databases, the VUFO (Traffic Accident Research Institute of TU Dresden GmbH) pre-crash matrix, was first introduced in 2013 and offers a range of about 5000 pre-crash scenarios based on the GIDAS database, which can be used for simulations [4, 14].

Furthermore, other institutions such as the Hannover Medical School, German insurance companies or vehicle manufacturers and some suppliers conduct traffic accidents on their own initiative [4, 15].

Accident databases can be divided into two different kinds: in-depth databases such as GIDAS (Germany), INTACT (Sweden), iGLAD (EU), NASS-CDS (US National Automotive Sampling System - Crashworthiness Data System) or CIREN (US Crash Injury Research and Engineering Network, and second national statistics (e. g. Destatis).

In-depth databases normally contain fewer accidents with many detailed variables (GIDAS in Germany investigates around 2000 accidents per year with up to 3000 variables). Conversely national statistics cover the huge amount of all recorded accidents (e.g. 2.4 million registered accidents in Germany) but only give limited information about these collisions.

In contrast to the two databases above, the scenarios in this publication provide both: a large database and more extensive information from police recording with regard to standardized validation- and testing. For the following analysis 1.28 million area-wide police accident data that was gathered between 2004 and 2014 from the Saxony State Interior Ministry (Sächsisches Ministerium des Inneren - SMI) has been used. The database covers all traffic accidents for the entire road network in Saxony. The Fraunhofer Institute for Transportation and Infrastructure Systems (IVI) provided exclusive access to the corresponding database. The process of this evaluation in cooperation with Fraunhofer IVI is based on 297 standardized accident types (UTYP).

Developing safe automated vehicles is a central requirement, which also means that the vehicle must make the right choices for its current environment. Safe driver cooperation and take-over interactions [4, 6, 16] have to be analyzed, if necessary [17].

2.2 Requirements for automated driving functions that minimize risk

The selected scenarios from Chapter 1.3 also support meeting requirements for automated vehicles. One minimum requirement that vehicles must meet is compliance with official directives and legal regulations.

Interdisciplinary coordinated development and approval processes are required for safe automated driving functions which permanently have to be adopted for new technologies. Standards and technical specifications with regard to automated or assisted vehicle functions have grown steadily over the years. As a part of the obligation to ensure traffic safety, new requirements for designing automated vehicles will be developed incrementally and previous approaches will be adapted. In particular minimizing risks, hazards or damage can prevent technical failures. Examples of these requirements in the European Union or the United States can be divided in two categories (see Fig. 2): type approval (grey) and duty of care (blue).

In general, a risk R can be formulated as a mathematical function F which consists of the severity S of an occurring damage as well as the frequency f a hazardous incident happens [4]:

$$ \mathrm{R}=\mathrm{F}\left(\mathrm{f},\mathrm{S}\right) $$

(1)

The frequency f is affected by varying factors. A further consideration, C (controllability), describes whether persons and road users are able to react in time, to avoid incurring potential injury or damage. To control the vehicle by a driver is not relevant here in terms of fully automated or driverless vehicles but for other persons could possibly be involved in an accident with this type of vehicle. One factor E (exposure) is how many times or the period of time a person is exposed to a hazard. The product of E x C is the likelihood that a defect will harm an effect in a specific scenario [4].

In addition the failure rate λ describes hardware random failures or systematic faults which can lead to hazardous events [4].

$$ \mathrm{f}=\mathrm{E}\times \uplambda $$

(2)

Furthermore, Failures in Time (FIT) of electronic or technical modules must also be analyzed for functional safety as defined in ISO standard 26,262 according to the International Organization for Standardization (ISO). The unit FIT indicates the quantity of devices that malfunction within 10^− 9 h [17].

$$ 1\ \mathrm{FIT}=\frac{1\ \mathrm{failure}}{10^9\mathrm{hours}\ \mathrm{of}\ \mathrm{device}\ \mathrm{operation}} $$

(3)

Thus one FIT corresponds to:

$$ 1\ \mathrm{FIT}=1\ast {10}^{-9}\kern0.50em \frac{1\ }{\mathrm{h}}\kern0.50em $$

(4)

Probability of occurrence f and – where possible – controllability C yield the Automotive Safety Integrity Levels (ASIL). There are four ASIL levels defined: ASIL A, ASIL B, ASIL C and ASIL D, where ASIL A is the lowest and ASIL D is the highest requirement. Either rating of ASIL B or ASIL C with a recommended probability of occurrence lower than 10^− 7 per hour – corresponds to a rate of 100 FIT) [17].

$$ \mathrm{ASIL}\ \mathrm{B},\mathrm{ASIL}\ \mathrm{C}<1\ast {10}^{-7}\kern0.5em \frac{1\ }{\mathrm{h}}=100\ \mathrm{FIT} $$

(5)

As already mentioned, the highest requirements are for ASIL D (required probability of occurrence is less than 10^− 8 per hour, which corresponds to a rate of 10 FIT).

$$ \mathrm{ASIL}\ \mathrm{D}<1\ast {10}^{-8}\kern0.5em \frac{1\ }{\mathrm{h}}=10\ \mathrm{FIT} $$

(6)

Apart from normal vehicle operation, ISO 26262 also takes into account service requirements, including decommissioning the vehicle. In this respect, developers have to consider the consequences of aging when selecting components. Control units or sensors have to be sufficiently protected though robust design. Any single failure must not close down any other safety-related functions (International Organization for Standardization, ISO 26262) [17].

2.2.1 Requirements for duty of care

To demonstrate the duty of care ISO several standards from the International Organization for Standardization (ISO) have to be proven as a state of the art requirement [18]. In the past several years, many ISO standards have been enhanced to accommodate new automated vehicle functions, which include: ACC - Adaptive Cruise Control (ISO 15622), APS - Assisted Parking System (ISO 16787), CSWS - Curve Speed Warning System (ISO 11067), ERBA - Extended Range Backing Aid (ISO 22840), FVCWS - Forward Vehicle Collision Warning System (ISO 15623) and FVCMS - Forward Vehicle Collision Mitigation System (ISO 22839) [6].

The ergonomic design of automated systems is also a key issue. Examples for standards based on ergonomic considerations of control systems as well as transport information are: “Calibration tasks for methods which assess driver demand due to the use of in-vehicle systems” (ISO 14198), “Specifications and test procedures for in-vehicle visual presentation” (ISO 15008) or a “simulated lane change test to assess in-vehicle secondary task demand” (ISO 26022). Central requirements for safe development are considered in standards such as the ADAS Code of Practice [19], ISO 26262 functional safety [20] or ISO/AWI PAS 21448 (Approved Work Item - AWI, Public Available Specification - PAS) [21] to support Safety Of The Intended Functionality (SOTIF).

The demands for automated driving can be ergonomically assigned to all three levels of tasks while driving. The focus is on the capabilities of sensor technology and data processing particularly with regard to those functions that relate to the primary driving tasks (Navigation, Maneuvering and Stabilization). Driving in these corresponding driving sections has changed significantly – especially in terms of supporting the maneuvering task – as compared to previous driving habits [22]. The aim is to focus on global technical harmonization of legislation, ethics, standards and tests (see Fig. 3) [17].

While ISO standards in the EU tend to have more of a minimum requirement character, safety standards set by SAE International in the US and Canada are seen as legally binding. SAE International initially was founded as the “Society of Automotive Engineers” (SAE) and organizes the preparation of technical standards for engineering professionals in various industries. Currently several SAE Standards for several functions, including “Adaptive Cruise Control” (ACC) and “Pedestrian Collision Mitigation System” (PCMS) exist (see Fig. 2).

2.2.2 Requirements for type approval

To bring an automated vehicle with all its modules to international market, it is essential to comply with the requirements of specific type approval regulations specific to each market.

Harmonized regulations apply to EU member states and other contractual partners. To receive type approval for motor vehicles, especially in terms of braking and steering as set by the “Economic Commission for Europe of the United Nations” (UN/ECE) must be fulfilled. Each country that joined the 1958 Agreement or the 1998 Agreement on Global Technical Regulations (GTRs) is allowed to test and authorize manufacturer designs. The Harmonization of Vehicle Regulations starts with regulation ECE R 1 (Headlights) and continues through ECE R 130 (Lane Departure Warning System LDWS) and ECE R 131 (Emergency Braking Systems AEBS).

ECE regulation number R 13 with uniform provisions concerning the approval for braking comply with automated driving. In contrast, ECE R 79 (Revision 2, Chapter 5) construction provisions with regard to steering equipment already include limitations for “low-speed maneuvering or parking operations”. Part 5.1.6.1 states: “It should be indicated to the driver and the control action should be automatically disabled if the vehicle exceeds 10 km/h by more than 20 per cent or the signals to be evaluated are no longer received.” To enable automated driving, the current limitation to drive slower than 10 km/h is planned to be removed for automatic steering functions [23].

The 1968 Convention on Road Traffic was introduced to improve safety of international road traffic by harmonizing traffic rules among contractors. It stipulates that the driver must keep the vehicle under control at all circumstances. An amendment in 2014 allows automated systems that the driver can turn off or override at any time. A future goal for fully automated vehicles is the modification that will call them to be treated like human drivers [24].

The vehicle of the future will no longer be an isolated means of transport, but rather will be an integral part of an integrated transport system in a connected-mobility world. Developing automated, autonomous and especially self-driving vehicles that drive reliably and safely under all conditions, is seen as an important component of predicted disruptive changes in the automotive industry.

In particular, development engineers working on perception and interpretation of complex traffic situations that involve difficult weather conditions are faced with considerable technical challenges rooted in ethics, legal requirements and social acceptance. Therefore, the provided scenarios include representative situations that can be transferred to similar road networks worldwide. These scenarios will be taken into account in during standards, development for early simulations as well as for subsequent real-world testing.

The 374 real-world scenarios considered that involved bad weather conditions were culled from the 1,286,109 police-recorded accidents that occurred in the state of Saxony over a ten-year period starting in 2004. A distribution of accident types under these circumstances shows that the driver lost control of the vehicle in 49% of the collisions. In particular left- and right-turn maneuvers or curves in roadways have to be considered as contributing factors (see Fig. 11).

Finally, the case-by-case analysis indicates only five collisions, where the drivers were able to mitigate the impact of an accident by implementing evasive maneuvers. Only 177 cases deemed relevant to prevent or mitigate collisions by evasive maneuvers. For a deeper understanding additional measurements and traffic simulations of the well-known accident locations must be analyzed, which were not considered in this paper.

The information covering the examined area-wide accident data in this analysis is limited to the degree of documentation depth in the respective police reports. A combination of traffic flow and detailed weather data for each single known event enables more precise information to be gathered in terms of general conditions related to the cause of each accident.

Furthermore, apart from actual accidents critical incidents involving successful evasive maneuvers must also be analyzed based on road-, traffic conditions and NDS data. It is recommended that geographic digital data from maps such as Tom-Tom, Google-Maps, OpenStreetMap or HERE be comprehensively linked with area wide accident collision and traffic-flow data that is based on mobile devices, vehicles or road traffic information. In the future, vehicle operation data and traffic simulations could be included as well.

Based on these relevant real-world scenarios the authors recommend further development of internationally valid guidelines such as ISO 26262 “functional safety”, the ADAS Code of Practice or ISO PAS 21448 to support the safety of the intended functionality (SOTIF). Additional virtual simulation techniques, driving simulations in virtual surroundings and Deep Learning methods to train automated systems will be included in final tests [28, 29].

In general, it is recommended to identify worldwide networks, collaborate with affected partners, engage government representatives, local non-governmental organizations (NGOs) such as European Association for Injury Prevention and Safety Promotion (EuroSafe, http://www.eurosafe.eu.com), European Automobile Manufacturers Association (ACEA, http://www.acea.be), European Commission – Road Safety, European Traffic Police Network (TISPOL, https://www.tispol.org), European Transport Safety Council (ETSC, https://etsc.eu), Fédération Internationale de l’Automobile (FIA, https://www.fia.com), National Highway Traffic Safety Administration (NHTSA, https://www.nhtsa.gov), National Safety Council (NSC, https://www.nsc.org), Network of Employers for Traffic Safety (NETS, http://trafficsafety.org), United Nations Road Safety Collaboration (WHO, http://www.who.int/roadsafety), US Department of Labor – Occupational Safety and Health Administration (OSHA, https://www.osha.gov) or the US Department of Transportation (DOT, https://www.transportation.gov). These organizations promote, spread and deal with road safety awareness. Many governments and authorities encourage the deployment of new technologies with the potential to save lives [30]. They work with industry, governmental partners, and other stakeholders to develop new technologies and accelerate their adoption in type approval regulations and standards [31].

Thanks to the Daimler Research, Daimler and Benz Foundation as well as Fraunhofer Institute for Transportation and Infrastructure Systems (IVI) for funding and supporting this research.