Data accuracy on automatic traffic counting: the SMART project results

Abstract

In this paper the results of an experimental survey on traffic monitoring devices, named SMART Project, are presented. Aim of the project was to define the requirements to be applied to traffic technologies to match specific applications. The project involved the construction of a test site on a main road, where seven different traffic monitoring systems were installed. A 1 year survey was carried out to assess technologies response to environmental and mechanical solicitations. A reference system specifically devoted to check and validate the devices under test was designed for the project. Test procedures were also developed to control the results achieved. The outcomes accomplished for vehicles counting are described in details in the essay hereunder.

Introduction

Traffic monitoring is a crucial task road agencies have to perform in order to improve operational and planning activities, as well as to supply policy makers reliable data to be used when attaining strategic decisions. Since 1970, the annual traffic census is accomplished according to EEC n° 1108/70, providing an estimate of the importance and usage of main national roads. So far Italian road agencies have been carrying out traffic census mainly by manual counts. The distrust toward available technologies and information has delayed the introduction of automatic monitoring devices.

In the last years, the rapid development of Intelligent Transportation Systems (ITS) has made available many technologies performing automatic traffic counts. Thanks to such systems it is now possible to carry out traffic monitoring and surveillance more efficiently. To test automatic traffic monitoring devices a research project was developed to draw up testing procedures and define technical requirements for traffic applications.

The SMART project’s objectives

The SMART Project was conceived to assess accuracy and reliability of some consolidated traffic monitoring systems in relationship to three main types of application: statistical (traffic census), road design and real traffic management.

The assessment was accomplished by means of an experimental survey carried out on seven different monitoring technologies:

• Inductive loops

• WIM (Weight In Motion) based on piezoelectric sensors

• WIM (Weight In Motion) based on quartz sensors

• Image recognition (video systems)

• Double technology (radar and laser sensors)

• Triple technology (radar, passive and ultracoustic sensors)

• Laser technology

The main characteristics of such devices can be found in [3].

The devices were tested on site for 12 months under different traffic and weather conditions, using an ad hoc measurement reference system and a series of validation procedures specifically developed for the survey [1] (see Fig. 1a).

Fig. 1

a View of the experimental site: on the left, the field laboratory used to store the collecting systems devices; on the right, some of the sensors installed at the site. b. The test site layout

The test site

The test site was located 23 km from Rome on a double carriageway road (S.S. 1 Aurelia). It was 200 m long and included an overpass bridge. The systems under test were installed on 4 sections on the same carriageway (see Fig. 1b). The first one was placed at the overpass bridge and hosted four devices: the video image recognition system, the triple technology system, the double technology system and the laser system. The second section was equipped with the weight in motion system based on piezoelectric sensors. On section three and four the inductive loops system and the weight in motion based on quartz sensors were respectively mounted. Each section was about 50 m distant from the previous one.

In order to check and validate the devices under test, a reference measuring system based on video cameras and active infrared lasers was set up. Two video cameras were mounted on an overpass bridge to record the traffic flowing on the road stretch where sensors were installed (see Fig. 2). The first camera was oriented to detect vehicles entering the test site and the second one those leaving it. To acquire images of the site also during night time, two infrared illuminators were added. In Fig. 2 the aforementioned video system is shown.

Fig. 2

Reference system: video cameras watching vehicles leaving the test site

The detected images were then automatically stored in a data logger put inside a field laboratory.

Laser diodes control systems were installed on the four sections and were specifically designed to check vehicle speed.

To guarantee temporal correspondence among data sets a centralized GPS time synchronization system was adopted.

Data check and counting

Vehicles counts were achieved manually checking the recorded video images: each vehicle was counted and associated to the correct lane at every passage over the detection areas.

Vehicles passing across the road centre line were opportunely assigned to a double virtual lane (one closer to the traffic lane and the other to the fast lane) and then redistributed to the real lanes after applying a checking procedure.

The procedure consisted in comparing the number of vehicles on the traffic lane detected by the devices under test with that manually counted. In case of comparable results, error on the traffic lane was computed according to § 5 and vehicles included in the virtual lane were assigned to the fast lane. Differently, if the previous condition was not met, data from the fast lane were compared with those manually detected. As above, in case of comparable results, error on the fast lane was computed according to § 5 and vehicles included in the virtual lane were assigned to the traffic lane.

Finally, if also data referred to the fast lane were not comparable, vehicles included in the double virtual lane were redistributed to both traffic and fast lanes.

Data collection method

Most of the tested systems count each vehicle passing across the detection area (vehicle by vehicle mode) and provide the results simply adding the single measured units in the sampling period.

Due to the large amount of traffic (about 30.000 veh/day) on the road stretch selected for the survey, many data including different traffic and environmental scenarios were collected.

The huge amount of data available allowed to assess devices performance in different traffic and environmental conditions, taking into account the sensitivity of the sensors installed at the test site to light, climate, weather and traffic flow. Data collected were stratified in terms of traffic and ambient conditions in order to guarantee their homogeneity and comparability as follows:

• traffic condition: low (volume ≤600 veh/h/lan), medium (600 veh/h/lan <volume ≤1,400 veh/h/lan), high (volume >1,400 veh/h/lan);

• lighting condition: day and night;

• weather condition: rainy and sunny/clear;

• climate condition: summer and winter.

Based on this stratification criterion 24 possible testing conditions were identified. Then, data were also evaluated with respect to 3 time collection intervals in order to assess the accuracy level of the systems in relationship to the following objectives:

• Real time monitoring (20 s collection interval)

• Road design (5 min collection interval)

• Statistical analysis (15 min collection interval; for it, only medium traffic flow was considered)

The resulting (24 × 2 + 8 × 1) = 56 measuring conditions are listed in the matrix reported in Table 1.

Table 1 Environmental conditions matrix set for the survey

Besides, for every device, 4 sets of data were collected and processed (4 matrices), with respect to:

• traffic lane;

• fast lane;

• carriageway (both lanes);

• average lane (average value coming from both lanes).

For any of the resulting 56 × 4 = 224 occurrences the statistical significance had been ensured gathering 25 samples. This implied dealing with 25 × 224 = 5,600 samples.

Testing methodologies and validation procedures

The goal of the experiment was to provide a quantitative performance assessment of the technologies under test and develop a method for their comparison in terms of precision, accuracy and reliability.

The method used to validate the devices was based on the calculation of the expected value of relative error, its error level and range. For every parameter measured by the tools installed at the test site a validation procedure was defined. Data provided by the reference system and fixtures under test were compared to determine the error affecting the devices for a known confidence level and degrees of freedom.

Each testing traffic monitoring technology was characterized by quantifying the divergence (e) between the detected and real values of the parameter to be validated. On the basis of [5], this value was computed adding the divergence between the outcome given by the testing technology and the reference value (e _str ), provided by the reference system, plus the divergence between the latter and the true value (e _rif ), that is to say:

(1)

The characterization of e _str and e _rif is given by:

• , s²(e _{str k} ) e s(), average value, variance of the observed data and standard uncertainty of e _str ; where str is the device index and k is the sampling interval.

• μ(e _rif ), σ²(e _rif ) e u(μ(e _rif )), expected value, variance and standard uncertainty of e _rif .

Being e the linear combination of e _str and e _rif , its statistical characterization can be described by the following equations:

(2)

(3)

(4)

where:

• u _c (μ(e)) is the combined standard uncertainty of μ(e);

• v(e _{str k} , e _rif ) is the covariance between e _str in the k sampling intervals and e _rif ;

• u(, μ(e _rif )) is the estimated covariance between and μ(e _rif ).

An assessment of the interval within which the values of the error can be confidently asserted to lie in is required. Such an interval, named confidence interval, represents the data range that, with a given probability (confidence level), includes the true value of the parameter and can be computed by defining the confidence level p affecting the t value with ν degrees of freedom; the expected value of the relative error is included in the interval:

(5)

which defines the quality of the characterization achieved for the mean error e; the term t _p (ν) u _c (e _m ) represents the error level of the estimation of the unknown true value μ(e) of the error e _m .

Moreover, it is important to define the interval of the possible error e values that can be collected. Being p the confidence level, the following:

(6)

defines the maximum and minimum error values that can be detected with p probability; i.e the range that usually is assumed as an accuracy indicator for the system to be checked. For further details see [1, 3, 4]. Analogous methodology was adopted in [2].

Vehicular counting accuracy

The vehicular counting accuracy of the monitoring systems can be evaluated by collecting data over many time intervals; for the k-th collecting interval, the relative error due to each system can be expressed as follows:

(7)

where:

n _str,k indicates the number of counted vehicle provided by the testing technology (str);

n _rif,k is the manually counted reference number.

The reference number for every class of vehicles in each interval was counted manually by visual inspection of the recorded video images relative to the traffic flowing across the test site sections. Manually counting is usually affected by human error that has to be evaluated; in a pre-test stage, the human error was statistically analysed by comparing counts given by three or more people watching the same recorded video images.

Indicating with n _i the number of vehicles counted by the i-th operator, the mean value for each class is given by:

(8)

where n is the number of operators. Thus, the maximum absolute relative error is defined as:

(9)

The human error was assumed to have a rectangular distribution with a mean value equal to 0 and lower and upper values equal to ±e_max.

The expected value of the relative error, its error level and the range within which the error can fall, can be computed according to (2), (5) and (6).

The results on traffic counting

The results reported in the following paragraphs refer to measurements collected during a 1 year test including different traffic, lighting, time interval, climate and weather conditions, as shown in Table 1.

Moreover, an “average profile” summarizing the behavior resulting from the average of the measurements collected during sunny/clear, winter, daylight conditions, was defined.

The results related to the devices were synthesized as shown in Fig. 4, where, for every collection interval, the “average profile” is expressed in terms of the expected error (e _m ) and its uncertainty (U). Values fluctuations due to variations of temperature (winter/summer), luminous intensity (day/night), traffic condition (low/medium/high level) are shown in scale reporting their increment/decrement with respect to the “average profile”.

The behavior of the device in severe weather condition (rain) was also evaluated from a qualitative perspective adopting a scale with 3 severity levels (high/medium/low).

Laser scanner system results

The laser system tested in the survey was composed of two devices mounted above the road on the overpass bridge inside the test site, at the centre of the lanes. Each device included six laser diodes that scanned the detection area, providing a three-dimensional vehicle profile (length, height and width). Vehicle characteristic data were processed by a recognition algorithm to supply vehicular classification. Additional parameters provided by the system were speed, headway, gap, presence of stopped vehicles and of vehicles heading in the wrong direction.

The outcomes of the survey relative to counting accuracy are reported in Fig. 3. As it can be seen, in the collection interval of 20 s the average error is equal to 40.1%. Due to such a high value the system doesn’t seem to be significantly influenced by environmental and traffic conditions. For larger collection intervals, the average error decreases to 24.1% in the period set for road design (5 min) and to 19.8% in that fixed for statistical analysis (15 min).

Fig. 3

Global results related to the laser scanner system tested in the survey

As shown in Fig. 3, the system performance depends on lightning condition. In summertime, when the intensity of the light is high, the average error increases with respect to its average profile, while in wintertime, when solar radiation weakens, its performance tends to improve. It can also be seen a slight difference in the behavior of the device between day and night; in particular, the performance of the system improves at daytime and worsen at nighttime.

Furthermore, regular traffic condition composed of vehicles moving at low speed, enables the device to acquire more vehicle information, so improving its performance.

No variation of the device response was detected in rainy condition.

Different results were observed between the devices installed on the two lanes. The device mounted on the fast lane provided far less precise results with respect to those observed on the traffic lane, so worsening the global average profile of the system.

Video image system results

The video image system installed at the test site was composed of a colour camera, a processing unit and a supervisor unit. The camera, set to control both lanes, was mounted at the centre of the gantry. The system was designed to count and classify vehicles, display average speed, vehicular density and traffic conditions. It was also equipped with a sophisticated image processing algorithm, based on neural network for motion detection and object tracking, to detect abnormal traffic conditions.

Video image systems performance is usually affected by temperature and lightning conditions, and, as it can be seen in Fig. 4, the results achieved for the tested device seems to confirm this assumption. At day time the system shows better performance with respect to the average profile of 34,1%, while at night its performance decays drastically, due to the dim lighting of the area.

Fig. 4

Global results related to the video image system tested in the survey

As many other electronic devices, its performance also reduces with high temperature, showing a better behaviour at winter time.

The system just allowed the collection of aggregated data within a minimum time interval of 15 min, so it was not possible to assess its performance within the time intervals of 20 s and 5 min.

No variation of the device performance was detected in rainy condition.

Double technology system results

The system was composed of two consolidated technologies, laser and radar, to guarantee the maximum reliability in detecting the most relevant parameters, such as vehicles count, classification and speed.

The laser scanner, installed at the centre of the gantry, was designed to control traffic flow, while radars, mounted at the scanner sides, were focused on measuring speed.

Speed measurements were based on Doppler effect, i.e. on frequency change between the signal emitted by radars and the signal reflected from vehicles passing through the propagation path. The frequency difference between the transmitted signal and the reflected signal is proportional to the instantaneous vehicle speed. Among non intrusive devices, the radar is the most accurate in measuring speed.

Vehicle classification and counting were accomplished by a laser scanner able to analyse the vertical profile of vehicles running on the carriageway. Vehicles classification was based on their height measurement. The longitudinal profile of the vehicle height allowed to reconstruct its shape and consequently to classify it.

This kind of system is known to be insensitive to environmental conditions, such as light, fog, rain and snow. As a matter of fact, the survey outcomes seem to confirm this trend. Little variations with respect to the average profile were detected for short time intervals. The best performance was recorded for long time intervals, where accuracy and precision assume their highest values (e_m = 2.6 ± 0.6%) (Figure 5).

Fig. 5

Global results related to the double technology system tested in the survey

Triple technology system results

The triple technology system installed at the test site was composed of two devices, mounted on a gantry above the street, at the centre of the lanes. Each device was made of three different kind of sensors:

• a radar sensor for vehicles speed measurements;

• an ultrasonic sensor, composed of an acoustic wave generator and a receiver, that measure the delay between the emitted and the reflected signal from objects moving across the detection area, revealing their passage and providing vehicles counting and classification;

• a passive infrared laser sensor for the measurement of occupancy, vehicles counting and classification in conjunction with the ultrasonic sensor. The sensor technology was made of an infrared sensitive material gathering the energy emitted from the road pavement or from the surface of the vehicles in the detection area. The occupancy of the road is revealed by their energy difference, that is directly proportional to the absolute temperature of the vehicle and the emissivity of its metallic surface.

The system was also able to measure vehicles headway and gap, and detect queue presence.

The performance of the system is shown in Fig. 6.

Fig. 6

Global results related to the triple technology system tested in the survey

As it can be seen the average error of counting measurements decreases as the collection interval widens, reducing its value from 10.9% (real time application) to 3.5% (statistical analysis). Besides, despite the system is known to be sensitive to temperature and air turbulence, its performance was not affected by climatic, lighting, traffic and rainy conditions, except for small variations detected in the narrow interval of 20 s.

WIM Piezoelectric system results

The WIM piezoelectric system tested in the survey was composed of two piezoelectric sensors and one inductive loop per lane. The piezoelectric sensors measured vehicle weight, speed, length and axes gap, while the inductive loop provided the occupancy time.

Data accuracy of this kind of system depends on the configuration installed. It is possible to adopt cheaper configurations with a single piezoelectric sensor and two inductive loops with less interesting performance.

To optimise its performance and accuracy, the system installed at the test site was equipped with a neural network algorithm to compensate sensors response to temperature variations.

Despite this compensation, the system seems to be extremely sensitive to temperature. As it can be seen in Fig. 7, the average error tends to decrease in summer, when the temperature of the road surface is high, and to increase in winter, when the temperature lowers. The average profile shows a better system performance for longer time intervals.

Fig. 7

Global results related to the WIM piezoelectric system tested in the survey

Variations due to light intensity and traffic are negligible. Modest changes with respect to the average profile were found in rainy conditions for short time intervals.

WIM Quartz system results

The WIM quartz system tested in the survey was composed of 2 bars of quartz sensors coupled with 2 inductive loop detectors on each lane. The quartz crystals were mounted on a strip of aluminum, covered with elastic material. Quartz is a material characterized by accurate piezoelectric properties, capable to generate an electric potential difference when subjected to mechanical solicitations.

The parameters measured by the system were vehicle count, speed, classification, weight, length, headway, vehicle and axis gap.

The results relative to counting measurements are reported in Fig. 8. As it can be observed, in the time collection interval of 20 s the average error is about 30.7%; enlarging the collection interval to 5 min, the average error reduces to 17.6% and to 12.3% when the time range widens to 15 min. The system performance is also resulted to be strongly affected by climatic conditions: specifically, a worsening of the performance was detected during winter (low temperature) and an improvement during summer (high temperature). The lighting and rain influence on the system performance can be attributed to temperature variations as well.

Fig. 8

Global results related to the WIM quartz system tested in the survey

Furthermore, in short collection intervals, regular traffic conditions, composed of vehicles moving at low speed, enables the device to acquire more vehicle information, so improving its performance, while high vehicular speed occurring with low traffic volumes tends to deteriorate it.

Inductive loops system results

The system tested in the survey was made of a couple of inductive loops of rectangular shape embedded in the road pavement. Vehicles counting and classification were operated sending to the loops a variable frequency signal from 10 kHz to 200 kHz, that resonates at frequencies depending on their inductance value. When a vehicle passes over the loops, a decrement of its inductance is generated. This decrement increases the oscillation frequency of the detection circuit, inducing an impulse indicating the presence of a stopped or moving vehicle in the electronic part of the device.

The system was able to count and classify vehicles, measuring speed, gap and occupancy.

The performance of the system is shown in Fig. 9. As it can be seen the average error of counting measurements decreases as the collection interval widens, reducing its value from 14.8% (real time applications) to 9.4% (statistical analysis). The device performance doesn’t seem to be significantly affected by lightning and traffic conditions. Differently, in summertime, with high temperature, the average error increases, while in wintertime, with low temperature, performance tends to improve. No variation of the device performance was detected in rainy condition.

Fig. 9

Global results related to the inductive loop system tested in the survey

Data analysis and final considerations

In Table 2 the average profile, the uncertainty and the environmental sensitivity of the devices under test are reported. Environmental sensitivity was assessed using a qualitative scale ranging from “very low sensitivity” (○), to “very high sensitivity” (●●●●●), through intermediate values expressed with a 0.5 score resolution (○ = 0.5; ● = 1).

Table 2 Average profile of the devices under test, expressed in terms of their average error (e_m) and uncertainties (U). Furthermore, a qualitative assessment of their sensitivity to environmental conditions is described using scores ranging from “very low sensitivity” (○) to “very high sensitivity” ●●●●●, through intermediate values expressed with a 0.5 score resolution (○ = 0.5; ● = 1)

As it can be seen in Table 2, better accuracies are related to those systems adopting different technologies, in particular when technologies are exploited to detect metrological parameters fitting the specific characteristics of the sensors (for example, laser for classification and counting, radar for speed, etc.). This assumption is confirmed by the results achieved for the double technology system, where counting and classification are operated by a laser scanner, reproducing the three-dimensional profile of the vehicles, and speed by radars.

This generalization seems to be untrue in case of the WIM Quartz system, where poor results, with respect to those attended, were obtained due to its high sensitivity to environmental conditions (see Table 2).

Data reported in Table 2 also show that for real time applications more demanding monitoring conditions are required. In particular, it can be observed that only one device (the one based on double technology) guarantees, for such an objective, an average error less than 10%. Differently, for statistical applications, four devices (double and triple technology, wim piezoelectric, inductive loops) are characterized by an accuracy with an average error lower than 10%.

However, it should be noted that the accuracy measured for the single device could be likely improved using a redundant system made of multiple measurement points.

For a global performance evaluation, systems accuracy should also be compared with their reliability. In this survey the most accurate and precise systems do not always meet good results in terms of robustness and reliability, as it can be seen from Table 3, where the parameter Mean Time Between Failure (MTBF) is reported. Data shown in Table 3 indicates that inductive loops are very reliable, while WIM devices seem to be characterized by poor performance. Good results were also obtained from double technology, laser scanner and video image processor systems.

Table 3 Reliability of the devices under test

Conclusions

In this paper an experimental survey, addressed to validate different technologies for automatic traffic monitoring, was described. Seven different systems were tested for a period of 12 months under different traffic and weather conditions. The parameters measured by the systems were assessed on the basis of procedures specifically developed for the survey and adopting a reference measuring system opportunely designed to obtain the reference values.

The results achieved from the survey reveal a different behavioural response of the systems to time collection intervals and environmental and traffic conditions.

Better accuracies were observed for the devices adopting multiple technologies, especially for those using sensors explicitly exploited to detect specific metrological parameters, while poor performance was achieved from the devices implementing a single technology. Accuracy and reliability do not always match, so the systems precision should also be compared to other parameters, such as robustness and MTBF.