Over the last 100 years, the transportation industry has been developed primarily for human operators. For example, roadway signs were designed to be easily seen, read and interpreted by humans. However, as vehicle automation technologies are evolving, driving responsibilities are beginning to shift from the human operator to the automobile.
The industry is rapidly moving toward vehicles equipped with automated driving system (ADS) technologies to navigate the roadway environments without human involvement. A key concern for this inevitable mass deployment of automated vehicles is earning public confidence that ADS-driven vehicles can safely and reliably operate in a mixed environment of automated and human drivers, with the expectation that the ADS systems perform equal to or better than their human counterparts.
In this vein, developers of an ADS and infrastructure owner operators (IOOs) that manage the roadways aspire to safe and efficient operation of automated vehicles. For this to occur, significant testing must take place. The USDoT’s Federal Highway Administration’s (FWHA) Testing and Pilot Design, Development and Evaluation Framework (“the Framework”) project for ADS-roadway interfaces is available to assist in collaboration among developers and infrastructure organisations.
The goal of this project’s research is to develop a framework that provides support in the creation of test and pilot programmes, the outcomes of which can benefit both ADS and infrastructure entities. This article outlines the aspects of the Framework.
Background
With no standards or evaluation procedures for ADS testing and no collaboration between ADS development and infrastructure stakeholders, the widespread deployment of this technology might never be achieved.
The successful operation of ADSs on our roadways is heavily dependent upon the ADS being able to interpret and navigate its surrounding environment. However, with the development of ADS technology still in its infancy, there are still many challenges to overcome, such as ADS navigating complex traffic scenarios, adapting in adverse weather, and building supportive infrastructure. Given that ADS developers and roadway stakeholders seem to own diverse, but complementary, visions on the testing and evaluation needs, the likelihood of safe and efficient deployment is necessarily contingent on productive industry-wide collaboration.
To address these challenges, FHWA is developing a Framework for successful testing and evaluation of ADS and roadway features that emphasises collaboration among the ADS and infrastructure organisations (FHWA-HOP-21-012).
“Historically, collaboration among ADS developers and IOOs has not been the norm”
The Framework is technology-independent and consists of a broad set of processes and examples that advise without prescribing regulations or policy. Instead, the Framework adopts the notion that validation and testing of ADS technologies and various infrastructure features are essential to paving the way for safe deployment of ADS-equipped vehicles into the road network. Aside from the sheer scope of the possible testing that may be needed, a common understanding of the capabilities that need to be implemented either in the ADS or roadway domains is required.
Collaborative testing promises to provide resolution to these issues, resulting in a safe and efficient deployment of ADS. Numerous stakeholders have recognised the need to investigate various ADS and roadway scenarios, including specific challenges to be addressed. Overall, these efforts will further advance deployment of ADS-equipped vehicles onto roadways throughout the US, with desired outcomes benefitting both ADS and infrastructure entities.
Introduction
Consider a typical state DoT operations scenario where a busy highway is being resurfaced or expanded. Presently, the procedures and standards outlined by the DoT are designed to maximize the safety of the crew within the workzone, while still ensuring continual mobility for drivers.
For instance, one direction of travel is shifted from left over the centre line into a lane typically used for the opposite direction of travel (see Figure 1 below). This scenario is common across the US and poses some clear safety concerns.
The work crew must ensure that enough advanced warning of the lane shift exists and that visible barricades are distributed with proper spacing, while drivers must steer through the exchange without hitting or crossing the barrels, which could cause injury to workers, damage to equipment, or collision with oncoming traffic.
DoTs can expect that drivers have enough experience with workzone navigation to understand how to proceed safely. Yet, what might happen if the driver is altogether removed from the situation? Can anyone be certain that the automated vehicles of tomorrow will be able operate safely through a workzone? The answer lies in the quality and scope of testing that will be performed for the ADS that is expected to displace human drivers over the course of the next two decades. Both the developers of the ADS and the workers within the DoT have a keen interest in how this testing is administered and evaluated.
Framework overview
The Framework provides a broad suite of tools, considerations and approaches intended to facilitate collaboration between the ADS developers and roadway stakeholders. As demonstrated in the workzone navigation example provided above, both ADS developers and DoTs have vested interests in the safe deployment of ADS technology.
Thus, the Framework was developed with extensive engagement and input from both ADS and roadway stakeholders, including automotive OEMs, suppliers, technology companies, and state, federal and regional government entities. It is far more likely that their combined perspectives can accomplish these goals more rapidly, and with fewer errors, than the respective individual entities.
“With no standards or evaluation procedures for ADS testing and no collaboration…the widespread deployment of this technology might never be achieved”
As shown in Figure 2 below, the Framework addresses nine over-arching themes, which support the four key elements and themes during various aspects of the test phases. The Framework embodies the contextual examples, real-world lessons learned, and various considerations, as they will vary across different tests and disciplines (e.g. ADS system developers, IOOs, first responders and fleet operators). All disciplines are interested in - or will be impacted by - ADS deployments on the road network. Each party has a responsibility to engage in the testing and evaluation process for automated driving systems, and this Framework enables collaborative support toward the shared goals.
Themes
Collaboration between ADS and IOO stakeholders is critical for successful testing and evaluation. Stakeholder collaboration allows for early detection and resolution of ADS issues related to technical, organisational and strategic test implementations. Collaboration allows testing participants from diverse organisations, backgrounds and skillsets to
solve specific ADS/roadway challenges. Open and frequent interactions lead to improved test outcomes. The quality of the testing is also enhanced because input from a stakeholder from another point of view can be collected.
As an example, Pennsylvania DoT (PennDOT) has already recognised the need to prepare for the mass deployment of ADS across the state. Therefore, PennDoT assembled nine partners (Blazina, 2019 and Paez, 2019) to work on changing infrastructure to support ADS in workzones by using distinctive coats for lane markings and barrels to aid in detection by the ADS.
In addition, the project will develop advanced mapping and communications systems for safe ADS navigation at and around 17 different workzones across urban, suburban and rural areas. Prior to testing in active workzones, the partnership will conduct validation in virtual environments, then at a track at the Pennsylvania State University. By engaging automation development teams and creating a comprehensive plan for testing, PennDoT will have a better understanding of how ADS systems operate in certain conditions, and thus will be better-equipped to fulfill its mandate for keeping state drivers safe and informed.
“As vehicle automation technologies are evolving, driving responsibilities are beginning to shift from the human operator to the automobile”
Another example of collaboration to achieve roadway safety goals is the Michigan DoT (MDoT)/3M Connected Roads I-75 Test Corridor. MDoT collaborated with 3M to deploy a 100-day test of 3M connected roads prototype solutions in a 3.3 mile construction workzone. The collaboration took place between 3M, MDoT, and a variety of automotive OEMs and sensor suppliers.
Common Ground refers to creating a common or shared working environment so that ADS and IOO stakeholders fully understand each other, which is critical for tests to be successful. When executing ADS/roadway tests, all parties will have clearly defined expectations, outcomes, and success criteria. There are three key components of Common Ground: 1) common goals and benefits; 2) common terminology; and 3) common metrics and measures.
Test Logistics refers to what to test, how to test, and where to test. This includes development of test scenarios, testing methodologies, and test environment. Test logistics is tailored to specific test scenarios and what aspect of ADS and roadway is being tested.
Institutional / Organisation Issues
Having organisational experts from both the ADS and IOO organisations participate early and throughout the test phases will greatly aid in navigating any challenges. The Framework adopts the premise that for ADS/roadway testing and evaluation, safety of all road users is the greatest priority. Within this premise, state and local regulatory policies must be developed, which requires that policy makers be well-informed on relevant topics and are kept up to date on developments. The Framework aids in navigating challenges that IOO and ADS representatives encounter when conducting tests, evaluations and pilots.
Roles and Responsibilities
In the process of ADS/roadway testing and evaluation it is important to identify who from the various organisations needs to participate, what roles within the organisations are needed, and when (i.e. which phases) they need to participate. Some participants may only be involved in one phase while others may be essential to all phases.
Plans
For a collaborative environment to exist, stakeholders’ participation as part of the test design plan, data collection plan and evaluation plan can be beneficial. For instance, ADS/roadway test and evaluation needs to account for roadway adaptations, which can be incorporated through collaboration with IOO.
One example is Waymo’s First Responder Engagement Plan. The objective of this plan is to provide first responders with the knowledge they need to safely identify, approach, and interact with a vehicle equipped with ADSs in an emergency scenario.
Sharing Opportunities
Data is a key issue that requires thorough discussions from IOO and ADS stakeholders to avoid challenges (e.g. proprietary data/information, use of data). Data is critical to evaluate the outcome of tests and data is key to effective road network operations. It is possible that ADS and IOO stakeholders can share a variety of resources. Resource sharing includes sharing of skills and expertise in addition to sharing of information and existing data.
An example is Arizona IAM Consortium Collaborative Data Sharing. Arizona’s IAM consortium is leveraging existing infrastructure to collect performance data on public roads. The research relies on a low data ask from collaborating partners. This may encourage greater openness and participation from ADS stakeholders.
A New Driver
With ADS, the new driver of tomorrow will be the vehicle. This Framework is an instrument that provides examples and scenarios to those conducting tests and pilots to prepare for a safe and functional road network with, at first, both types of drivers (i.e. human and ADS) sharing the road, and then aid in transitioning the network into the new normal.
Success Factors
Many factors influence the success of ADS and roadway testing and evaluation. The most critical success factors include enhancement of technical maturity, comprehension of ADS, roadway test elements, the process, stakeholder engagement and collaboration, and ongoing public communications. The Framework aids in assisting the ADS and IOO participants in defining test success factors within each test phase.
Framework Application
The activities for successful collaborative testing and evaluation can be categorised into four phases as shown in Figure 4.
Pre-Test Phase
The output of the pre-test phase is a clearly-identified problem statement that is based both on the internal needs of the ADS developers and the industry. A clearly-defined problem statement supports collaboration with IOO and others to then work on a test definition (the next step). Programme risks, including operational, technical, data, legal and financial, should be identified during this phase. Overall, this collaborative effort is designed to expand the scope of those impacted by the testing effort in order to minimise longer-term issues and delay.
Test Definition Phase
The objective of the Test Definition phase is to conduct activities, which help define the technical and data facets of a collaborative test programme. The completion of the Test Definition phase produces a Test Plan, a Data Management Plan (DMP), and a Quality Plan, which facilitates subsequent test execution. Critically, the success factors, both technical and organisational, are defined for the programme. Having clearly-defined success criteria ensures an overall higher test quality.
Test Execution Phase
In the Test Execution phase, both technical and data facets of the collaborative ADS and roadway testing proceed as defined in the Test Plan. The focus is on efficient collection of performance data. The phase includes operational collaboration, ongoing communications among stakeholders, and monitoring and adjustments to the test. After completion of this phase, ADS performance data is gathered and reviewed to determine if the system is ready to advance to the next phase. For example, if the performance failure at night is considered critical by the stakeholders, then it is likely new tests will be scheduled after updates are made.
Post-Test Phase
In the Post-Test phase, stakeholders aim to close the collaborative testing and evaluation activity. They review data insights, store data, and discuss lessons learned. Equally important is the evaluation of the processes used, which drives future programmes. Overall, this collaborative effort directly leads to a higher level of confidence by the stakeholders, and by extension the public, that the ADS-equipped vehicles will be safely deployed among existing drivers.
Conclusion
Historically, collaboration among ADS developers and IOOs has not been the norm, nor has testing of ADS-equipped vehicles with varying transportation infrastructure components. ADS technologies have been developed and tested, traditionally, with little or no discussion with roadway stakeholders. Communication between IOO and ADS stakeholders has focused mainly on gaining approval to test on the road network. Proprietary data concerns have been identified as a roadblock to the willingness of ADS developers to work collaboratively with IOOs.
However, the Collaborative Framework for Automated Driving Systems/Roadway Testing addresses collaboration from multiple perspectives throughout the test lifespan. It provides examples where collaboration has yielded successful outcomes, the benefits of collaboration, and information on how and when to collaborate.
Successful collaboration among stakeholders will allow for early detection and resolution of ADS and infrastructure issues related to technical, organisation and strategic test implementations. Ultimately, this helps focus the efforts of both the ADS development teams and IOO stakeholders working toward the common goal of safe and efficient deployment of autonomous vehicles.
ABOUT THE AUTHOR
John Harding is USDoT/FHWA Office of Transportation Management, Connected/Automated Vehicles and Emerging Technologies team leader
Click here to see Collaborative Research Framework for Automated Driving System Developers and Infrastructure Owners and Operators
