6 Steps in the Process of Mobility
Vehicles traveling through space follow a series of six steps: scenario, sense, perceive, plan, control, and response. The continuous cycle of these steps creates a Process of Mobility.
The Process of Mobility applies to 2-wheel, 4-wheel, and 18-wheel vehicles, or any powered vehicle that traverses through space. Historically, these six steps have all been conducted by humans, but the introduction of Automated Driving Systems (ADSs), allows the steps to be supplemented with technology. Understanding how humans conduct these steps is imperative to developing and implementing ADS technologies. In reviewing the activities related to each step, we start to understand why each step is important. It is also crucial to be aware of the research being conducted around each step, and what future work is needed to advance the technologies. GCAPS is conducting research into all six steps in the Process of Mobility, leading to an overall understanding where the whole is greater than the sum of its parts. Here, we will briefly examine the steps of the overall Process of Mobility to provide insight into related activities. In future blogs, we will go into more detail concerning our activities related to each step.
Scenario
The scenario step encapsulates the types of situations that the vehicle may encounter during its life. When working within the scenario step, we must consider the Operational Design Domain (ODD). The ODD contains all conceivable overlapping conditions, use cases, and restrictions that a vehicle may encounter1. The ODD is continuously evolving and has been considered in vehicle development ever since the first vehicle was created. Early vehicles had limited power, didn’t have dedicated marked lanes, and had to navigate through conditions that primarily included horses, pedestrians, and bicycles. These conditions didn’t require side impact crash testing or high speed stability consideration because vehicles never encountered those situations. As the ODD expands, it is necessary to consider performance for new ODDs. Vehicles now have different barriers and speeds under which crashworthiness is tested and vehicles are driven at speeds of up to 100 mph to evaluate their stability. Defining an ODD and the scenarios within it is powerful in that it provides the means to generate specifications and requirements to support vehicle development. For example, to support vehicle safety, current production vehicles need to satisfy performance requirements associated with specific scenarios, such as a sine with dwell maneuver (similar to a double lane change) from FMVSS No. 126, or for Automatic Emergency Braking (AEB) in the Euro NCAP regulations.
As the scenario space (collection of all scenarios) changes or evolves, the manner in which the vehicle system is evaluated with respect to the scenario space can change as well. Using a small number of specific test cases (i.e., small scenario space), physical testing could be used for vehicle development, testing, and evaluation. For example, it is relatively easy to place a soft vehicle (e.g., foam or balloon car) on the road, drive the vehicle being tested toward it, and measure the performance of the AEB system to evaluate compliance with Euro NCAP AEB regulations. But as the ODD and scenario space expands or vehicle model variations expand, physically testing all the conditions becomes impossible. This is when simulation becomes critically important. Simulation has been conducted for many years in virtual vehicle testing of specific scenarios and their variants. With the expansion of the ODD for ADSs, simulation is now used for examining scenarios across the entire scenario space to develop, test, and evaluate the vehicle technology. This is a necessity, as ADS-equipped vehicles have to deal with every scenario (known and unknown) not just a set of specific use cases defined by requirements or regulations. Being able to establish the scenario space using real-world conditions and data can provide a good basis for creating useful simulations.
Using Naturalistic Driving Data
Utilizing the Virginia Tech Transportation Institute’s (VTTI) 70 million miles of naturalistic driving data (NDD), GCAPS produces simulation-ready data sets that represent normal driving, near crash, and crash scenarios for inclusion in many simulation platforms. Scenario replication includes all items important to the event, such as dynamic actors, static actors, and terrain. Along with replicating these scenarios in simulations, being able to vary the scenarios intelligently, decide which scenario to use, and having a common descriptive language are all vital in properly using the resulting data for the development of a vehicle’s technology. GCAPS is performing work in all of these areas to identify, select, and utilize test cases within simulation as effectively as possible.
Sense
The next step in the Process of Mobility is sense. This step relates to how the vehicle operator collects data about the important aspects of the scenario. During the sense process, there are many things that need to be considered relative to research and vehicle design. Considering the human first, four of the five human senses are used during the Process of Mobility. Sense of smell might not be used often, but still contributes to evaluating the condition of the vehicle being driven (e.g., burning smells may indicate an overheated engine or burning brakes). When the human is operating the vehicle, the research primarily starts with human-machine-interface considerations, such as steering wheel size, location of buttons, etc. The vehicle’s dimensions, shape, and geometry must also be considered relative to senses. For example, if the slope of the windshield is too steep and the headliner location extends, the view of a traffic light may be blocked and the vehicle has just interfered with the sense of sight. Then there is of course the environment (weather, light, etc.), which can change what the human operator can see or hear. Lastly, the human operator’s state can also affect what they sense (tired, drowsy, distracted, etc.). The factors that affect the human operator can also be factors that affect the ADS operator, and thus need to be considered within vehicle sensing solutions. A wide variety of sensors are being added to vehicles, from simple accelerometers to inertial measurement units to high-definition cameras and LiDAR. These sensors can be impacted by the same things that impact the human senses. The geometry of the vehicle can block part of the camera’s field of vision or radar signal. The environment can cause interference in the LiDAR beam due to reflections from rain drops or sun glare could “blind” a camera. Understanding the sensor’s manner of “sensing,” the factors that can affect “sensing,” and then the sensor’s integration into the system are all key elements of research within this area. Utilizing VTTI’s Virginia Smart Roads, GCAPS has been conducting research into these topics as well as other characterization methods for implementing sensor models into simulation. Additionally, assessing the sensors relative to the scenario provides content and guidance on the sensors’ requirements.
Perceive
Continuing along with the Process of Mobility, once the scenario is defined and the sensors provide sense data to the operator, the next step, perceive, uses the information or data from the senses to identify the key attributes of the operating environment. Depending on the operator, this process is carried out by a computer or human, and involves separating the information into its key components. For human operators, processing is performed by the brain, which can be affected by many things from various substances to age. For the ADS or advanced technology, the processing is done by algorithms such as vision object detection. Within perceive, identifying an object is not always enough—tracking the object, understanding what is “real,” and what information is useful are crucial aspects of this step. In all these cases, the time to process the data will directly affect the next step, plan. Understanding how sensor information is used during the perceive step allows us to determine what is required in the sense step. GCAPS has developed object detection and sensor fusion algorithms to extract the key attributes and important information from on-road vehicle data for use in automated annotation or inputs to planning algorithms.
Plan
In the plan step, the human or computer needs to decide what to do with the information received in the perceive step. During on-road driving, the vehicle operator has to continuously update the plan as the scenario evolves. For example, a sudden cut-in requires a quick response to avoid a collision and the operator planning the maneuver has to adjust accordingly. The planning mechanism needs to decide whether the vehicle should change lanes or slow down or maybe even stay at the current location because of the surrounding vehicles. Once the vehicle starts to perform the maneuver, another vehicle may move into the desired space, requiring an adjustment to the plan. Further, when merging onto a highway, planning starts as soon as the vehicle enters the on-ramp, and evolves as the vehicle continues through the merge process. The driver’s behavior, which could be aggressive, conservative, experienced, or inexperienced, can dictate how other actors will react, which may also require an update to the plan.
Determining Vehicle Capabilities
When using technology, implementing a binary approach for maintaining recommended safe distances may be too restrictive. Utilizing the VTTI NDD, GCAPS uses statistics from scenarios to feed behavioral-based algorithms as well as safety model approaches to support the planning phase. One might assume that an ADS-equipped vehicle doesn’t need to worry about the chassis’ performance because it is always monitoring and planning, but this performance knowledge is actually crucial to this step. Knowing that the vehicle can achieve a lateral displacement of 4 meters in 0.1 seconds instead of 0.2 seconds can dramatically change the decision between changing lanes or slamming on the brakes to avoid a vehicle. For proper vehicle dynamics prediction, the tire is critically important. The tire is the only component that can produce forces at the ground to change the vehicle’s trajectory. Knowing what the tire can do will directly affect the vehicle’s planning. Can the vehicle stop in 40 meters under dry highway conditions? What if the conditions are wet? In addition to supporting planning through algorithm development and safety model evaluation, GCAPS also operates the world’s most capable indoor tire testing equipment to quantify the tire’s performance and create high fidelity tire models. Through the knowledge of the tire, GCAPS supports improvements to the vehicle’s dynamic response and to the fidelity of the planning step.
Control
Once the decision has been made as to the vehicle’s environment and what the vehicle should do, the control step executes that decision. This involves steering the wheels, and accelerating or decelerating the vehicle. Sometimes the scenario requires the controls to be as fast as possible to avoid a crash. In this case, safety is the only consideration; driver or passenger comfort are not considered. But in many other normal driving scenarios, the rate at which the planned maneuver is conducted will be affected by the driver’s behavior and may take into account driver/passenger comfort. If a single person is driving, they may be more aggressive when changing lanes. But if this same driver has passengers, the rate of the maneuver may be less aggressive in order to keep the other occupants more comfortable. The chassis’ vehicle dynamics will also contribute to maneuver rates. Not all human drivers will be aware of the chassis’ capabilities, but ADS technology certainly should be. As ADS technology is implemented in stages, a blend of human control and technology control occurs. Knowing when the technology can control a scenario and when human control is required is a difficult assessment. The continuing research into when and how this occurs has been one of the larger challenges for SAE Level 1-3 ADS-equipped vehicles. This handoff between technology and human feeds back into the plan step as well. For Level 4 and 5 vehicles, the control algorithms need to account for the awareness a human driver has for their occupants’ comfort. Most people will not appreciate being driven by an ADS as if in an F1 race, which may result in objects being thrown around the vehicle and possibly causing nausea or other discomforts. Using the same statistics gathered from VTTI’s NDD as in the plan step, GCAPS has been conducting research and developing behavioral-based control methods via simulation, integrating vehicle dynamic modeling, occupant comfort, and driver behavior.
Response
The last step in the Process of Mobility is response. This is when the end results of the prior steps are evaluated. When driving on-road and interacting with all the dynamic actors, static actors, and terrain elements, the evaluation process can be a bit convoluted. When a human driver changes lanes and they do so at a rate that keeps them comfortable, does that mean that they performed a successful lane change? What if the act of changing lanes actually caused another person to react in a way that resulted a crash? The person initiating the lane change may not even realize that this happened. The ADS technology, however, will need to be aware of this possibility and assess its choice of actions. This raises the question of whether this might lead to ultra-conservative algorithms. We must consider the basis for the vehicle’s performance in each scenario. Again, we come back to the 70 million miles of data from VTTI’s NDD. With the statistics from the data, a basis of performance can be established. Once this performance basis is established, testing, either physically or virtually, can be performed. Utilizing the simulation-ready data sets, statistics from the NDD, and vehicle dynamic knowledge, GCAPS conducts behavioral competency analysis of advanced vehicle technologies. To complement the analysis, GCAPS physical testing expertise has led to techniques to test and evaluate advanced and ADS technologies.
Cycle of Steps
The steps within the Process of Mobility are a constant cycle. As the vehicle moves through space, it experiences a sequence of scenarios that can be navigated with this process. For example, when driving along a highway and exiting, the driver may experience at least three scenario types: first lane keep, then change lanes, and then move into an exit lane. As the vehicle navigates through these scenarios it can face a variety of challenges from other dynamic actors, static actors, and road conditions that need to be addressed. The scenario step starts the process in which dynamic actors, static actors, and terrain all interact to create a set of conditions within which the vehicle must navigate. The sense step provides a means to collect the required data to successfully navigate the scenario. The perceive step interprets the data from the sense step and creates the understanding of the environment required to successfully plan the navigation actions. In the plan step, the desired navigation actions are determined based on an understanding of the data created by the perceive step. The understanding of the scenario includes information about historical and predicted ego and dynamic actor trajectories, vehicle information, behavior, the location of relevant static actors, and the road geometry and conditions. The control step executes the navigation actions developed from the plan step, considering vehicle capability, behavior, and comfort. Finally, in the response step, an evaluation of the ego vehicle’s actions and associated vehicle responses is conducted to determine if the vehicle is progressing through the scenario in a safe or unsafe manner. GCAPS has knowledge and experience in supporting the development, testing, and evaluation of the various components associated with the Process of Mobility.
Further Considerations
As we move closer and closer to technology transporting humans, we need to decide if we want automated technologies to behave like robots or humans. Assessing what these technologies’ behaviors and performance should be if based on how humans behave and perform is one of the challenges. The VTTI NDD provides many of these answers by recording human driving response, behavior, and performance. GCAPS recognizes the power of implementing this into many ADS-related research and development projects. Further, GCAPS has formed a strong relationship with VTTI to advance research, testing, and simulation practices for the mobility industry.
Look for our Tech Takes, which will dive into the details of identifying, describing, creating, varying, and selecting scenarios for the development of ADS technologies.
