New Air Force Safety Tool Forecasts Mishap Risk

AURORA, Colo.—When you check the weather, the forecast projects the chance of rain based on barometric pressure, wind, humidity, and other factors. Now the Air Force Safety Center is offering a way to forecast accident risks, a dashboard intended to help mitigate against factors that historically indicate higher risks for mishaps. 

Unit Risk Forecasting is “not going to tell you where the lighting is going to strike, but it’s going to tell you when the weather’s in the area for a mishap to potentially happen,” AFSEC Chief Analytics Officer Laura Pick told Air & Space Forces Magazine at AFA’s Warfare Sympoisum last month. 

The new forecasting dashboard was rolled out across the Air Force in January, drawing on more than a decade of safety data to build profiles and forecasts for more than 2,400 squadrons across the department. It classifies risks for a squadron as low, medium, or high for mishaps over the next month. Using 12 years of data, the center found that 82 percent of class A mishaps occurred in squadrons that the model designated high risk, while none occurred in low-risk squadrons—despite the fact that only a quarter of squadrons were labeled high-risk and more than half were classified as low-risk. 

Even going down to the lowest mishap level, class E, nearly 76 percent of incidents occurred in high-risk squadrons, and only 7.8 percent occurred in low-risk ones. 

Using an open-source machine learning algorithm called XGBoost and Palantir’s “Envision” data analysis system, the model shows “risk reduction” factors and draws on four data sets:  

Air Force Safety Automated System, a historical database of mishap information 
Manpower Programming and Execution System, which includes information about units and their manning requirements
Military Personnel Data System, for information about military personnel 
Defense Civilian Personnel Data System, for information about civilian personnel 

“We went with personnel and manpower data first, because one, it’s what we had access to,” Pick said. “And it’s a great base to add things to, so we can standardize our metrics at that point. If we wanted to join in medical readiness data, we could do that. I think right now we are looking at ops tempo data, what that means to different groups, what kind of data sets we can pull in to support that. We’re also looking to spin off a different model that is aviation specific, so it would pull in any aviation-specific data source as well as maintenance.” 

Yet even just the personnel-focused data yielded insights once run through the model to find any evidence of correlation.

“It was very important that we get variables from the databases that we were pulling from that might have some connection to risk, either hypothesized or already established,” said Stephen Cook, a contractor from Booz Allen Hamilton supporting the project. “So what we did is we went through and we built variables off of those manpower planning and unit composition type databases … and then let the model decide what is important, because it’s really important that as data scientists or as analysts, we don’t let any bias sneak through.” 

The model produced a list of 50-plus variables that could drive risk, and Air Force Safety Center experts cut that list down to 30 or so based on overlap and some being rooted in others. 

On the dashboard that commanders and safety officers see, the model lists the top five variables or factors driving risk, along with AFSEC strategies and suggestions for mitigating those risks. 

“What we are most invested in … is providing you with your risk drivers and then providing you with strategies to mitigate those risks,” Pick said. “These [strategies] are not AI generated. We get this a lot. These are developed strategies by the safety center.” 

Some risk drivers are constant throughout the force. 

“Generally, the top couple are going to be seasonal,” Pick said. “Many units have a very seasonal curve to when there is risk, and that’s because of mission and the operational tempo that you’re experiencing.”

But other factors are more complex and specific to certain units—anything from the time since the last mishap to the percent of commissioned officers to discrepancies between number of enlisted personnel and the requirements set out in a unit’s manning document. Even the length of time a commander has been with a unit shows correlation with risk. 

“When there’s a new commander, there’s kind of a negative effect,” Pick said. “They’re new to the mission, that’s increasing risk, and then as they get farther into the seat, they’re more comfortable, it lowers that risk, even in high-risk squadrons. But then there’s kind of a spillover effect, they’ve been there for a longer period” and they can get too comfortable.

Wing and major command officials can use the platform to compare and contrast different squadrons, something Pick said has already resulted in positive feedback. 

When a new leader says, “‘Hey, I’m taking over as a wing or group commander. What am I walking into?’ This is something that provides that to them, so they’re able to see every squadron under their purview and get an idea of where the risk is at,” Pick said. “And that might not happen unless they had a really, really good breakdown from a previous commander.” 

The dashboard is available to anyone with a Common Access Card, wide access that the Air Force Safety Center hopes will result in feedback to improve the system. 

The team is also working to add more data sets so it can add new variables driving risk—Pick said the focus right now is on better quantifying operational tempo, a difficult task given that the there is no singular metric that tracks that. Instead, there are “different data sets, and they all mean different things to different types of units,” she said. 

Over time, Pick added, the hope is to extend the forecasting tool to cover not just one month, but two. 

For now, the model will update monthly with new data, and safety officials will keep refining. 

“We are developing features based on different data sets, and the model will allow things to fall off that are no longer predictive,” Pick said.