At a Pentagon press conference on February 12, 2002, when asked about the probability that Iraq had obtained weapons of mass destruction, Donald Rumsfeld said the following:

“There are known knowns. There are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know”

At the time, pundits mocked him for this statement and many people viewed it as handwaving nonsense meant to cover over reality. But with time, this statement started to gain respect. In actuality, he was trying to represent the important grey area in risk assessment that we have all become quite familiar with. Some people say, “I don’t know what I don’t know” and others call it second-order ignorance. 

VitreosHealth has been a pioneer in the healthcare predictive risk analytics space. Our mission has remained the same from day one of the company – Predict, Intervene & Prevent avoidable poor outcomes. Based on our 8 years of experience implementing AI-based predictive models in healthcare, we have also come across numerous unknown unknowns. By sharing these, I am hoping that I can assist providers and payers avoid these mistakes and minimize wastage and inefficiencies.

These unknown unknowns of predictive analytics in healthcare fall into three categories:

  1. Efficacy Traps: doing things in a way that ensures a reduction in negative outcomes (read more about those here)
  2. Efficiency Traps: doing things in a way that minimizes resource wastage (read more about those here)
  3. Effectiveness Traps: doing things accurately

Care Management Effectiveness Traps and the Importance of Predictive Accuracy

After we figured out how to be more efficient, the next step was to maximize care management resource effectiveness. In predictive modeling, effectiveness is directly tied to the accuracy of our models. We needed to be able to assimilate all this information and come up with one risk score that predicted poor outcomes – emergent events, high costs, etc. Exhibit 2 shows the prioritized ranks for the 11 members from Exhibit 1 using an Average Clinical Risk Score, an HCC Risk Score, and an AI-based Predictive Model for Avoidable Outcomes. The customized AI-based predictive model used EMR data, external demographics, labs, pharmacy data inputs that include all the disease specific scores, non-clinical risk scores, gaps-in-care along with other social determinants data. Each of the three risk scores are calculated for each patient and then each patient is ranked for prioritized outreach. A high-risk score will result in a top rank (indicated by a low number) since those patients need immediate attention from a care manager.

Exhibit 2: Ranks of the 11 patients from Exhibit 1. A low number in yellow indicates a top clinical risk rank and a high clinical risk. A low number in grey indicates a top HCC risk rank and a high HCC risk. A low number in black indicates a top AI-based predictive model rank and a high risk for avoidable emergent events.

Let me illustrate the unknown unknowns in interpreting these different scores and linking it to the predictive risk of poor outcomes.

  • Use Case 1: The Chronic Clinical Risk score and the HCC Risk score track the Critical and Healthy members well (Members A, B, C, K) as they are based only on presence of chronic conditions
  • Use Case 2: HCC fails to capture the latest State-of-Health of a chronic condition unlike EMR-based Clinical Risk scores using latest vitals and lab results (Member E). Member E has a high clinical risk score based on vitals collected from EMR data, but HCC does not factor in latest vitals and lab results to determine the state of a specific disease at any point of time.
  • Use Case 3: The AI-based model captures all clinical, non-clinical and utilization patterns and hence identifies opportunities missed by EMR-based clinical risk scores and HCC Utilization risk scores (Members F, H, G). Hence AI-based models have much higher accuracies in identifying rising-risk member cohorts.

 VitreosHealth’s AI-driven customized models are delivering accuracies that are 2-3 times better than those of CMS HCC, CDPSRx, etc. These accuracies translate to accomplishing the same results in outcomes with half the number of care managers. See Exhibit 3 to illustrate the ROI in care management resources to achieve a targeted savings through proactive member engagement.

Exhibit 3: Care management productivity is directly tied to predictive accuracy. In this graph, I am trying to illustrate how four different models with different predictive accuracies can give you different results. Suppose you only have the resources to target 20% of your population for proactive outreach. You would want to choose a model that helps you pick the top 20% of members that are most at risk for avoidable emergent visits. If you choose which members to focus outreach efforts on by using CMS HCC, you will most likely only prevent 15% of avoidable ER events, while if you used Vitreos Regression models, you could prevent around 38% of emergent events. Our new AI models will give you the ability to prevent 65% of ER events. A perfect model (powered by whatever deity you choose to believe in) will probably help you prevent 85% of emergent visits (in our experience, the top 20% of members result in 85% of avoidable cost).

For a typical Medicare Advantage Plan with 10,000 members, preventing 40% of ER events results in $6 million in cost savings. You need a smaller care management team to achieve that $6 million if you are using higher accuracy AI predictive models (since you only need to touch around 10% of your population) as compared to HCC models (since you need to touch about 40% of your population).  In the above Medicare Advantage example, you need a care management team with 17 members if you are using CMS HCC risk stratification compared to only 8 care managers if you are using Vitreos AI-based risk stratification predictive model.

After spending 2-3 years and millions of dollars in predictive analytics, you as a leader is thinking, “What have I gotten myself into?”. This is the point at which most risk-based organizational leadership teams realize the need for an analytics partner that is experienced and well-versed in helping their team members make strategic data-driven decisions at every level early in the process.

Partner with the right experts who can transform your unknown unknowns to their known knowns. Leverage the collective learnings of your predictive analytics partner’s multiple customers to avoid the emerging technology pitfalls. Pick a partner whose core competency is Healthcare Predictive Analytics.

Did you miss parts 1 and 2 of this series? Read all the parts here.

Are you ready to learn more? Fill out your info and we will contact you shortly! 

The Unknown Unknowns of AI-based Risk Stratification in Healthcare – Effectiveness Unknowns

Leave a Reply

Your email address will not be published. Required fields are marked *