At a Pentagon press conference on February 12, 2002, when asked about the probability that Iraq had obtained weapons of mass destruction, Donald Rumsfeld said the following:
“There are known knowns. There are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know”
At the time, pundits mocked him for this statement and many people viewed it as handwaving nonsense meant to cover over reality. But with time, this statement started to gain respect. In actuality, he was trying to represent the important grey area in risk assessment that we have all become quite familiar with. Some people say, “I don’t know what I don’t know” and others call it second-order ignorance.
VitreosHealth has been a pioneer in the healthcare predictive risk analytics space. Our mission has remained the same from day one of the company – Predict, Intervene & Prevent avoidable poor outcomes. Based on our 8 years of experience implementing AI-based predictive models in healthcare, we have also come across numerous unknown unknowns. By sharing these, I am hoping that I can assist providers and payers avoid these mistakes and minimize wastage and inefficiencies.
These unknown unknowns of predictive analytics in healthcare fall into three categories:
- Efficacy Traps: doing things in a way that ensures a reduction in negative outcomes (read more about those here)
- Efficiency Traps: doing things in a way that minimizes resource wastage (read more about those here)
- Effectiveness Traps: doing things accurately
Care Management Effectiveness Traps and the Importance of Predictive Accuracy
After we figured out how to be more efficient, the next step was to maximize care management resource effectiveness. In predictive modeling, effectiveness is directly tied to the accuracy of our models. We needed to be able to assimilate all this information and come up with one risk score that predicted poor outcomes – emergent events, high costs, etc. Exhibit 2 shows the prioritized ranks for the 11 members from Exhibit 1 using an Average Clinical Risk Score, an HCC Risk Score, and an AI-based Predictive Model for Avoidable Outcomes. The customized AI-based predictive model used EMR data, external demographics, labs, pharmacy data inputs that include all the disease specific scores, non-clinical risk scores, gaps-in-care along with other social determinants data. Each of the three risk scores are calculated for each patient and then each patient is ranked for prioritized outreach. A high-risk score will result in a top rank (indicated by a low number) since those patients need immediate attention from a care manager.
Let me illustrate the unknown unknowns in interpreting these different scores and linking it to the predictive risk of poor outcomes.
- Use Case 1: The Chronic Clinical Risk score and the HCC Risk score track the Critical and Healthy members well (Members A, B, C, K) as they are based only on presence of chronic conditions
- Use Case 2: HCC fails to capture the latest State-of-Health of a chronic condition unlike EMR-based Clinical Risk scores using latest vitals and lab results (Member E). Member E has a high clinical risk score based on vitals collected from EMR data, but HCC does not factor in latest vitals and lab results to determine the state of a specific disease at any point of time.
- Use Case 3: The AI-based model captures all clinical, non-clinical and utilization patterns and hence identifies opportunities missed by EMR-based clinical risk scores and HCC Utilization risk scores (Members F, H, G). Hence AI-based models have much higher accuracies in identifying rising-risk member cohorts.
VitreosHealth’s AI-driven customized models are delivering accuracies that are 2-3 times better than those of CMS HCC, CDPSRx, etc. These accuracies translate to accomplishing the same results in outcomes with half the number of care managers. See Exhibit 3 to illustrate the ROI in care management resources to achieve a targeted savings through proactive member engagement.
For a typical Medicare Advantage Plan with 10,000 members, preventing 40% of ER events results in $6 million in cost savings. You need a smaller care management team to achieve that $6 million if you are using higher accuracy AI predictive models (since you only need to touch around 10% of your population) as compared to HCC models (since you need to touch about 40% of your population). In the above Medicare Advantage example, you need a care management team with 17 members if you are using CMS HCC risk stratification compared to only 8 care managers if you are using Vitreos AI-based risk stratification predictive model.
After spending 2-3 years and millions of dollars in predictive analytics, you as a leader is thinking, “What have I gotten myself into?”. This is the point at which most risk-based organizational leadership teams realize the need for an analytics partner that is experienced and well-versed in helping their team members make strategic data-driven decisions at every level early in the process.
Partner with the right experts who can transform your unknown unknowns to their known knowns. Leverage the collective learnings of your predictive analytics partner’s multiple customers to avoid the emerging technology pitfalls. Pick a partner whose core competency is Healthcare Predictive Analytics.
Did you miss parts 1 and 2 of this series? Read all the parts here.