At a Pentagon press conference on February 12, 2002, when asked about the probability that Iraq had obtained weapons of mass destruction, Donald Rumsfeld said the following:

“There are known knowns. There are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know”

At the time, pundits mocked him for this statement and many people viewed it as handwaving nonsense meant to cover over reality. But with time, this statement started to gain respect. In actuality, he was trying to represent the important grey area in risk assessment that we have all become quite familiar with. Some people say, “I don’t know what I don’t know” and others call it second-order ignorance.  

VitreosHealth has been a pioneer in the healthcare predictive risk analytics space. Our mission has remained the same from day one of the company – Predict, Intervene & Prevent avoidable poor outcomes. Based on our 8 years of experience implementing AI-based predictive models in healthcare, we have also come across numerous unknown unknowns. By sharing these, I am hoping that I can assist providers and payers avoid these mistakes and minimize wastage and inefficiencies.

These unknown unknowns of predictive analytics in healthcare fall into three categories:

  1. Efficacy Traps: doing things in a way that ensures a reduction in negative outcomes
  2. Efficiency Traps: doing things in a way that minimizes resource wastage
  3. Effectiveness Traps: doing things accurately

Efficacy Traps

This is the trap most people in our industry fall for. The purpose of healthcare predictive analytics in population health is to identify rising risk members and design appropriate programs to intervene and avoid poor health outcomes (like ER visits, admissions, and procedures). How can you ensure positive movement in your bottom-line?

Too Much Focus on Acute Care Utilization

Population health vendors and analytics teams are focused too narrowly on models that simply predict acute care utilization. When I say acute care utilization, I am referring to readmission, sepsis, length of stay, etc. Is the intent of value-based care and population health to avoid acute care events? If that is the case, why not focus on making sure we minimize the members showing up in the acute care facilities? You can do that by keeping populations healthy by zeroing in on proactive and preventive care. The big mistake many leadership teams make is that they get their priorities wrong; they need to look at the value chain as a whole, rather than getting trapped focusing on healthcare costs as silos based on settings of care. Let me share an example from the retail perspective. If you see lot of damaged product on the shelf on a Walmart store, is it because of poor retail operations or is it really because of poor product packaging design or poor demand planning? If acute care settings are high cost centers, let’s make sure nobody needs to use these facilities by keeping populations healthy. Look at your population to avoid falling into an efficacy trap.

Funneling Much of Your Resources into Reactive Programs

Case management, transitions-in-care, and readmissions are important; but they should not be your sole focus. You must invest in proactive programs just as heavily as you do in reactive. Proactive and preventive programs are an after-thought because the impacts are longer term and difficult to measure. Avoidance is a difficult concept to comprehend and most current dashboards and analytics fail to measure these. But, it is not impossible. The new VitreosHealth Performance Insights-as-a-Service (IaaS) application is built to measure the efficacy of both proactive and reactive programs and highlight opportunities to make the transition to proactive care programs in a way that is intuitive for C-level leadership.

Finding the Right Type of Care Managers for Each Type of Program

The skill set required for care managers in proactive care is very different from that required for reactive care. It is easy to call a member who is sick and get them to speak to you and listen to your care plan instructions. What about reaching out to members who think they are healthy, but your models show that they are ticking time bombs? Most care managers are not taught the cold call skills necessary for these situations. Care management requires alignment of team members with the most appropriate skills.

Efficiency Traps in Predictive Analytics Applications

In 2013, VitreosHealth developed disease-specific scores based on evidence-based clinical studies like the Framingham model (CHF, CHD) and LACE, and readmission scores predicting the risk of high cost events due to those underlying conditions. These scores were calculated using the latest encounter vitals, lab results and health risk assessments. We embedded these algorithms into our customer’s EMRs and Care Management work flows, and these scores were re-calculated within the EMR or Care Management application as new data was entered. We also developed non-clinical risk scores independently, leveraging social determinants data at a member level, and exhibited those along with the disease specific scores as shown in Exhibit 1. But this approach failed to get user adoption, and we were forced to go back to the drawing board. We were left wondering what happened.

Exhibit 1: Examples of risk scores that our care managers had access to in 2013. Red indicates high risk, yellow indicates medium risk, and green indicates low risk.

During our discovery process, we ran into the following challenges.

Predictive Analytics Can Not be a Black Box

Care managers did not understand what went into a risk score. By trade, care managers are data-driven and used to assessing a patient’s health based on many clinical factors. When they were given a single clinical risk score, they were understandably suspicious. Initially, we made no effort to explain to them what kinds of data went into compiling that score or why one diabetic patient had a clinical risk score that was 35 points higher than another diabetic patient. They asked us, “How are co-morbidities represented in the overall clinical risk score and a disease-specific risk score?” and “How do each of these clinical and non-clinical risk scores interact?”. The first way you can combat inefficiency is transparency and education for the end users of your predictive models. They need to be able to trust and understand the output of your models before they will be willing to act on it.

Eliminate the “How do I Move Forward?” Conundrum

When care managers started understanding our risk scores and prioritizing patients for outreach based on the results of our models, they asked us what the call to action should be. They would spend a long time looking at one patient’s data but were unsure what action they should be recommending for that patient. That gave us the idea that we should use our analytics capabilities not just to prioritize patients for outreach, but also to prioritize which risk factors need to be addressed and which gaps in care should be closed for each patient. Give the users of your predictive model a prescriptive action plan for each high-risk member.

Give Your User the Whole Story

Care managers needed to be able to translate all the data and risk scores into a coherent story that would convince a patient to make changes in their lifestyle. This is difficult without the added pressure of a time crunch. Our care managers asked for more access to information like where their patients lived, their income levels, and proximity to their PCPs. We had to display this socio-economic data alongside clinical data in a way that highlighted what was most important because care managers only have a few minutes between calls. Ensure your predictive models give the whole story in a digestible, visual way. The Vitreos solution to these problems was to develop the member engagement VitreosHealth Insights application which told the complete risk story for each high-risk member.  

Garbage-in, Garbage-out

We found that there were countless errors made during data entry. What happens if a Physician inputs the wrong information? Does the EMR systems alert the user if the body mass index (BMI) data is out of range? Does the EMR check steep increases or decreases in BMI for each member from one appointment to the next? We needed to be able to perform a thorough data validation and quality control step before we run the predictive algorithms on this data.

Importance of External Data Sources

What about external data sources capturing the clinical details about the member external to EMR? If the patient has gone out-of-network for a lab test, those results were not entered in the EMR data warehouse. Our care managers needed this data and other external data sources like mental health reports, HRAs, and social determinants. We needed to think through how the risk score is factoring in the data from sources not captured in the native EMR application or data warehouse.

Care Management Effectiveness Traps and the Importance of Predictive Accuracy

After we figured out how to be more efficient, the next step was to maximize care management resource effectiveness. In predictive modeling, effectiveness is directly tied to the accuracy of our models. We needed to be able to assimilate all this information and come up with one risk score that predicted poor outcomes – emergent events, high costs, etc. Exhibit 2 shows the prioritized ranks for the 11 members from Exhibit 1 using an Average Clinical Risk Score, an HCC Risk Score, and an AI-based Predictive Model for Avoidable Outcomes. The customized AI-based predictive model used EMR data, external demographics, labs, pharmacy data inputs that include all the disease specific scores, non-clinical risk scores, gaps-in-care along with other social determinants data. Each of the three risk scores are calculated for each patient and then each patient is ranked for prioritized outreach. A high-risk score will result in a top rank (indicated by a low number) since those patients need immediate attention from a care manager.

Exhibit 2: Ranks of the 11 patients from Exhibit 1. A low number in yellow indicates a top clinical risk rank and a high clinical risk. A low number in grey indicates a top HCC risk rank and a high HCC risk. A low number in black indicates a top AI-based predictive model rank and a high risk for avoidable emergent events.

Let me illustrate the unknown unknowns in interpreting these different scores and linking it to the predictive risk of poor outcomes.

  • Use Case 1: The Chronic Clinical Risk score and the HCC Risk score track the Critical and Healthy members well (Members A, B, C, K) as they are based only on presence of chronic conditions
  • Use Case 2: HCC fails to capture the latest State-of-Health of a chronic condition unlike EMR-based Clinical Risk scores using latest vitals and lab results (Member E). Member E has a high clinical risk score based on vitals collected from EMR data, but HCC does not factor in latest vitals and lab results to determine the state of a specific disease at any point of time.
  • Use Case 3: The AI-based model captures all clinical, non-clinical and utilization patterns and hence identifies opportunities missed by EMR-based clinical risk scores and HCC Utilization risk scores (Members F, H, G). Hence AI-based models have much higher accuracies in identifying rising-risk member cohorts.

 VitreosHealth’s AI-driven customized models are delivering accuracies that are 2-3 times better than those of CMS HCC, CDPSRx, etc. These accuracies translate to accomplishing the same results in outcomes with half the number of care managers. See Exhibit 3 to illustrate the ROI in care management resources to achieve a targeted savings through proactive member engagement.

Exhibit 3: Care management productivity is directly tied to predictive accuracy. In this graph, I am trying to illustrate how four different models with different predictive accuracies can give you different results. Suppose you only have the resources to target 20% of your population for proactive outreach. You would want to choose a model that helps you pick the top 20% of members that are most at risk for avoidable emergent visits. If you choose which members to focus outreach efforts on by using CMS HCC, you will most likely only prevent 15% of avoidable ER events, while if you used Vitreos Regression models, you could prevent around 38% of emergent events. Our new AI models will give you the ability to prevent 65% of ER events. A perfect model (powered by whatever deity you choose to believe in) will probably help you prevent 85% of emergent visits (in our experience, the top 20% of members result in 85% of avoidable cost).

For a typical Medicare Advantage Plan with 10,000 members, preventing 40% of ER events results in $6 million in cost savings. You need a smaller care management team to achieve that $6 million if you are using higher accuracy AI predictive models (since you only need to touch around 10% of your population) as compared to HCC models (since you need to touch about 40% of your population).  In the above Medicare Advantage example, you need a care management team with 17 members if you are using CMS HCC risk stratification compared to only 8 care managers if you are using Vitreos AI-based risk stratification predictive model.

After spending 2-3 years and millions of dollars in predictive analytics, you as a leader is thinking, “What have I gotten myself into?”. This is the point at which most risk-based organizational leadership teams realize the need for an analytics partner that is experienced and well-versed in helping their team members make strategic data-driven decisions at every level early in the process.

Partner with the right experts who can transform your unknown unknowns to their known knowns. Leverage the collective learnings of your predictive analytics partner’s multiple customers to avoid the emerging technology pitfalls. Pick a partner whose core competency is Healthcare Predictive Analytics.

Ready to learn more? Fill out your info and we will get back to you:

The Unknown Unknowns of AI-based Risk Stratification in Healthcare

Leave a Reply

Your email address will not be published. Required fields are marked *