Healthcare Risk Assessment Model Development
Budget: $25 – $50 USD
Need help with a complex question involving case mix adjustment for a logistic regression model in SAS.
Clinical problem:
Patients with heart failure are supposed to be treated with 3 or 4 evidence-based medications at particular target doses. This is a subject of clinical practice guidelines.
There is a large body of work showing that only about 3% are actually on all drugs at target doses, and providers rarely add or increase medications when it is indicated. This is termed “clinical inertia”. There are many complex reasons suggested for clinical inertia; there are provider-related factors, patient factors, and healthcare system-related factors. Examples: patient refuses or cannot tolerate a medication, a provider may fear side effects or may be distracted by other problems, different specialties treat differently etc.
However, the largest proportion of clinical inertia is unexplained, and this is a central issue- providers even when interviewed on the spot about their decision say “patient was stable” 75% of the time even though medication adjustments were indicated! There is a wide variation in how providers and patients behave that is not otherwise explained (i.e. there are simply “bad” patients and providers and “good” patients towns and providers, and this is the general experience in the field as well).
Statistically, there are multiple factors affecting treatment that have been identified, but the largest amount of variance is intrinsic to the provider and intrinsic to the patient: the provider treats a variety of different patients in the same way no matter their condition, and a given patient is treated the same way by multiple different providers. There are only a handful of papers looking at these patient and provider factors. As might be expected, all of them show poor model fit, both because some factors are difficult to assess from administrative data, but even more because providers just do what they do and patients just do what they do. Part of the solution in my view is to add a variable to capture some of this as an “inertia” variable, which is the observed rate at which the provider adjusts their patients or the patient gets their medications appropriately adjusted, over a long period of time. This will be termed the increase rate.
To do this it is necessary to see how this plays out across the entire healthcare system over a long period of time.
For this project patient factors and provider factors are studied. System-related factors covary so much with provider factors that we are combining the two. Both a priori and because overcorrection occurs when both patient and provider increase rates are present, there are two separate models- patient factors, adjusted from provider case mix, and provider factors, adjusted for patient case mix, hence the interest in using MLM for this.
In this discussion we will only talk about the provider model; the methods are the same for both.
Issues: The dataset was intended for a multilevel model, which I am pretty sure is impossible now. GLIMMIX fails with any more than a 1% sample of the dataset, even with a bare-bones number of variables (and the memory-sparing fast quadrature option on). It is so highly multi-membered and cross classified that I can’t see if an ecological fallacy is relevant or whether it could be ignored. In this respect, its complexity, this dataset differs from others published.
Prior to the current popularity of MLM, regulatory organizations that ranked providers of care for quality measure and reimbursement, usually hospitals, used case mix adjustment methodology for this, and some still do. Although the nature of this data differs in terms of complexity, this may be a suitable alternative.
However, since this is an encounter-level analysis I am not even sure that either method is suitable. However, I need to adjust the model for the expected likelihood of the patient being treated in a given encounter since patients differ greatly. Case mix adjustment as described in the article seems appropriate.
The dataset:
The dataset is clinic visits and other encounters for HF patients where medication changes were indicated. The data is encounter-level, with each encounter having a patient and a provider. The dependent variable is whether any indicated medication change was made. Overall, medication changes were made an average of 6% of the time. The independent variables are specific patient related factors and specific provider related factors, including the inertia factors for both. The dataset has 200k patients, 90k providers and 8M encounters over 13 years.
In the provider model, the R2 in a model containing only provider factors is about .15. Adding the provider score variable it is around .3, still not great. The same is true in the patient model. When both patient and provider vars are placed in a model, it is higher. This is consistent with other work in the field.
Implementing case mix (risk) adjustment:
Here is the SAS paper that has detailed info about implementing both risk adjustment and MLM.
Risk adjustment is discussed on pages 1 and 2, the math for “Adjusted binary outcome via logistic model” on page 3, and the SAS code on page 4.
MLM is discussed on page 6 “Model-based outcome via hierarchical model” and the math and code on page 7. The sections on continuous outcomes are not relevant. Ref for the paper (also attached): 4697-2020.pdf (sas.com)
The paper shows how to calculate O/E risk and Risk Adjusted Ratio. How to use those in a LR model is one of my questions.
Expected risk comes from a model using GENMOD, using patient factors to calculate an expected risk (0-1, mean runs about .8) for that encounter.
Observed risk is the increase rate for that provider (mean is .06).
When I have O/E risk or RAR from the model that includes patient variables, I placed them in a final model that contains the provider variables and the patient expected risk or O/E as the only patient variable.
I have questions about that and I don’t completely understand how to implement it or if I am calculating the scores correctly.
Clinical problem:
Patients with heart failure are supposed to be treated with 3 or 4 evidence-based medications at particular target doses. This is a subject of clinical practice guidelines.
There is a large body of work showing that only about 3% are actually on all drugs at target doses, and providers rarely add or increase medications when it is indicated. This is termed “clinical inertia”. There are many complex reasons suggested for clinical inertia; there are provider-related factors, patient factors, and healthcare system-related factors. Examples: patient refuses or cannot tolerate a medication, a provider may fear side effects or may be distracted by other problems, different specialties treat differently etc.
However, the largest proportion of clinical inertia is unexplained, and this is a central issue- providers even when interviewed on the spot about their decision say “patient was stable” 75% of the time even though medication adjustments were indicated! There is a wide variation in how providers and patients behave that is not otherwise explained (i.e. there are simply “bad” patients and providers and “good” patients towns and providers, and this is the general experience in the field as well).
Statistically, there are multiple factors affecting treatment that have been identified, but the largest amount of variance is intrinsic to the provider and intrinsic to the patient: the provider treats a variety of different patients in the same way no matter their condition, and a given patient is treated the same way by multiple different providers. There are only a handful of papers looking at these patient and provider factors. As might be expected, all of them show poor model fit, both because some factors are difficult to assess from administrative data, but even more because providers just do what they do and patients just do what they do. Part of the solution in my view is to add a variable to capture some of this as an “inertia” variable, which is the observed rate at which the provider adjusts their patients or the patient gets their medications appropriately adjusted, over a long period of time. This will be termed the increase rate.
To do this it is necessary to see how this plays out across the entire healthcare system over a long period of time.
For this project patient factors and provider factors are studied. System-related factors covary so much with provider factors that we are combining the two. Both a priori and because overcorrection occurs when both patient and provider increase rates are present, there are two separate models- patient factors, adjusted from provider case mix, and provider factors, adjusted for patient case mix, hence the interest in using MLM for this.
In this discussion we will only talk about the provider model; the methods are the same for both.
Issues: The dataset was intended for a multilevel model, which I am pretty sure is impossible now. GLIMMIX fails with any more than a 1% sample of the dataset, even with a bare-bones number of variables (and the memory-sparing fast quadrature option on). It is so highly multi-membered and cross classified that I can’t see if an ecological fallacy is relevant or whether it could be ignored. In this respect, its complexity, this dataset differs from others published.
Prior to the current popularity of MLM, regulatory organizations that ranked providers of care for quality measure and reimbursement, usually hospitals, used case mix adjustment methodology for this, and some still do. Although the nature of this data differs in terms of complexity, this may be a suitable alternative.
However, since this is an encounter-level analysis I am not even sure that either method is suitable. However, I need to adjust the model for the expected likelihood of the patient being treated in a given encounter since patients differ greatly. Case mix adjustment as described in the article seems appropriate.
The dataset:
The dataset is clinic visits and other encounters for HF patients where medication changes were indicated. The data is encounter-level, with each encounter having a patient and a provider. The dependent variable is whether any indicated medication change was made. Overall, medication changes were made an average of 6% of the time. The independent variables are specific patient related factors and specific provider related factors, including the inertia factors for both. The dataset has 200k patients, 90k providers and 8M encounters over 13 years.
In the provider model, the R2 in a model containing only provider factors is about .15. Adding the provider score variable it is around .3, still not great. The same is true in the patient model. When both patient and provider vars are placed in a model, it is higher. This is consistent with other work in the field.
Implementing case mix (risk) adjustment:
Here is the SAS paper that has detailed info about implementing both risk adjustment and MLM.
Risk adjustment is discussed on pages 1 and 2, the math for “Adjusted binary outcome via logistic model” on page 3, and the SAS code on page 4.
MLM is discussed on page 6 “Model-based outcome via hierarchical model” and the math and code on page 7. The sections on continuous outcomes are not relevant. Ref for the paper (also attached): 4697-2020.pdf (sas.com)
The paper shows how to calculate O/E risk and Risk Adjusted Ratio. How to use those in a LR model is one of my questions.
Expected risk comes from a model using GENMOD, using patient factors to calculate an expected risk (0-1, mean runs about .8) for that encounter.
Observed risk is the increase rate for that provider (mean is .06).
When I have O/E risk or RAR from the model that includes patient variables, I placed them in a final model that contains the provider variables and the patient expected risk or O/E as the only patient variable.
I have questions about that and I don’t completely understand how to implement it or if I am calculating the scores correctly.