We're building a better ClinicalTrials.gov. Check it out and tell us what you think!
Working…
ClinicalTrials.gov
ClinicalTrials.gov Menu

Encouraging Flu Vaccination Among High-Risk Patients Identified by ML

The safety and scientific validity of this study is the responsibility of the study sponsor and investigators. Listing a study does not mean it has been evaluated by the U.S. Federal Government. Read our disclaimer for details.
 
ClinicalTrials.gov Identifier: NCT04323137
Recruitment Status : Completed
First Posted : March 26, 2020
Last Update Posted : November 23, 2021
Sponsor:
Information provided by (Responsible Party):
Christopher F Chabris, PhD, Geisinger Clinic

Brief Summary:

The purpose of the current study is to test different interventions to determine the most effective way to promote flu vaccine uptake in a high-risk population identified by an "artificial intelligence" (AI) or machine learning (ML) algorithm. The specific aims are:

  1. Evaluate the effect on flu vaccination rates of informing health-system patients who are identified by an ML analysis of EHR data to be at high risk for flu complications that they are at high risk with either (a) no additional explanation, (b) an explanation that this determination comes from an analysis of their medical records, and (c) the additional explanation that an AI or ML algorithm made this determination.
  2. Evaluate the effects of the same three interventions on diagnoses of flu in the same patients.

Condition or disease Intervention/treatment Phase
Influenza Vaccination Health Promotion Health Behavior Risk Reduction Behavioral: Risk reduction Behavioral: Medical records-based recommendation Behavioral: Algorithm-based recommendation Not Applicable

Show Show detailed description

Layout table for study information
Study Type : Interventional  (Clinical Trial)
Actual Enrollment : 46602 participants
Allocation: Randomized
Intervention Model: Parallel Assignment
Intervention Model Description:

Patients from the high-risk sample will be randomly assigned to one of 4 groups:

  1. Control: group that receives no additional pro-vaccination intervention beyond Geisinger's normal efforts. Although some patients are currently targeted for flu vaccination messages due to a non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted
  2. High Risk Only: group that receives messages telling them they have been identified to be at high risk for flu complications without specifying how/why Geisinger believes this to be the case
  3. High Risk Based on Medical Records: group that receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records
  4. High Risk Based on Algorithm: group that receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by AI/ML
Masking: Double (Participant, Care Provider)
Masking Description: Participants (i.e., patients) will not be informed specifically of their assignment to different arms throughout the study. Providers who prescribe vaccination and diagnose conditions will not be randomized to study arms or informed of patient assignment.
Primary Purpose: Prevention
Official Title: Encouraging Flu Vaccination Among High-Risk Patients Identified by a Machine-Learning Model of Flu Complication Risk
Actual Study Start Date : September 21, 2020
Actual Primary Completion Date : May 31, 2021
Actual Study Completion Date : September 21, 2021

Resource links provided by the National Library of Medicine

MedlinePlus related topics: Flu Flu Shot Vaccines

Arm Intervention/treatment
No Intervention: Control
This group receives no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.
Experimental: High risk only
This group receives messages telling them they have been identified to be at high risk for flu complications without specifying how or why the health system believes this to be the case.
Behavioral: Risk reduction
Mailed letter, SMS, and/or patient portal message

Experimental: High risk based on medical records
This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records.
Behavioral: Risk reduction
Mailed letter, SMS, and/or patient portal message

Behavioral: Medical records-based recommendation
Mailed letter, SMS, and/or patient portal message
Other Name: Credibility

Experimental: High risk based on algorithm
This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by an AI/ML system.
Behavioral: Risk reduction
Mailed letter, SMS, and/or patient portal message

Behavioral: Medical records-based recommendation
Mailed letter, SMS, and/or patient portal message
Other Name: Credibility

Behavioral: Algorithm-based recommendation
Mailed letter, SMS, and/or patient portal message
Other Name: Credibility




Primary Outcome Measures :
  1. Flu vaccination rate [ Time Frame: Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration ]
    Patient received a flu vaccination

  2. High confidence flu diagnosis rate [ Time Frame: Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration ]
    Patient received a flu diagnosis via a positive PCR/antigen/molecular test


Secondary Outcome Measures :
  1. "Likely flu" diagnosis rate [ Time Frame: Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration ]
    Patient received a diagnosis that was likely flu, as assessed via ICD codes or Tamiflu administration or positive PCR/antigen/molecular test. Note that this outcome is a superset of the "high confidence flu diagnosis rate" outcome.

  2. Flu complications rate [ Time Frame: Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration ]
    Patient was diagnosed with flu-related complications

  3. Change in ER visits from pre- to post-intervention [ Time Frame: Within 12 months pre-intervention (Time 1) and within 12 months post-intervention (Time 2) ]
    Number of patient visits to the ER, examining relative rate of visits across Time 1 and 2

  4. Change in hospitalizations from pre- to post-intervention [ Time Frame: Within 12 months pre-intervention (Time 1) and within 12 months post-intervention (Time 2) ]
    Number of patient hospital visits, examining relative rate of visits across Time 1 and 2

  5. Flu vaccination among fellow household members [ Time Frame: Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration ]
    Non-targeted fellow household members of targeted patients received a flu vaccination

  6. High confidence flu diagnosis among fellow household members [ Time Frame: Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration ]
    Non-targeted fellow household members of targeted patients received a flu diagnosis (via a positive PCR/antigen/molecular test)

  7. "Likely flu" diagnosis among fellow household members [ Time Frame: Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration ]
    Non-targeted fellow household members of targeted patients received a diagnosis that was likely flu (as assessed via ICD codes or Tamiflu administration or positive PCR/antigen/molecular test)

  8. Flu complications among fellow household members [ Time Frame: Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration ]
    Non-targeted fellow household members of targeted patients were diagnosed with flu-related complications

  9. Flu vaccination among those at sub-threshold risk [ Time Frame: Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration ]
    Non-targeted sub-threshold risk patients received a flu vaccination

  10. High confidence flu diagnosis among those at sub-threshold risk [ Time Frame: Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration ]
    Non-targeted sub-threshold risk patients received a flu diagnosis (via a positive PCR/antigen/molecular test)

  11. "Likely flu" diagnosis among those at sub-threshold risk [ Time Frame: Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration ]
    Non-targeted sub-threshold risk patients received a diagnosis that was likely flu (as assessed via ICD codes or Tamiflu administration or positive PCR/antigen/molecular test)

  12. Flu complications among those at sub-threshold risk [ Time Frame: Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration ]
    Non-targeted sub-threshold risk patients were diagnosed with flu-related complications



Information from the National Library of Medicine

Choosing to participate in a study is an important personal decision. Talk with your doctor and family members or friends about deciding to join a study. To learn more about this study, you or your doctor may contact the study research staff using the contacts provided below. For general information, Learn About Clinical Studies.


Layout table for eligibility information
Ages Eligible for Study:   17 Years and older   (Child, Adult, Older Adult)
Sexes Eligible for Study:   All
Accepts Healthy Volunteers:   No
Criteria

Inclusion Criteria:

  • Current Geisinger patient at the time of study
  • Falls in the top 10% of patients at highest risk, as identified by the flu-complication risk scores of Medial's machine learning algorithm (which operates on coded EHR data)
  • May limit inclusion to patients that are under Geisinger primary care, depending on algorithm performance of patients who have non-Geisinger PCPs

Exclusion Criteria:

  • Has contraindications for flu vaccination
  • Has opted out of receiving communications from Geisinger via all of the modalities being tested

Information from the National Library of Medicine

To learn more about this study, you or your doctor may contact the study research staff using the contact information provided by the sponsor.

Please refer to this study by its ClinicalTrials.gov identifier (NCT number): NCT04323137


Locations
Layout table for location information
United States, Pennsylvania
Geisinger
Danville, Pennsylvania, United States, 17822
Sponsors and Collaborators
Geisinger Clinic
Investigators
Layout table for investigator information
Principal Investigator: Christopher Chabris, PhD Geisinger Clinic
  Study Documents (Full-Text)

Documents provided by Christopher F Chabris, PhD, Geisinger Clinic:
Study Protocol  [PDF] September 13, 2021
Statistical Analysis Plan  [PDF] January 8, 2021

Publications:
Logg, J.M., Minson, J.A., & Moore, D.A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90-103. https://doi.org/10.1016/j.obhdp.2018.12.005
Centers for Disease Control and Prevention. (2020). Disease Burden of Influenza. https://www.cdc.gov/flu/ about/burden/index.html (Jan 10).
Centers for Disease Control and Prevention. (2019a). Who Needs a Flu Vaccine and When. https://www.cdc.gov/flu/prevent/vaccinations.htm (Oct 11).
Centers for Disease Control and Prevention. (2019b). Flu Vaccination Coverage, United States, 2018-19 Influenza Season. https://www.cdc .gov/flu/fluvaxview/coverage-1819estimates.htm

Layout table for additonal information
Responsible Party: Christopher F Chabris, PhD, Faculty Co-Director, Behavioral Insights Team, Geisinger Clinic
ClinicalTrials.gov Identifier: NCT04323137    
Other Study ID Numbers: 2020-0290
First Posted: March 26, 2020    Key Record Dates
Last Update Posted: November 23, 2021
Last Verified: November 2021
Individual Participant Data (IPD) Sharing Statement:
Plan to Share IPD: Yes
Plan Description: Data with no personally identifiable information will be made available to other researchers on the Open Science Framework for transparency. This will include the essential data and code needed to replicate the analysis that yielded reported findings.
Supporting Materials: Study Protocol
Time Frame: The data will become available after publication of study results in a scientific journal and will be available as long as the Open Science Framework hosts the data.
Access Criteria: The data on the Open Science Framework will be open to anyone requesting that information.
URL: http://osf.io

Layout table for additional information
Studies a U.S. FDA-regulated Drug Product: No
Studies a U.S. FDA-regulated Device Product: No
Keywords provided by Christopher F Chabris, PhD, Geisinger Clinic:
Flu Vaccine
Choice Architecture
Machine Learning
Perceived Credibility