Call for Third-Party Evaluation Sites: Evaluation Study of the AI4HF Model in Heart Failure Management
Download here the application form:
AI4HF_ApplicationForm_OpenCall_3JUNE2024
1. The AI4HF project
Trustworthy Artificial Intelligence for Personalised Risk Assessment, or AI4HF (https://www.ai4hf.com) is a 4-year project that will co-design, develop and evaluate the first trustworthy artificial intelligence (AI) tool for personalising the care and management of patients with heart failure (HF). AI4HF adopts a personalised medicine approach, and the tools and care pathways are co-created with all relevant stakeholders.
AI4HF is a project funded by research funding from the European Union's Horizon Research and Innovation Programme Horizon Europe. The consortium has 16 partners from Europe, Peru, and Tanzania.
2. Background
HF remains a significant global health challenge. The challenge with HF is that it is associated with a significant variability in its aetiologies, manifestations and risks, as well as in its progression over time. There is a need for a personalised medicine approach to tailor the care models (i.e. medication, dietary changes, physical exercise, implantable devices, among others) to each patient’s risk profile and hence optimise the clinical outcomes.
The AI4HF consortium is now initiating a comprehensive, international study to evaluate the developed AI models in several, real-world healthcare settings. The AI4HF tools will be evaluated across different HF stages and clinical settings to assess all aspects of the tools. For the last stages we will evaluate applicability and durability of the AI4HF tool in an external setting.
3. Aim of the open call
We invite proposals from qualified institutions to serve as third-party evaluation sites, focusing on usability, explainability, acceptability, clinical integration, and clinical utility. The primary objective is to assess the AI models' applicability in clinical settings with a focus on end-user perspectives. External evaluators, who have not participated in the AI design and development, will provide insights into usability, explainability, and the overall acceptance of the technology in unfamiliar healthcare environments.
4. Workflow of the evaluation study
In the evaluation study, the AI4HF tools will be deployed in 8 clinical sites, including three external hospitals selected through this Open Call. Using a "virtual" trial model with a prospective approach, clinicians will evaluate the AI model's predictions against clinical information available in Electronic Health Records (EHRs). Clinicians will undergo 1-day training and hands-on sessions to assess 25 HF cases each.
In the next stage, the novel AI4HF monitoring platform will be evaluated for performance, utility, and acceptance, including functionalities like quality control and data drift detection, involving both data/IT managers and clinicians. Usability questionnaires will gauge user behaviours, human-AI interactions, and the utility of time-series visualizations.
5. Key criteria for third-party evaluation sites
5.1 Target population
In the evaluation study we will include individuals referred to hospitals for diagnosis or clinical management of HF with age ≥ 18 years from at least 2016 onwards. We will consider factors such as age (must have), sex (must have), and ethnicity (nice to have) as critical for a diverse patient population. Therefore, we request the prospective evaluation sites to provide aggregated meta-data on their HF population on the application form. This criterion enhances the external validity of the evaluation, ensuring that the AI model's performance is assessed across a broad spectrum of patients and clinical environments.
5.2 HF subgroups
The evaluation site should have the capability to identify and provide data on distinct HF subgroups, including patients with preserved, mildly reduced and reduced ejection fraction. This criterion is essential to assess the AI model's performance across different subtypes of HF, allowing for a more granular understanding of its effectiveness.
5.3 Data input and available predictors
Provide details on the available data input, including the range and depth of predictors, please see the application form for a list of the predictors in the AI4HF tool (all must have). This criterion aims to ensure that the evaluation sites have diverse and relevant predictors, allowing for a comprehensive assessment of the AI model's ability to leverage various data inputs for accurate predictions.
5.4 Access to end point information
Sites must have access to relevant clinical endpoints and outcomes data. This includes critical information such as hospitalizations (must have), mortality (must have), and other key adverse events (nice to have) essential for a comprehensive evaluation of the AI model's impact on patient outcomes, please see the application form.
5.5 Data standardisation
Implementation of HL7 Fast Healthcare Interoperability Resources (FHIR) or Observational Medical Outcomes Partnership (OMOP) mapping is a preference for standardized diverse healthcare data across sites. For the AI tool to be implemented, this criterion ensures that data from different sources are harmonized, allowing for consistent and interoperable representation, thus enhancing the reliability of the evaluation. If custom code systems or value sets are used in the source data, mapping to standardized ones such as ICD-10, LOINC, ATC etc. will be required.
5.6 Sample size
A minimum sample size of 500 patients (must have) with varying HF subgroups and risks is required to ensure a diverse population of patients for the ability to generalize findings across diverse populations. This criterion aims to provide sufficient diversity for a meaningful evaluation of the AI model's performance.
5.7 Diversity coverage
The AI4HF seeks to broaden its sites in terms of diversity to increase heterogeneity to challenge the model. This includes geographical, care setting and hospital information system diversity. Representation from diverse geographical regions and across care settings is vital to account for variations in healthcare practices and patient demographics. This criterion seeks to ensure that the AI model's applicability is evaluated across a range of healthcare ecosystems, promoting generalizability. Current clinical partners of the AI4HF consortium are based in: Czechia, Netherlands, Peru, Spain, Tanzania, and United Kingdom. There is a preference for clinical sites outside these regions (nice to have). The current clinical partners are all public tertiary referral/university hospitals. There is a preference for private and/or secondary care hospitals (nice to have).
5.8 Clinical end-users
The prospective evaluation site needs to include at least 6 healthcare professionals (must have) seeing HF patients and 4 GPs (must have). This inclusion of diverse clinical end-users ensures a holistic evaluation of the AI model's integration into different healthcare workflows. This criterion aims to capture varied perspectives and user experiences within clinical practice. Of these clinical end users, 50% need to be early-career (nice to have) and 50% need to have at least > 5-years’ experience (nice to have) with a sufficient gender-balanced selection of evaluators (must have). Prospective evaluation sites are encouraged to take part in stakeholder engagement meetings (nice to have).
5.9 IT and data management
Each selected site is required to have sufficient and demonstrable support by data and or IT specialists (must have). These professionals will play a crucial role in installing the monitoring platform offline locally, and recalibrating the AI tools as needed. This criterion ensures the seamless integration and technical support necessary for the successful implementation and evaluation of the AI model.
6. Financial support
Each selected hospital will be allocated a maximum of €75,000 (including overhead and VAT if applicable) to cover the costs of local ethical approval, data preparation, and evaluation activities. This financial support is intended to facilitate the seamless execution of the evaluation process. A budget should be provided in addition to the application form to specify the budget needed.
7. Submission guidelines
Interested institutions are requested to submit detailed application form to the Netherlands Heart Institute at bo.van.ieperen@heart-institute.nl addressing their ability to meet the key criteria outlined above. Proposals should include information on the institution's experience, data infrastructure, meta data on their HF population and commitment to the objectives of the evaluation.
8. Application timeline
Proposal Submission Deadline: 15 September 2024
Notification of Selection: November 2024
9. Study timeline
The suggested timeline for the evaluation study is as follows: the administrative process but also the preparation of the local data requires submission for ethical approval, local installation of the models and recruitment of healthcare professionals and should be finalised between June 2025 and June 2026. Between June 2026 and November 2026 training sessions should be scheduled and healthcare professionals will evaluate the selected HF cases using the locally installed models. Lastly, structured questionnaires for healthcare professionals to assess the behaviour towards the monitoring tool should be administered between November 2026 and May 2027.
10. Contact us via email for any questions
For inquiries and proposal submission, please contact Bo van Ieperen at bo.van.ieperen@heart-institute.nl. We appreciate your interest in advancing the applicability of AI models in heart failure management.
