You are here

Article Text

Article menu
Arrhythmias and sudden death
Original research
External validation and updating of prediction models of bleeding risk in patients with cancer receiving anticoagulants
  1. EM Trinks-Roerdink1,
  2. GJ Geersing1,
  3. MEW Hemels2,3,
  4. IC van Gelder4,
  5. FA Klok5,
  6. M van Smeden6,
  7. FH Rutten1 and
  8. S van Doorn1
  1. 1Department of General Practice & Nursing Science, Julius Centre for Health Sciences and Primary Care, UMCU, Utrecht, The Netherlands
  2. 2Department of Cardiology, Rijnstate Hospital, Arnhem, The Netherlands
  3. 3Department of Cardiology, Radboudumc, Nijmegen, The Netherlands
  4. 4Department of Cardiology, UMCG, Groningen, The Netherlands
  5. 5Department of Medicine - Thrombosis and Haemostasis, LUMC, Leiden, The Netherlands
  6. 6Department of Epidemiology & Health Economics, Julius Centre for Health Sciences and Primary Care, UMCU, Utrecht, The Netherlands
  1. Correspondence to EM Trinks-Roerdink; e.m.roerdink{at}umcutrecht.nl

Abstract

Objective Patients with cancer are at increased bleeding risk, and anticoagulants increase this risk even more. Yet, validated bleeding risk models for prediction of bleeding risk in patients with cancer are lacking. The aim of this study is to predict bleeding risk in anticoagulated patients with cancer.

Methods We performed a study using the routine healthcare database of the Julius General Practitioners’ Network. Five bleeding risk models were selected for external validation. Patients with a new cancer episode during anticoagulant treatment or those initiating anticoagulation during active cancer were included. The outcome was the composite of major bleeding and clinically relevant non-major (CRNM) bleeding. Next, we internally validated an updated bleeding risk model accounting for the competing risk of death.

Results The validation cohort consisted of 1304 patients with cancer, mean age 74.0±10.9 years, 52.2% males. In total 215 (16.5%) patients developed a first major or CRNM bleeding during a mean follow-up of 1.5 years (incidence rate; 11.0 per 100 person-years (95% CI 9.6 to 12.5)). The c-statistics of all selected bleeding risk models were low, around 0.56. Internal validation of an updated model accounting for death as competing risk showed a slightly improved c-statistic of 0.61 (95% CI 0.54 to 0.70). On updating, only age and a history of bleeding appeared to contribute to the prediction of bleeding risk.

Conclusions Existing bleeding risk models cannot accurately differentiate bleeding risk between patients. Future studies may use our updated model as a starting point for further development of bleeding risk models in patients with cancer.

  • atrial fibrillation
  • epidemiology
  • pharmacology
  • pharmacology, clinical

Data availability statement

Data may be obtained from a third party and are not publicly available.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/openhrt-2023-002273

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    WHAT IS ALREADY KNOWN ON THIS TOPIC

    • Bleeding risk is increased in patients with cancer, yet validated models for the prediction of bleeding risk in patients with cancer are lacking.

    WHAT THIS STUDY ADDS

    • External validation of existing bleeding risk models was performed in routine primary healthcare data. These models cannot accurately predict bleeding in patients with cancer. On updating, only age and history of bleeding contributed to the prediction of bleeding risk.

    HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

    • Only age and a history of bleeding were shown to have incremental predictive value and could be considered in future prediction models for bleeding in patients with cancer at risk for thromboembolic complications.

    Introduction

    Patients with cancer using anticoagulants for treatment of venous thromboembolism (VTE) or prevention of stroke in atrial fibrillation (AF) are at increased risk of thrombosis due to the hypercoagulability state of cancer itself as well as caused by its treatment.1 However, these patients are also at an increased risk of bleeding.2 For example, patients with cancer may need to undergo chemotherapy which often causes thrombocytopaenia, drug interactions, periods of reduced renal function and suboptimal nutritional status, which all may increase bleeding risk. Invasive procedures such as surgery, intravenous access lines and biopsies also carry an increased bleeding risk, notably in anticoagulated patients.2 Finally, the malignant tumour itself may also cause spontaneous bleeding, for example, gastrointestinal, lung or brain tumours.3–5 Hence, even though often clearly indicated, anticoagulant therapy in patients with cancer with VTE or AF is a complicated clinical endeavour, notably during certain periods of the disease, warranting constantly a vigorous balance between bleeding and thrombosis risk.

    Therefore, it is important to identify patients with cancer with an increased risk of bleeding. While on average patients may have a net benefit of anticoagulation, for individualised treatment decisions a bleeding risk assessment can aid in shared decision-making.6 A bleeding risk model could be used to monitor the bleeding risk during anticoagulant treatment and to identify individual moments of increased bleeding risk where monitoring or perhaps temporarily reduced dosing may be warranted. Moreover, in patients with advanced cancer and a limited life expectancy, the bleeding risk can outweigh the thrombosis risk reduction, and stopping anticoagulant treatment may perhaps be the preferred option. Knowledge of what then defines and differentiates bleeding risk between patients is paramount.

    To assess the risk of bleeding, various risk prediction models have been developed for patients with VTE or AF. These bleeding risk models include various common predictors, for example, age, comorbidities and the concurrent use of medication such as antiplatelet drugs. Some models also include cancer as a predictor to account for the increased bleeding risk in these patients. However, none of the published bleeding risk models has ever been evaluated in a cohort of patients with cancer using healthcare data from primary care including clinically relevant non-major (CRNM) bleeding complications occurring outside the hospital. Therefore, we aimed to externally validate commonly used bleeding risk models in patients with cancer in the primary care setting. We will update and internally validate the best-performing bleeding risk model for use in this vulnerable population.

    Methods

    Selection and appraisal of existing bleeding risk models

    A priori we selected five existing bleeding risk models based on their common use and the availability of predictors in routine primary healthcare. We selected models developed to predict bleeding in either AF and/or VTE patients receiving anticoagulants. The selected models were as follows: HAS-BLED, ATRIA, ORBIT, VTE-bleed and the AF-bleed.7–11 The VTE-bleed was developed specifically for the prediction of bleeding in VTE patients, the HAS-BLED, ORBIT and ATRIA for the prediction of bleeding in AF patients. The AF-bleed is an adaptation of the VTE-bleed for the prediction of bleeding in AF patients. The risk of bias was assessed using the PROBAST tool.12

    Source of the external validation cohort

    To validate each bleeding risk model, we used a retrospective, observational cohort from the Julius General Practitioners’ Network (JGPN). The JGPN database contains longitudinal routine healthcare data of more than 450 000 individuals from the 1990s of the last century onward, deidentified extracted from the electronic medical files of general practitioners (GPs) in the vicinity of Utrecht in the Netherlands.13 14 People included in the JGPN database represent the Dutch population, except for nursing home residents who are not represented in this database.

    Inclusion and exclusion criteria for the external validation cohort

    From May 2000 to January 2022, all patients with a new cancer episode during anticoagulant treatment either with a vitamin K antagonist (VKA), a direct oral anticoagulant (DOAC), low-molecular-weight heparin (LMWH) or heparin indicated for either the treatment of VTE or stroke prevention in AF were included. We only selected patients with a new cancer diagnosis to limit the validation cohort to patients with active cancer. Patients with all types of cancer were eligible for inclusion, except for patients with basal cell carcinoma of the skin. Both patients already using anticoagulants, that is, before the cancer diagnosis (most often for AF), and patients who initiated anticoagulant treatment no longer than 6 months after the index cancer diagnosis were eligible for inclusion. For patient selection, we used the ICPC codes (International Classification of Primary Care) for AF, VTE, and any type of cancer excluding basal cell carcinoma, and ATC (Anatomical Therapeutic Chemical) codes for anticoagulants.15 For an overview, see online supplemental file 2.

    Data collection

    For patients already using anticoagulants before the cancer diagnosis, data collection started on the date of the cancer diagnosis (ie, the index date). For patients with active cancer who initiated anticoagulants within 6 months after the cancer diagnosis, the index date is the date of the first prescription of the anticoagulant. We used a maximum of 3 years of follow-up data, after which the disease episode of active cancer was considered dissolved. Because the theoretical end date of a drug prescription often does not correspond with the actual end of anticoagulant treatment (due to, eg, stockpiling or medication non-compliance), we used a ‘grace’ period of 14 days to extend the anticoagulant treatment period after the date of the last prescription to adjust for this, an approach often applied in the field of bleeding risk analyses.16 ICPC codes and ATC codes used for predictors and bleeding outcomes are listed in online supplemental file 2.

    Outcome definition

    The outcome of this study was the composite of major and CRNM bleeding defined as all bleeding events which at least led to face-to-face evaluation by a healthcare professional, based on the criteria of the International Society on Thrombosis and Haemostasis (ISTH).17 For data on bleeding, both coded data and free-text data were used. For every face-to-face contact registered under any bleeding-related, cancer-related, VTE-related or AF-related ICPC, the free text of the consultation was evaluated to verify that the patient in fact had a (new) bleeding event and to assess the location and type of bleeding. Outcome events were assessed without knowledge of the predictor information.

    Predictor definition

    Where possible we aimed to use the same definition for each predictor as described in the original validation study of each included risk model. If the exact information was not available, we used the best available approximation. See online supplemental file 3 for an overview of the predictor definitions used in our study. The absence or presence of predictors was assessed without knowledge of the outcome. In absence of an ICPC code or ATC code, the predictor was considered absent. Missing data on predictors were not imputed to reflect daily practice.

    Data analysis

    An incidence rate of the number of bleeds per 100 person-years with a 95% CI for a first CRNM or major bleeding during anticoagulant treatment was calculated. Data analyses were performed in R V.4.0.5.

    External validation

    To determine the predictive performance of the selected bleeding risk models in our study population, we aimed to assess calibration and discrimination. For discrimination, expressing the proportion in which the bleeding risk model correctly assigns the highest risk to those with a bleed in a random pair of patients (one with a bleed and one without a bleed) we calculated the c-statistic with corresponding 95% CI. For calibration, in absence of the expected risk compared with the observed bleeding risk (O/E ratio) and subsequent calibration plots, we reported the observed bleeding risk in each of the risk categories (as a proxy of the expected risk) where relevant.

    Model updating and internal validation

    Three Cox proportional hazards models were fitted. First, a simple baseline model was fitted including only the predictors ‘age’ and ‘sex’. Next, this baseline model was updated by including all predictors included in at least two out of four of the existing bleeding risk models selected for external validation. Finally, we added in a third model as a dichotomous variable the cancer types with a high risk of bleeding (ie, mucosal tumours and lung tumours) versus all other types of cancer. To account for possible non-linearity of age, a restricted cubic spline with four knots was used. For all three models, Akaike’s information criterion (AIC) was determined and likelihood ratio tests (LRT) were performed to compare the fit of the models. A final model was selected based on AIC and LRT. To account for the competing risks of non-bleeding-related death, a cause-specific hazard model was used for internal validation.18 The model was internally validated using 10-fold cross-validation, and calibration and discrimination were calculated. Finally, as an additional explorative analysis, backward elimination based on AIC was used for model reduction,19 and the predicted bleeding risk associated with the remaining predictors was plotted. Proportional hazard assumptions were visually checked for the final model by plotting scaled Schoenfeld residuals for every predictor.

    Results

    Descriptive statistics study population

    The validation cohort consisted of 1304 patients with cancer, mean age 74.0±10.9 years, 52.2% male. In total, 365 (28%) patients had a VTE diagnosis, 883 (67.7%) an AF diagnosis and 56 (4.3%) patients had both an AF and VTE diagnosis (table 1). At the index date, 655 (50.2%) patients used a VKA, 361 (27.7%) a DOAC, 285 (21.9%) LMWH and 3 (0.2%) patients heparin.

    View this table:
    Table 1

    Characteristics of the 1304 patients with cancer and anticoagulant treatment subdivided in patients with VTE, AF or both

    Bleeding events in the validation cohort

    In total 215 (16.5%) patients had at least one major or CRNM bleeding event during a mean follow-up of 1.5 (SD 1.2) years. The incidence rate for a first major or CRNM bleeding was 11.0 per 100 person-years (95% CI 9.6 to 12.5). Tables 2 and 3 provide an overview of the bleeding locations and characteristics of patients with and without bleeding, respectively.

    View this table:
    Table 2

    Bleeding location of major and clinically relevant non-major bleeding for which 215 of the 1304 patients with cancer and anticoagulant treatment contacted the GP during a mean follow-up of 1.5 years

    View this table:
    Table 3

    Characteristics of the 1304 patients with cancer and anticoagulation divided into those with and without bleeding event

    External validation of the existing bleeding risk models

    See online supplemental file 1 for the risk of bias assessment of the selected bleeding risk models. In table 4, the c-statistics with 95% CI for each of the existing bleeding risk models are presented. The c-statistics ranged between 0.55 (95% CI 0.51 to 0.59) and 0.56 (95% CI 0.52 to 0.60). Since we were not able to formally assess the calibration of the models, table 5 shows the distribution of patients and bleeding events across the risk strata of each bleeding risk model. Because all patients in our study have cancer, all patients were categorised in the high-risk category in case of the VTE-bleed. For some bleeding risk models, there was a doubling of the observed bleeding risk in the higher risk categories compared with the lower risk categories, for other bleeding risk models this increase was less pronounced.

    View this table:
    Table 4

    External validation of five bleeding risk models in the total study population; C-statistics with 95% CI

    View this table:
    Table 5

    Distribution of patients and bleeding events across the risk categories of each bleeding risk model

    Model updating and internal validation

    Our second model, consisting of the variables age, sex, hypertension, history of bleeding, renal insufficiency, anaemia and use of antiplatelet drugs performed best. For an elaboration, see online supplemental file 4. The adjusted c-statistic of the competing risk model after internal validation was 0.61 (95% CI 0.54 to 0.70). Table 6 demonstrates the HRs, CIs and internal validation performance measures of the competing risk model. In figure 1, the calibration plot of the final model is shown. Based on backward selection using model 2, only the predictors ‘age’ and ‘history of bleeding’ remained in the model. See figure 2 for the association between the expected bleeding risk and the predictors ‘age’ and ‘history of bleeding’.

    View this table:
    Table 6

    Internal validation of the competing risk model

    Figure 1
    Figure 1

    Calibration plot final model.

    Figure 2
    Figure 2

    Relation between estimated bleeding risk, age and history of bleeding.

    Discussion

    Our study shows that while bleeding is common in patients with cancer receiving anticoagulants (11 per 100 patient-years), existing and updated bleeding risk models with commonly available predictor variables were unable to differentiate the risk of bleeding in this population. Only age and a history of bleeding appeared to be relevant to estimate bleeding risk in our primary care dataset.

    Comparison with literature

    The incidence rate of a first major or CRNM bleeding event in our study was 11.0 per 100 person-years, which was higher than in previous studies in AF8–11 and in VTE patients.7 Because we selectively focused on patients with cancer and could include CRNM bleedings only seen by the GP and not necessarily by specialists, our incidence rate indeed is much higher. In a recently published study performed in patients with cancer-associated thrombosis who were included in a randomised trial, an incidence of major and CRNM bleeding of 14.2% was found during a follow-up period of 12 months,20 which is more comparable to what we found. Such a high bleeding risk certainly calls for a careful and shared decision on anticoagulant treatment based on an individually predicted bleeding risk.

    Performance of existing bleeding risk models including an updated competing risk model

    This study showed that predicting bleeding risk in patients with cancer is difficult with existing bleeding risk models. The c-statistics of 0.56 for all existing models indicate that in two individual patients with cancer on anticoagulant treatment, one experiencing bleeding and the other not, the probability that the patient experiencing the bleeding event receives a higher estimated bleeding risk from available models is ‘only’ 56%, thus almost similar as to flipping a coin. Model updating using state-of-the-art methodology only slightly improved model performance, yet this still would be considered poor performance against current standards. Moreover, this updated model did not yield sufficient calibration, likely due to insufficient sample size.

    Likely other predictors, not included in the valuated prediction models, may be predictive of bleeding in patients with cancer. Identifying these predictors and incorporating these in prediction models for bleeding should be the focus of further studies. The recently developed CAT-BLEED model (developed in patients with cancer-associated thrombosis) includes cancer-related factors such as cancer subtypes and associated chemotherapy.20 Although detailed information on chemotherapy was not available in our dataset, we were not able to confirm that cancer subtypes provide reliable incremental prognostic information with relation to bleeding risk. We did, however, observe that age and a prior history of bleeding are useful, and these predictors are ‘ready at hand’.

    Clinical implications and future considerations

    For patients with active cancer using anticoagulants, a clinically relevant or major bleed may be an impactful event that perhaps could influence further anticoagulant treatment decisions. Our analyses demonstrate that such bleeding events indeed occur frequently in anticoagulated patients with cancer, highlighting the need for shared decision-making with respect to anticoagulant treatment. To support such decisions, for example, (temporarily) withholding anticoagulation, reducing the dose or switching between anticoagulants, an accurately predicted risk of bleeding for balancing against the benefits of anticoagulation is an important necessity. There are several steps to be taken to improve prediction in future research. First, to improve the value of known predictors, reporting on bleeding risk models and the modelling itself need improvement. All models included in our study had a high risk of bias according to the PROBAST guideline due to the lack of relevant information on, for example, predictor selection or predictor assessment, not handling missing data appropriately, or not accounting for competing risk. This in fact is a more general call for better reporting on bleeding risk models, not only for the subgroup of patients with cancer. Second, further research is needed to identify future cancer-specific risk factors for bleeding and to include these in bleeding risk models. Ultimately, though, after accurate prediction, randomized controlled trials are needed that evaluate clinically relevant outcomes when anticoagulation is reduced with the aim to mitigate bleeding risk in high-risk individuals. Thus, not only identifying those who may experience a bleed but also what to employ in order to prevent these bleeds. Finally, future developments in anticoagulants with lower risk of bleeding, such as with factor XIa inhibitors are promising. These drugs are currently tested in phase II and III trials, and they may also reduce bleeding risk in patients with cancer as they do in patients without cancer.21 22

    Strengths and limitations

    A strength of this study is that the dataset consisted of a large and representative sample of anticoagulated patients with cancer with VTE and/or AF, managed in the community (with data extracted from a longitudinal primary care database), encompassing all patients with cancer, also those only seen by the GP for bleeding complications. All clinically relevant bleeding outcomes could be included, also CRNM bleeding only reported by patients to the GP, including, for example, so-called ‘nuisance’ bleeds’ (euphemistically), which are not always reported or registered in hospital datasets. Also, clinically relevant bleeding which occurred in the hospital setting was registered in our database as well, and thus included in our analysis. For model updating, we used state-of-the-art methodology including a cause-specific Cox-proportional hazard model, which is a recommended method for analysis of time-to-event data in the presence of competing risk (ie, non-bleeding related death).18 Not accounting for competing risks can lead to an overestimation of the bleeding risk, certainly in a population of patients with cancer.23 A limitation of this study is that both misclassification of the outcome and of predictors could have occurred. Regarding the predictors, most predictor definitions used in our validation closely resembled the definitions used in the development studies, however, some minor differences could not be avoided. We relied on data from routine care, and while we were able to assess free text for all patients, a distinction between major and CRNM was not possible. Last, due to a lack of power we were not able to study the predictive performance of the models in subgroups (eg, for AF and VTE patients separately, type of cancer or type of anticoagulant).

    Conclusions

    Bleeding in patients with cancer could not be accurately predicted using commonly used existing bleeding risk models. Only age and a history of bleeding were shown to have incremental predictive value and should be considered in future prediction models for bleeding in patients with cancer at risk for thromboembolic complications.

    Data availability statement

    Data may be obtained from a third party and are not publicly available.

    Ethics statements

    Patient consent for publication

    Ethics approval

    The study was conducted in accordance with the General Data Protection Regulation (GDPR) and other regulations, acts and guidelines. The Medical Research Ethics Committee (MREC) of the UMC Utrecht confirmed that the Dutch law on Medical Research Involving Human Subjects Act (WMO) does not apply to this study and that official approval of this study by the MREC was not required under Dutch legislation. For this study, only deidentified data were used, meaning data cannot be directly traced back to the patient.

    References

    1. ↵
      1. Khorana AA
      . Venous thromboembolism and prognosis in cancer. Thromb Res 2010;125:490–3. doi:10.1016/j.thromres.2009.12.023
      OpenUrlCrossRefPubMed
    2. ↵
      1. Kamphuisen PW,
      2. Beyer-Westendorf J
      . Bleeding complications during anticoagulant treatment in patients with cancer. Thromb Res 2014;133 Suppl 2:S49–55. doi:10.1016/S0049-3848(14)50009-6
      OpenUrl
    3. ↵
      1. Kaptein FHJ,
      2. Stals MAM,
      3. Kapteijn MY, et al
      . Incidence and determinants of thrombotic and bleeding complications in patients with glioblastoma. J Thromb Haemost 2022;20:1665–73. doi:10.1111/jth.15739
      OpenUrl
    4. ↵
      1. Angelini DE,
      2. Radivoyevitch T,
      3. McCrae KR, et al
      . Bleeding incidence and risk factors among cancer patients treated with anticoagulation. Am J Hematol 2019;94:780–5. doi:10.1002/ajh.25494
      OpenUrl
    5. ↵
      1. Prommer E
      . Management of bleeding in the terminally ill patient. Hematology 2005;10:167–75. doi:10.1080/10245330500093237
      OpenUrlPubMed
    6. ↵
      1. den Exter PL,
      2. Woller SC,
      3. Robert-Ebadi H, et al
      . Management of bleeding risk in patients who receive anticoagulant therapy for venous thromboembolism: communication from the ISTH SSC Subcommittee on predictive and diagnostic variables in thrombotic disease. J Thromb Haemost 2022;20:1910–9. doi:10.1111/jth.15776
      OpenUrl
    7. ↵
      1. Klok FA,
      2. Hösel V,
      3. Clemens A, et al
      . Prediction of bleeding events in patients with venous thromboembolism on stable anticoagulation treatment. Eur Respir J 2016;48:1369–76. doi:10.1183/13993003.00280-2016
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Pisters R,
      2. Lane DA,
      3. Nieuwlaat R, et al
      . A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro heart survey. Chest 2010;138:1093–100. doi:10.1378/chest.10-0134
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Fang MC,
      2. Go AS,
      3. Chang Y, et al
      . A new risk scheme to predict warfarin-associated hemorrhage: the atria (anticoagulation and risk factors in atrial fibrillation) study. J Am Coll Cardiol 2011;58:395–401. doi:10.1016/j.jacc.2011.03.031
      OpenUrlFREE Full Text
    10. ↵
      1. Chu G,
      2. Valerio L,
      3. van der Wall SJ, et al
      . Tailoring anticoagulant treatment of patients with atrial fibrillation using a novel bleeding risk score. Heart 2020;107:549–55. doi:10.1136/heartjnl-2019-316305
      OpenUrl
    11. ↵
      1. O’Brien EC,
      2. Simon DN,
      3. Thomas LE, et al
      . The orbit bleeding score: a simple bedside score to assess bleeding risk in atrial fibrillation. Eur Heart J 2015;36:ehv476. doi:10.1093/eurheartj/ehv476
    12. ↵
      1. Wolff RF,
      2. Moons KGM,
      3. Riley RD, et al
      . PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med 2019;170:51–8. doi:10.7326/M18-1376
      OpenUrlCrossRefPubMed
    13. ↵
      1. Smeets HM,
      2. Kortekaas MF,
      3. Rutten FH, et al
      . Routine primary care data for scientific research, quality of care programs and educational purposes: the Julius General Practitioners’ Network (JGPN). BMC Health Serv Res 2018;18:735. doi:10.1186/s12913-018-3528-5
    14. ↵
    15. ↵
      1. Hofmans-Okkes IM,
      2. Lamberts H
      . The International Classification of Primary Care (ICPC): new applications in research and computer-based patient records in family practice. Fam Pract 1996;13:294–302. doi:10.1093/fampra/13.3.294
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. van den Dries CJ,
      2. van Doorn S,
      3. Souverein P, et al
      . The number of concomitant drugs and the safety of direct oral anticoagulants in routine care patients with atrial fibrillation. TH Open 2020;4:e417–26. doi:10.1055/s-0040-1721499
      OpenUrl
    17. ↵
      1. Kaatz S,
      2. Ahmad D,
      3. Spyropoulos AC, et al
      . Definition of clinically relevant non-major bleeding in studies of anticoagulants in atrial fibrillation and venous thromboembolic disease in non-surgical patients: communication from the SSC of the ISTH. J Thromb Haemost 2015;13:2119–26. doi:10.1111/jth.13140
      OpenUrlCrossRefPubMed
    18. ↵
      1. van Geloven N,
      2. Giardiello D,
      3. Bonneville EF, et al
      . Validation of prediction models in the presence of competing risks: a guide through modern methods. BMJ 2022;377:e069249. doi:10.1136/bmj-2021-069249
    19. ↵
      1. van Smeden M,
      2. Moons KG,
      3. de Groot JA, et al
      . Sample size for binary logistic prediction models: beyond events per variable criteria. Stat Methods Med Res 2019;28:2455–74. doi:10.1177/0962280218784726
      OpenUrlCrossRefPubMed
    20. ↵
      1. de Winter MA,
      2. Dorresteijn JAN,
      3. Ageno W, et al
      . Estimating bleeding risk in patients with cancer-associated thrombosis: evaluation of existing risk scores and development of a new risk score. Thromb Haemost 2022;122:818–29. doi:10.1055/s-0041-1735251
      OpenUrl
    21. ↵
      1. Weitz JI,
      2. Strony J,
      3. Ageno W, et al
      . Milvexian for the prevention of venous thromboembolism. N Engl J Med 2021;385:2161–72. doi:10.1056/NEJMoa2113194
      OpenUrl
    22. ↵
      1. Piccini JP,
      2. Caso V,
      3. Connolly SJ, et al
      . Safety of the oral factor xia inhibitor asundexian compared with apixaban in patients with atrial fibrillation (PACIFIC-AF): a multicentre, randomised, double-blind, double-dummy, dose-finding phase 2 study. Lancet 2022;399:1383–90. doi:10.1016/S0140-6736(22)00456-1
      OpenUrl
    23. ↵
      1. Ramspek CL,
      2. Teece L,
      3. Snell KIE, et al
      . Lessons learnt when accounting for competing events in the external validation of time-to-event prognostic models. Int J Epidemiol 2022;51:615–25. doi:10.1093/ije/dyab256
      OpenUrl

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Contributors EMT-R, SvD, G-JG, FR and MvS were involved in the design of the study. EMT-R and SvD performed data analysis and wrote the first draft of the manuscript. All authors were involved in the interpretation of the results, revising the manuscript and approval of the final manuscript. EMT-R is responsible for the overall content as guarantor.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests SvD reports an unrestricted institutional grant for performing research in the field of stroke diagnosis from Stoffels-Hornstra. G-JG, FR and MH report unrestricted institutional grants for performing research in the field of atrial fibrillation from Boehringer-Ingelheim, Bayer Healthcare, BMS Pfizer and Daiichi Sankyo. FAK reports research support from Bayer, BMS, BSCI, MSD, Leo Pharma, Actelion, The Netherlands Organisation for Health Research and Development, The Dutch Thrombosis Association, The Dutch Heart Foundation and the Horizon Europe Program. ICVG reports consultancy fees from Boston, BMS and Bayer to the institution, unrestricted research grants from the Netherlands Cardiovascular Research Initiative, unrestricted research grant from the European Union’s Horizon 2020 research and innovation programme under grant agreement; EHRA-PATHS (945260).

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.