The Methodology of Human Diseases Risk Prediction Tools
DOI:
https://doi.org/10.6000/1929-6029.2013.02.03.9Keywords:
Disease risk prediction, missing data, model validation, model updating, model utilityAbstract
Disease risk prediction tools are used for population screening and to guide clinical care. They identify which individuals have particularly elevated risk of disease. The development of a new risk prediction tool involves several methodological components including: selection of a general modelling framework and specific functional form for the new tool, making decisions about the inclusion of risk factors, dealing with missing data in those risk factors, and performing validation checks of a new tool’s performance. There have been many methodological developments of relevance to these issues in recent years. Developments of importance for disease detection in humans were reviewed and their uptake in risk prediction tool development illustrated. This review leads to guidance on appropriate methodology for future risk prediction development activities.
References
Steyerberg EW. Clinical prediction models: A practical approach to development, validation, and updating. New York: Springer 2009.
van Steenkiste B, van der Weijden T, Timmermans D, et al. Patients’ ideas, fears and expectations of their coronary risk: barriers for primary prevention. Patient Educ Counsel 2004; 55(2): 301-307. http://dx.doi.org/10.1016/j.pec.2003.11.005 DOI: https://doi.org/10.1016/j.pec.2003.11.005
Moons KGM, Harrell FE, Steyerberg EW. Should scoring rules be based on odds ratios or regression coefficients? [letter] J Clin Epidemiol 2002; 55(10): 1054-55. http://dx.doi.org/10.1016/S0895-4356(02)00453-5 DOI: https://doi.org/10.1016/S0895-4356(02)00453-5
Baker SG, Cook NR, Vickers A, et al. Using relative utility curves to evaluate risk prediction. J Royal Stat Soc A 2009; 172(4): 729-48. http://dx.doi.org/10.1111/j.1467-985X.2009.00592.x DOI: https://doi.org/10.1111/j.1467-985X.2009.00592.x
Vickers AJ, Cronin AM. Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: Towards a decision analytic framework. Sem Oncol 2010; 37(1): 31-38. http://dx.doi.org/10.1053/j.seminoncol.2009.12.004 DOI: https://doi.org/10.1053/j.seminoncol.2009.12.004
Beswick A, Brindle P. Risk scoring in the assessment of cardiovascular risk. Curr Opin Lipidol 2006; 17(4): 375-86. http://dx.doi.org/10.1097/01.mol.0000236362.56216.44 DOI: https://doi.org/10.1097/01.mol.0000236362.56216.44
Truett J, Kornfeld J, Kannel W. A multivariate analysis of the risk of coronary heart disease in Framingham. J Chronic Diseases 1967; 20(7): 511-24. http://dx.doi.org/10.1016/0021-9681(67)90082-3 DOI: https://doi.org/10.1016/0021-9681(67)90082-3
Walker SH, Duncan DB. Estimation of the probability of an event as a function of several independent variables. Biometrika 1967; 54(1): 167-79. DOI: https://doi.org/10.1093/biomet/54.1-2.167
Knuiman MW, Vu HTV, Segal MR. An empirical comparison of multivariable methods for estimating risk of death from coronary heart disease. Eur J Cardio Prev Rehab 1997; 4(2): 127-34. http://dx.doi.org/10.1177/174182679700400209 DOI: https://doi.org/10.1097/00043798-199704000-00009
Chambless LE, Diao G. Estimation of time-dependent area under the ROC curve for long-term risk prediction. Stat Med 2006; 25(20): 3474-86. http://dx.doi.org/10.1002/sim.2299 DOI: https://doi.org/10.1002/sim.2299
Tyrer J, Duffy SW, Cuzick J. A breast cancer prediction model incorporating familial and personal risk factors. Stat Med 2004; 23(7): 1111-30. http://dx.doi.org/10.1002/sim.1668 DOI: https://doi.org/10.1002/sim.1668
Anderson KM. A nonproportional hazards Weibull accelerated failure time regression model. Biometrics 1991; 47(1): 281-88. http://dx.doi.org/10.2307/2532512 DOI: https://doi.org/10.2307/2532512
Assmann G, Cullen P, Schulte H. Simple scoring scheme for calculating the risk of acute coronary events based on the 10-Year follow-up of the Prospective Cardiovascular Münster (PROCAM) study. Circulation 2002; 105(3): 310-15. http://dx.doi.org/10.1161/hc0302.102575 DOI: https://doi.org/10.1161/hc0302.102575
Conroy RM, Pyorala K, Fitzgerald AP. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J 2003; 24(11): 987-1003. http://dx.doi.org/10.1016/S0195-668X(03)00114-3 DOI: https://doi.org/10.1016/S0195-668X(03)00114-3
D’Agostino RB, Sr,Vasan RS, Pencina MJ, et al. General cardiovascular risk profile for use in primary care. Circulation 2008; 117(6): 743-53. http://dx.doi.org/10.1161/CIRCULATIONAHA.107.699579 DOI: https://doi.org/10.1161/CIRCULATIONAHA.107.699579
Hippisley-Cox J, Coupland C, Vinogradova Y, et al. Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2. BMJ 2008; 336(7659): 1475-82. http://dx.doi.org/10.1136/bmj.39609.449676.25 DOI: https://doi.org/10.1136/bmj.39609.449676.25
Hippisley-Cox J, Coupland C, Robson J, et al. Predicting risk of type 2 diabetes in England and Wales: prospective derivation and validation of QDScore. [electronic article] BMJ 2009; 338: b880. http://dx.doi.org/10.1136/bmj.b880 DOI: https://doi.org/10.1136/bmj.b880
Anderson KM, Odell, PM, Wilson PWF, et al. Cardiovascular disease risk profiles. Am Heart J 1990; 121(1) part 2: 293-98. DOI: https://doi.org/10.1016/0002-8703(91)90861-B
Odell PM, Anderson KM, Kannel WB. New models for predicting cardiovascular events. J Clin Epidemiol 1994; 47(6): 583-92. http://dx.doi.org/10.1016/0895-4356(94)90206-2 DOI: https://doi.org/10.1016/0895-4356(94)90206-2
Bang H, Vupputuri S, Shoham DA, et al. Screening for Occult Renal Disease (SCORED) A Simple prediction model for chronic kidney disease. Arch Internal Med 2007; 167(4): 374-81. http://dx.doi.org/10.1001/archinte.167.4.374 DOI: https://doi.org/10.1001/archinte.167.4.374
Bang H, Mazumdar M, Newman G, et al. Screening for kidney disease in vascular patients: SCreening for Occult REnal Disease (SCORED) experience. Nephrol Dial Transplant 2009; 24(8): 2452-57. http://dx.doi.org/10.1093/ndt/gfp124 DOI: https://doi.org/10.1093/ndt/gfp124
Panthangi V, West P, Savoy-Moore RT, et al. Is seasonal variation another risk factor for postpartum depression? J Am Board Family Med 2009; 22(5): 492-97. http://dx.doi.org/10.3122/jabfm.2009.05.080066 DOI: https://doi.org/10.3122/jabfm.2009.05.080066
Li G, Fansi AK, Boivin J-F, et al. Screening for glaucoma in high-risk populations using optical coherence tomography. Ophthalmology 2010; 117(3): 453-61. http://dx.doi.org/10.1016/j.ophtha.2009.07.033 DOI: https://doi.org/10.1016/j.ophtha.2009.07.033
Harrell FE. Regression Modeling Strategies, New York: Springer-Verlag 2001. DOI: https://doi.org/10.1007/978-1-4757-3462-1
Raghunathan TE. What do we do with missing data? Some options for analysis of incomplete data. Annual Rev Public Health 2004; 25: 99-117. http://dx.doi.org/10.1146/annurev.publhealth.25.102802.124410 DOI: https://doi.org/10.1146/annurev.publhealth.25.102802.124410
Ambler G, Omar RZ, Royston P. A comparison of imputation techniques for handling missing predictor values in a risk model with a binary outcome. Stat Methods Med Res 2007; 16(3): 277-98. http://dx.doi.org/10.1177/0962280206074466 DOI: https://doi.org/10.1177/0962280206074466
Anderson AB, Basilevsky A, Hum DPJ. Missing Data. A Review of the Literature. In: Handbook of Survey Research, New York, NY: Academic Press 1983; pp. 415-492. DOI: https://doi.org/10.1016/B978-0-12-598226-9.50018-5
Tice JA, Cummings SR, Smith-Bindman R, et al. Using clinical factors and mammographic breast density to estimate breast cancer risk: development and validation of a new predictive model. Ann Intern Med 2008; 148(5): 337-47. http://dx.doi.org/10.7326/0003-4819-148-5-200803040-00004 DOI: https://doi.org/10.7326/0003-4819-148-5-200803040-00004
Bach PB, Kattan MW, Thornquist MD, et al. Variations in lung cancer risk among smokers. J Natl Cancer Inst 2003; 95(6): 470-78. http://dx.doi.org/10.1093/jnci/95.6.470 DOI: https://doi.org/10.1093/jnci/95.6.470
Barlow WE, White E, Ballard-Barbash R, et al. Prospective breast cancer risk prediction model for women undergoing screening mammography. J Natl Cancer Inst 2006; 98(17): 1204-14. http://dx.doi.org/10.1093/jnci/djj331 DOI: https://doi.org/10.1093/jnci/djj331
Greenland S, Finkle WD. A critical look at methods for handling missing covariates in epidemiologic regression analyses. Am J Epidemiol 1995; 142(12): 1255-64. DOI: https://doi.org/10.1093/oxfordjournals.aje.a117592
Vach W, Blettner M. Biased estimation of the odds ratio in case-control studies due to the use of ad hoc methods of correcting for missing values for confounding variables. Am J Epidemiol 1991; 134(8): 895-907. DOI: https://doi.org/10.1093/oxfordjournals.aje.a116164
van der Heijden GJ, Donders ART, Stijnene T, et al. Imputation of missing values is superior to complete case analysis and the missing indicator method in multivariable diagnostic research: A clinical example. J Clin Epidemiol 2006; 59(10): 1102-109. http://dx.doi.org/10.1016/j.jclinepi.2006.01.015 DOI: https://doi.org/10.1016/j.jclinepi.2006.01.015
Schemper M, Heinze G. Probability imputation revisited for prognostic factor studies. Stat Med 1997; 16(1): 73-80. http://dx.doi.org/10.1002/(SICI)1097-0258(19970115)16:1<73::AID-SIM472>3.0.CO;2-Z DOI: https://doi.org/10.1002/(SICI)1097-0258(19970115)16:1<73::AID-SIM472>3.0.CO;2-Z
Yuhara T, Takemura H, Akama T, et al. Predicting infection in hospitalized patients with systemic lupus erythematosus. Intern Med 1996; 35(8): 629-36. http://dx.doi.org/10.2169/internalmedicine.35.629 DOI: https://doi.org/10.2169/internalmedicine.35.629
Little RJA, Rubin DB. Statistical Analysis with Missing Data, 2nd ed. New York: Wiley 2002. DOI: https://doi.org/10.1002/9781119013563
Rubin DB, Schenker N. Multiple imputation for interval estimation from simple random samples with ignorable nonresponse. JASA 1986; 81(394): 366-74. http://dx.doi.org/10.1080/01621459.1986.10478280 DOI: https://doi.org/10.1080/01621459.1986.10478280
Little RJA. Missing-data adjustments in large surveys. J Bus Econ Stat 1988; 6(3): 287-96. DOI: https://doi.org/10.1080/07350015.1988.10509663
Schafer JL. Analysis of Incomplete Multivariate Data, 1st ed. London: Chapman and Hall 1997. http://dx.doi.org/10.1201/9781439821862 DOI: https://doi.org/10.1201/9781439821862
van Buuren S, Brand J, Groothuis-Oudshoorn C, et al. Fully conditional specification in multivariate imputation. J Stat Comp Simul 2006; 76(12): 1049-64. http://dx.doi.org/10.1080/10629360600810434 DOI: https://doi.org/10.1080/10629360600810434
van Buuren S. Multiple imputation of discrete and continuous data by fully conditional specification. Stat Meth Med Res 2007; 16(3): 219-42. http://dx.doi.org/10.1177/0962280206074463 DOI: https://doi.org/10.1177/0962280206074463
van Buuren S, Groothuis-Oudshoorn K. MICE 2.0: Multivariate Imputation by Chained Equations in R. J Stat Soft 2011; 45(3). DOI: https://doi.org/10.18637/jss.v045.i03
Stata Corporation. Stata statistical software, release 11. Multiple Imputation Reference Manual. College Station, TX: StataCorp LP 2005.
Royston P. Multiple imputation of missing values. The Stata J 2004; 4(3): 227-41. DOI: https://doi.org/10.1177/1536867X0400400301
Royston P. Multiple imputation of missing values: Update of ice. The Stata J 2005; 5(4): 527-36. DOI: https://doi.org/10.1177/1536867X0500500404
Royston P, Carlin JB, White IR. Multiple imputation of missing values: New features for mim. The Stata J 2009; 9(2): 252-64. DOI: https://doi.org/10.1177/1536867X0900900205
Allison PD. Multiple imputation of missing data: A Cautionary Tale. Sociol Meth Res 2000; 28(3): 301-309. http://dx.doi.org/10.1177/0049124100028003003 DOI: https://doi.org/10.1177/0049124100028003003
Moons KG, Donders RA, Stijnen T, et al. Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol 2006; 59(10): 1092-101. http://dx.doi.org/10.1016/j.jclinepi.2006.01.009 DOI: https://doi.org/10.1016/j.jclinepi.2006.01.009
Hippisley-Cox J, Coupland C, Vinogradova Y, et al. Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study. BMJ 2007; 335(7611): 136-47. http://dx.doi.org/10.1136/bmj.39261.471806.55 DOI: https://doi.org/10.1136/bmj.39261.471806.55
van Buuren S, Boshuizen HC, Knook DL. Multiple imputation of missing blood pressure covariates in survival analysis. Stat Med 1999; 18(6): 681-94. http://dx.doi.org/10.1002/(SICI)1097-0258(19990330)18:6<681::AID-SIM71>3.0.CO;2-R DOI: https://doi.org/10.1002/(SICI)1097-0258(19990330)18:6<681::AID-SIM71>3.0.CO;2-R
Clark TG, Altman DG. Developing a prognostic model in the presence of missing data: an ovarian cancer case study. J Clin Epidemiol 2003; 56(1): 28-37. http://dx.doi.org/10.1016/S0895-4356(02)00539-5 DOI: https://doi.org/10.1016/S0895-4356(02)00539-5
Spratt M, Carpenter J, Sterne JAC, Carlin JB, Heron J, Henderson J, Tilling K. Strategies for Multiple Imputation in Longitudinal Studies. Am J Epidemiol 2010; 172: 478-87. http://dx.doi.org/10.1093/aje/kwq137 DOI: https://doi.org/10.1093/aje/kwq137
Meng X-L. Multiple-Imputation Inferences with Uncongenial Sources of Input. Stat Sci 1994; 9: 538-73. DOI: https://doi.org/10.1214/ss/1177010269
Schafer JL, Graham JW. Missing Data: Our View of the State of the Art. Psycho Meth 2002; 7: 147-77. http://dx.doi.org/10.1037/1082-989X.7.2.147 DOI: https://doi.org/10.1037/1082-989X.7.2.147
Kenward MG, Carpenter J. Multiple imputation: current perspectives. Stats Meth Med Res 2007; 16: 199-18. http://dx.doi.org/10.1177/0962280206075304 DOI: https://doi.org/10.1177/0962280206075304
Fazio VW, Paris PT, Remzi F, et al. Assessment of operative risk in colorectal cancer surgery: The Cleveland Clinic Foundation colorectal cancer model. Dis Colon Rectum 2004; 47(12): 2015-24. http://dx.doi.org/10.1007/s10350-004-0704-y DOI: https://doi.org/10.1007/s10350-004-0704-y
Gencer B, Vaucher P, Herzig L, et al. Ruling out coronary heart disease in primary care patients with chest pain: a clinical prediction score [electronic article]. BMC Med 2010; 8: 9. http://dx.doi.org/10.1186/1741-7015-8-9 DOI: https://doi.org/10.1186/1741-7015-8-9
Steyerberg EW, Harrell FE Jr., Borsboom GJ, et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001; 54(8): 774-81. http://dx.doi.org/10.1016/S0895-4356(01)00341-9 DOI: https://doi.org/10.1016/S0895-4356(01)00341-9
Gal GL, Righini M, Roy P-M, et al. Prediction of pulmonary embolism in the Emergency Department: The revised Geneva score. Ann Intern Med 2006; 144(3): 165-71. http://dx.doi.org/10.7326/0003-4819-144-3-200602070-00004 DOI: https://doi.org/10.7326/0003-4819-144-3-200602070-00004
Stallard N. Simple tests for the external validation of mortality prediction scores. Stat Med 2009; 28: 377-88. http://dx.doi.org/10.1002/sim.3393 DOI: https://doi.org/10.1002/sim.3393
Aujesky D, Obrosky DS, Stone RA, et al. A prediction rule to identify low-risk patients with pulmonary embolism. Arch Intern Med 2006; 166(2): 169-75. http://dx.doi.org/10.1001/archinte.166.2.169 DOI: https://doi.org/10.1001/archinte.166.2.169
Antolini L, Nam BH, Agostino RB. Inference on correlated discrimination measures in survival analysis: a nonparametric approach. Commun Stat Theory Methods 2004; 33(9): 2117-35. http://dx.doi.org/10.1081/STA-200026579 DOI: https://doi.org/10.1081/STA-200026579
Pepe MS, Janes H, Longton G, et al. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 2004; 159(9): 882-90. http://dx.doi.org/10.1093/aje/kwh101 DOI: https://doi.org/10.1093/aje/kwh101
Greenland P, O’Malley PG. When is a new prediction marker useful? A consideration of lipoprotein-associated phospholipase A2 and C-reactive protein for stroke risk. Arch Intern Med 2005; 165(21): 2454-56. http://dx.doi.org/10.1001/archinte.165.21.2454 DOI: https://doi.org/10.1001/archinte.165.21.2454
Ware JH. The limitations of risk factors as prognostic tools. NEJM 2006; 355(25): 2615-17. http://dx.doi.org/10.1056/NEJMp068249 DOI: https://doi.org/10.1056/NEJMp068249
Janes H, Pepe MS, Gu W. Assessing the value of risk predictions by using risk stratification tables. Annals Intern Med 2008; 149: 751-60. http://dx.doi.org/10.1002/sim.2929
Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27(2): 157-72. DOI: https://doi.org/10.1002/sim.2929
Cui J. Overview of risk prediction models in cardiovascular disease research. Annals Epidemiol 2009; 19(10): 711-17. http://dx.doi.org/10.1016/j.annepidem.2009.05.005 DOI: https://doi.org/10.1016/j.annepidem.2009.05.005
Tice JA, Cummings SR, Ziv E, et al. Mammographic breast density and the Gail model for breast cancer risk prediction in a screening population. Breast Cancer Res Treat 2005; 94(2): 115-22. http://dx.doi.org/10.1007/s10549-005-5152-4 DOI: https://doi.org/10.1007/s10549-005-5152-4
van der Steeg WA, Boekholdt SM, Stein EA, et al. Role of the apolipoprotein B-apolipoprotein A-I ratio in cardiovascular risk assessment: a case-control analysis in EPIC-Norfolk. Ann Intern Med 2007; 146(9); 640-48. http://dx.doi.org/10.7326/0003-4819-146-9-200705010-00007 DOI: https://doi.org/10.7326/0003-4819-146-9-200705010-00007
Lauer MS, Pothier CE, Magid DJ, et al. An externally validated model for predicting long-term survival after exercise treadmill testing in patients with suspected coronary
artery disease and a normal electrocardiogram. Ann Intern Med 2007; 147(12): 821-28. http://dx.doi.org/10.7326/0003-4819-147-12-200712180-00001 DOI: https://doi.org/10.7326/0003-4819-147-12-200712180-00001
Janes H, Pepe MS, Gu W. Assessing the value of risk predictions by using risk stratification tables. Ann Intern Med 2008; 149(10): 751-60. http://dx.doi.org/10.7326/0003-4819-149-10-200811180-00009 DOI: https://doi.org/10.7326/0003-4819-149-10-200811180-00009
Zorn KC, Capitanio U, Jeldres C, et al. Multi-institutional external validation of seminal vesicle invasion nomograms: head-to-head comparison of Gallina nomogram versus 2007 Partin tables. Int J Radiat Oncol Biol Phys 2009; 73(5): 1461-67. http://dx.doi.org/10.1016/j.ijrobp.2008.06.1913 DOI: https://doi.org/10.1016/j.ijrobp.2008.06.1913
Harrell FE Jr, Lee, KL, Califf RM, et al. Regression modelling strategies for improved prognostic prediction. Stat Med 1984; 3(2): 143-52. http://dx.doi.org/10.1002/sim.4780030207 DOI: https://doi.org/10.1002/sim.4780030207
Lee KL, Pryor DB, Harrell FE Jr., et al. Predicting outcome in coronary disease: Statistical models versus expert clinicians. Am J Med 1986; 80(4): 553-60. http://dx.doi.org/10.1016/0002-9343(86)90807-7 DOI: https://doi.org/10.1016/0002-9343(86)90807-7
D 'Agostino RB Sr, Grundy S, Sullivan LM, et al. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA 2001; 286(2): 180-7. http://dx.doi.org/10.1001/jama.286.2.180 DOI: https://doi.org/10.1001/jama.286.2.180
Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999; 130(6): 515-24. http://dx.doi.org/10.7326/0003-4819-130-6-199903160-00016 DOI: https://doi.org/10.7326/0003-4819-130-6-199903160-00016
Chambless LE, Cummiskey CP, Cui G. Several methods to assess improvement in risk prediction models: Extension to survival Analysis. Stat Med 2011; 30(1): 22-38. http://dx.doi.org/10.1002/sim.4026 DOI: https://doi.org/10.1002/sim.4026
Whittemore AS. Evaluating health risk models. Stat Med 2011; 29(23): 2438-52. http://dx.doi.org/10.1002/sim.3991 DOI: https://doi.org/10.1002/sim.3991
van Calster B, Vickers AJ, Pencina MJ, et al. Evaluation of markers and risk prediction models: Overview of relationships between NRI and decision-analytic measures. Med Decis Making 2013; 33(4): 490-501. http://dx.doi.org/10.1177/0272989X12470757 DOI: https://doi.org/10.1177/0272989X12470757
Royston P, Sauerbrei W. A new measure of prognostic separation in survival data. Stat Med 2004; 23: 723-48. http://dx.doi.org/10.1002/sim.1621 DOI: https://doi.org/10.1002/sim.1621
Royston P, Altman DG. Visualizing and assessing discrimination in the logistic regression model. Stat Med 2010; 29: 2508-20. http://dx.doi.org/10.1002/sim.3994 DOI: https://doi.org/10.1002/sim.3994
Pencina MJ, Steyerberg EW, D’Agostino RB. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 2011; 30(1): 11-21. http://dx.doi.org/10.1002/sim.4085 DOI: https://doi.org/10.1002/sim.4085
Pfeiffer RM. Extensions of criteria for evaluating risk prediction models for public health applications. Biostatistics 2013; 14(2): 366-381. http://dx.doi.org/10.1093/biostatistics/kxs037 DOI: https://doi.org/10.1093/biostatistics/kxs037
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2013 H. Mannan, R. Ahmed, M. Sanagou, S. Ivory, R. Wolfe
This work is licensed under a Creative Commons Attribution 4.0 International License.
Policy for Journals/Articles with Open Access
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are permitted and encouraged to post links to their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
Policy for Journals / Manuscript with Paid Access
Authors who publish with this journal agree to the following terms:
- Publisher retain copyright .
- Authors are permitted and encouraged to post links to their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work .
