High-Dimensional Fixed Effects Profiling Models and Applications in End-Stage Kidney Disease Patients: Current State and Future Directions

Danh V.  Nguyen; Qi  Qian; Amy S.  You; Esra  Kurum; Connie M.  Rhee; Damla  Senturk

doi:10.6000/1929-6029.2023.12.24

Authors

Danh V. Nguyen Department of Medicine, University of California Irvine, Orange, CA 92868, USA https://orcid.org/0000-0002-4025-8239
Qi Qian Department of Biostatistics, University of California, Los Angeles, CA 90095, USA
Amy S. You Department of Medicine, University of California Irvine, Orange, CA 92868, USA
Esra Kurum Department of Statistics, University of California, Riverside, CA 92521, USA https://orcid.org/0000-0003-1767-1671
Connie M. Rhee Department of Medicine, University of California, Los Angeles, CA 90095; VA Greater Los Angeles Medical Center, Los Angeles, CA 90073, USA
Damla Senturk Department of Biostatistics, University of California, Los Angeles, CA 90095, USA

DOI:

https://doi.org/10.6000/1929-6029.2023.12.24

Keywords:

Dialysis facility staffing, end-stage kidney disease, fixed effects, generalized linear mixed model, high-dimensional parameters, multilevel varying coefficient model, Poisson regression, propensity score, random effects, United States Renal Data System

Abstract

Profiling analysis aims to evaluate health care providers, including hospitals, nursing homes, or dialysis facilities among others with respect to a patient outcome, such as 30-day unplanned hospital readmission or mortality. Fixed effects (FE) profiling models have been developed over the last decade, motivated by the overall need to (a) improve accurate identification or “flagging” of under-performing providers, (b) relax assumptions inherent in random effects (RE) profiling models, and (c) take into consideration the unique disease characteristics and care/treatment processes of end-stage kidney disease (ESKD) patients on dialysis. In this paper, we review the current state of FE methodologies and their rationale in the ESKD population and illustrate applications in four key areas: profiling dialysis facilities for (1) patient hospitalizations over time (longitudinally) using standardized dynamic readmission ratio (SDRR), (2) identification of dialysis facility characteristics (e.g., staffing level) that contribute to hospital readmission, and (3) adverse recurrent events using standardized event ratio (SER). Also, we examine the operating characteristics with a focus on FE profiling models. Throughout these areas of applications to the ESKD population, we identify challenges for future research in both methodology and clinical studies.

References

United States Renal Data System. USRDS 2022Annual Data Report: Epidemiology of kidney disease in the United States. Bethesda, MD: National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases. Available fromhttps://adr.usrds.org/2022.

United States Renal Data System. USRDS 2020 Annual Data Report: Epidemiology of Kidney Disease and in the United States. National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases, Bethesda, MD. [cited 2020]: Available from https://adr.usrds.org/2020.

Kalantar-Zadeh K, Kovesdy CP, Streja E, Rhee MC, Soohoo M, Chen JLT, Molnar MZ, Gillen D, Nguyen DV, Norris KC, Sim JJ, Jacobsen SS Transition of care from pre-dialysis prelude to renal replacement therapy: the blueprints of emerging research in advanced chronic kidney disease. Nephrol Dial Transplant 2017; 32(suppl_2): ii91-ii98. https://doi.org/10.1093/ndt/gfw357 DOI: https://doi.org/10.1093/ndt/gfw357

United States Renal Data System. USRDS 2015 Annual Data Report: Epidemiology of Kidney Disease in the United States.National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases, Bethesda, MD.

Soohoo M, Streja E, Obi Y, Rhee CM, Gillen DL, Sumida K, Nguyen DV, Kovesdy CP, Kalantar-Zadeh K. Predialysis kidney function and its rate of decline predict mortality and hospitalization after starting dialysis. Mayo Clinic Proceedings 2018; 93(8): 1074-1085. https://doi.org/10.1016/j.mayocp.2018.01.030 DOI: https://doi.org/10.1016/j.mayocp.2018.01.030

Foley RN, Chen SC, Solid CA, Gilbertson DT, Collins AJ. Early mortality in patients starting dialysis appears to go unregistered. Kidney Int 2014; 86: 392-398. https://doi.org/10.1038/ki.2014.15 DOI: https://doi.org/10.1038/ki.2014.15

Lukowsky LR, Kheifets L, Arah OA, Nissenson AR, Kalantar-Zadeh K. Patterns and predictors of early mortality in incident hemodialysis patients: new insights. Am J Nephrol 2012; 35: 548-558. https://doi.org/10.1159/000338673 DOI: https://doi.org/10.1159/000338673

Robinson BM, Zhang J, Morgenstern H, Bradbury BD, Ng LJ McCullough KP, Gillespie BW, Hakim R, Rayner H, Fort J, Akizawa T, Tentori F, Pisoni RL. Worldwide, mortality risk is high soon after initiation of hemodialysis. Kidney Int 2014; 85: 158-165. https://doi.org/10.1038/ki.2013.252 DOI: https://doi.org/10.1038/ki.2013.252

United States Renal Data System. USRDS 2018 Annual Data Report: Epidemiology of Kidney Disease in the United States.National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases, Bethesda, MD.

Nissenson AR, Fine RN, Mehrotra R, Zaritsky J (Ed.) 2023. Handbook of Dialysis Therapy, 6th edition. Elsevier.

CMS. CMS 2016 Quality Strategy Overview. Available fromhttps://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityInitiativesGenInfo/Downloads/CMS-2016-Quality-Strategy-Slides.pdf

Codman E. Hospitalization standardization. Surgery, Gynecology, and Obsterrics1916; 22: 119-120.

Keenan PS, Normand SL, Lin Z, Drye EE, et al. An administrative claims measure suitable for profiling hospital performance on the basis of 30-day all-cause readmission rates among patients with heart failure. Circulation Cardiovascular Quality and Outcomes 2008; 1: 29-37. https://doi.org/10.1161/CIRCOUTCOMES.108.802686 DOI: https://doi.org/10.1161/CIRCOUTCOMES.108.802686

Krumholz HM, Lin Z, Drye EE, Desai MM, Han HF, Rapp MT, Mattera JA, and Normand S-L. An administrative claims measure suitable for profiling hospital performance based on 30-day all-cause readmission rates among patients with acute myocardial infarction. Circulation Cardiovascular Quality and Outcomes 2011; 4: 243-252. https://doi.org/10.1161/CIRCOUTCOMES.110.957498 DOI: https://doi.org/10.1161/CIRCOUTCOMES.110.957498

Lindenauer PK, Normand SL, Drye EE, Lin Z, Goodrich K, Desai MM, Bratzler DW, O’Donnell WJ, Metersky ML, Krumholz HM. Development, validation, and results of a measure of 30-day readmission following hospitalization for pneumonia. Journal of Hospital Medicine 2011; 6: 142-150. https://doi.org/10.1002/jhm.890 DOI: https://doi.org/10.1002/jhm.890

Horwitz L, Partovain C, Lin ZQ, Herrin, et al. 2011. Hospital-wide (all-condition) 30 day risk- standardized readmission measure. https://www.cms.gov/Medicare/ Quality-Initiatives-Patient-Assessment-Instruments/MMS/downloads/MMSHospital-WideAll-ConditionReadmissionRate.pdf Accessed August 25, 2023.

Horwitz L, Partovain C, Lin ZQ, Grady, et al. Development and use of an administrative claims measure for profiling hospital-wide performance on 30-day unplanned readmission. Annals of Internal Medicine 2014; 161: S66-75. https://doi.org/10.7326/M13-3000 DOI: https://doi.org/10.7326/M13-3000

Ross JS, Normand SL, Wang Y, Ko DT, J, C, Drye EE, Keenan PS, Lichtman JH, Bueno H, Schreiner GC, and Krumholz HM. Hospital volume and 30-day mortality for three common medical conditions. New England Journal of Medicine 2010; 362: 1110-1118. https://doi.org/10.1056/NEJMsa0907130 DOI: https://doi.org/10.1056/NEJMsa0907130

Normand ST, Glickman ME, Gatsonis CA. Statistical methods for profiling providers of medical care: Issues and applications. Journal of the American Statistical Association 1997; 92: 803-814. https://doi.org/10.1080/01621459.1997.10474036 DOI: https://doi.org/10.1080/01621459.1997.10474036

Normand ST, Shahian DM. Statistical and clinical aspects of hospital outcomes profiling. Statistical Science 2007; 22: 206-226. https://doi.org/10.1214/088342307000000096 DOI: https://doi.org/10.1214/088342307000000096

Ohlssen DI, Sharples LD, Spiegelhalter DJ. A hierarchical modelling framework for identifying unusual performance in health care providers. J R Statist Soc A 2007; 170: 865-890. https://doi.org/10.1111/j.1467-985X.2007.00487.x DOI: https://doi.org/10.1111/j.1467-985X.2007.00487.x

Jones HE, Spiegelhalter DJ. The identification of ‘unusual’ health-care providers from a hierarchical model. American Statistician 2011; 65: 154-163. https://doi.org/10.1198/tast.2011.10190 DOI: https://doi.org/10.1198/tast.2011.10190

Goldstein H, Spiegelhalter DJ. League tables and their limitations: Statistical issues in comparisons of institutional performance. J Roy Statis Soc A 1996; 159: 385-443. https://doi.org/10.2307/2983325 DOI: https://doi.org/10.2307/2983325

Ash AS, Fienberg SE, Louis TA, Normand ST, Stukel TA, Utts J Ash AS, Fienberg SE, Louis TA, Normand ST, Stukel TA, Utts J. Statistical issues in assessing hospital performance. The COPSS-CMS White Paper Committee, CMS, Washington D.C. [cited 2012]: Available from https://www.cms.gov/medicare/quality-initiatives-patient-assessment-instruments/hospitalqualityinits /downloads/statistical-issues-in-assessing-hospital-performance.pdf.Accessed August 28, 2023.

McGee G, Schildcrout J, Normand S-L, Haneuse S. Outcome-dependent sampling in custer-correlated data settings with application to hospital profiling. J R Statis Soc A 2020; 183: 379-402. https://doi.org/10.1111/rssa.12503 DOI: https://doi.org/10.1111/rssa.12503

Paddock SM, Ridgeway G, Lin R, Louis T. Flexible distributions for triple-goal estimates in two-stage hierarchical models. Computational Statistics and Data Analysis 2006; 50: 3243-3262. https://doi.org/10.1016/j.csda.2005.05.008 DOI: https://doi.org/10.1016/j.csda.2005.05.008

Silber JH, Satopaa VA, Mukherjee N, Rockova V, Wang W, Hill AS, EvenShoshan O, Rosenbaum PR, George EI (2016) Improving Medicare’s Hospital Compare mortality model. Health Services Research 2016; 51: 1229-1247. https://doi.org/10.1111/1475-6773.12478 DOI: https://doi.org/10.1111/1475-6773.12478

George EI, Rockova V, Rosenbaum PR, Satopaa VA, Silber JH (2017) Mortality rate estimation and standardization for public reporting: Medicare’s Hospital Compare. Journal of the American Statistical Association 2017; 112: 933-947. https://doi.org/10.1080/01621459.2016.1276021 DOI: https://doi.org/10.1080/01621459.2016.1276021

CMS Hospital Compare. https://www.medicare.gov/care- compare/?providerType=Hospital. Accessed August 30, 2023.

Kalbfleisch JD, Wolfe RA. On monitoring outcomes of medical providers. Statistics in Biosciences 2013; 5: 286–302. https://doi.org/10.1007/s12561-013-9093-x DOI: https://doi.org/10.1007/s12561-013-9093-x

Chen Y, Senturk D, Estes JP, Campos LF, Rhee CM, Dalrymple LS, Kalantar-Zadeh K, Nguyen DV. Performance characteristics of profiling methods and the impact of inadequate case-mix adjustment. Communications in Statistics - Simulation and Computation 2021; 50: 1854–1871. https://doi.org/10.1080/03610918.2019.1595649 DOI: https://doi.org/10.1080/03610918.2019.1595649

Centers for Medicare & Medicaid Services (CMS)/UM-KECC. Report for the standardized readmission ratio. [cited 2017]: Available from https://www.cms.gov/Medicare/Quality- Initiatives-Patient-Assessment-Instruments/ESRDQIP/Downloads/SRR_Methodology_Report_June2017.pdf Accessed August 28, 2023.

Senturk D, Chen Y, Estes JP, Campos LF, Rhee CM, Kalantar-Zadeh K, Nguyen DV. Impact of case-mix measurement error on estimation and inference in profiling of health care providers. Communications in Statistics - Simulation and Computation 2020; 49: 2206-2224. https://doi.org/10.1080/03610918.2018.1515360 DOI: https://doi.org/10.1080/03610918.2018.1515360

Estes JP, Chen Y, Senturk D, Rhee CM, Kurum E, You AS, Streja E, Kalantar-Zadeh K, Nguyen DV. Profiling dialysis facilities for adverse recurrent events. Statistics in Medicine 2020; 39: 1374-1389. https://doi.org/10.1002/sim.8482 DOI: https://doi.org/10.1002/sim.8482

Estes JP, Nguyen DV, Chen Y, Dalrymple LS, Rhee CM, Kalantar-Zadeh K, Senturk D. Time-dynamic profiling with application to hospital readmission among patients on dialysis (with discussion). Biometrics 2018; 74: 1383-1394. https://doi.org/10.1111/biom.12908 DOI: https://doi.org/10.1111/biom.12908

Estes JP, Nguyen DV, Chen Y, Dalrymple LS, Rhee CM, Kalantar-Zadeh K, Senturk D. Rejoinder: Time-dynamic profiling with application to hospital readmission among patients on dialysis. Biometrics 2018b; 74: 1404-1406. https://doi.org/10.1111/biom.12905 DOI: https://doi.org/10.1111/biom.12905

Chen Y, Rhee CM, Senturk D, Kurum E, Campos LF, Li Y, Kalantar-Zadeh K, Nguyen DV. Association of U.S. dialysis facility staffing with profiling of hospital-wide 30-day unplanned readmission. Kidney Diseases 2019; 5: 153-162. https://doi.org/10.1159/000496147 DOI: https://doi.org/10.1159/000496147

He K, Kalbfleisch JD, Li Y, Li Y. Evaluating hospital readmission rates in dialysis facilities; adjusting for hospital effects. Lifetime Data Analysis 2013; 19: 490-512. https://doi.org/10.1007/s10985-013-9264-6 DOI: https://doi.org/10.1007/s10985-013-9264-6

Wu W, He K, Shi X, Schaubel DE, Kalbfleisch JD. Analysis of hospital readmissions with competing risks. Stat Methods Med Res 2022; 31(11): 2189-2200. https://doi.org/10.1177/09622802221115879 DOI: https://doi.org/10.1177/09622802221115879

Wu W, Yang Y, Kang J, He K. Improving large-scale estimation and inference for profiling health care providers. Stat Med 2022; 41(15): 2840-2853. https://doi.org/10.1002/sim.9387 DOI: https://doi.org/10.1002/sim.9387

Xia L, He K, Li Y, Kalbfleisch J. Accounting for total variation and robustness in profiling health care providers. Biostatistics 2022; 23(1): 257-273. https://doi.org/10.1093/biostatistics/kxaa024 DOI: https://doi.org/10.1093/biostatistics/kxaa024

He K, Dahlerus C, Xia L, Li Y, Kalbfleisch JD. The profile inter-unit reliability. Biometrics 2020; 76(2): 654-663. https://doi.org/10.1111/biom.13167 DOI: https://doi.org/10.1111/biom.13167

He K, Kalbfleisch JD, Yang Y, Fei Z. Inter-unit reliability for nonlinear models. Stat Med 2019; 38(5): 844-854. https://doi.org/10.1002/sim.8005 DOI: https://doi.org/10.1002/sim.8005

Kalbfleisch JD, He K, Xia L, et al. Does the inter-unit reliability (IUR) measure reliability? Health Serv Outcomes Res Method 2018; 18: 215-225. https://doi.org/10.1007/s10742-018-0185-4 DOI: https://doi.org/10.1007/s10742-018-0185-4

Estes JP, Senturk D, Kurum E, Rhee CM, Nguyen DV. Fixed effects high-dimensional profiling models in low information context. Int J Stat Med Res 2021; 10: 118-131. https://doi.org/10.6000/1929-6029.2021.10.11 DOI: https://doi.org/10.6000/1929-6029.2021.10.11

He, K. Indirect and direct standardization for evaluating transplant centers. Journal of Hospital Administration 2019, 8(1): 9-15. https://doi.org/10.5430/jha.v8n1p9 DOI: https://doi.org/10.5430/jha.v8n1p9

Kurland BF, Heagerty PJ. Directly parameterized regression conditioning on being alive: Analysis of longitudinal data truncated by deaths. Biostatistics 2005; 6: 241-258. https://doi.org/10.1093/biostatistics/kxi006 DOI: https://doi.org/10.1093/biostatistics/kxi006

Estes J, Nguyen DV, Dalrymple LS, Mu Y, Senturk D. Cardiovascular event risk dynamics over time in older patients on dialysis: A generalized multiple-index varying coefficient model approach. Biometrics 2014; 70: 754-764. https://doi.org/10.1111/biom.12176 DOI: https://doi.org/10.1111/biom.12176

Estes J, Nguyen DV, Dalrymple LS, Mu Y, Senturk D. Time-varying effect modeling with longitudinal data truncated by death: Conditional models, interpretations and inference. Statistics in Medicine 2015; 35(11): 1834-1847. https://doi.org/10.1002/sim.6836 DOI: https://doi.org/10.1002/sim.6836

Kurum E, Nguyen DV, Li Y, Rhee CM, Kalantar-Zadeh K, Senturk D. Multilevel joint models of hospitalization and survival in patients on dialysis, Stat 2021; 10: e356 (p. 1-13). https://doi.org/10.1002/sta4.356 DOI: https://doi.org/10.1002/sta4.356

Kurum E, Nguyen DV, Banerjee S, Li Y, Rhee CM, Senturk D. A Bayesian multilevel time-varying framework for joint modeling of hospitalization and survival in patients on dialysis. Statistics in Medicine 2022; 41(29): 5597-5611. https://doi.org/10.1002/sim.9582 DOI: https://doi.org/10.1002/sim.9582

Qian Q, Nguyen DV, Telesca D, Kurum E, Rhee CM, Banerjee S, Li Y, Senturk D. Multivariate spatiotemporal functional principal component analysis for modeling hospitalization and mortality rates in dialysis patients. Biostatistics 2023; in-press. https://doi.org/10.1093/biostatistics/kxad013 DOI: https://doi.org/10.1093/biostatistics/kxad013

Centers for Medicare & Medicaid Services (CMS) [cited 2014]. Report for the standardized readmission ratio. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/ESRDQIP/Downloads/MeasureMethodologyReportfortheProposedSRRMeasure.pdf Accessed September 1, 2023.

Rosenbaum PR, Rubin DB: The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika 1983; 70(1): 41-55. https://doi.org/10.1093/biomet/70.1.41 DOI: https://doi.org/10.1093/biomet/70.1.41

Rubin DB, Thomas N. Combining propensity score matching with additional adjustments for prognostic covariates. J Am Stat Assoc 2000; 95: 573-585. https://doi.org/10.1080/01621459.2000.10474233 DOI: https://doi.org/10.1080/01621459.2000.10474233

D’agostino RB Jr: Tutorial in biostatistics: Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med 1998; 17: 2265-2281. https://doi.org/10.1002/(SICI)1097-0258(19981015)17:19<2265::AID-SIM918>3.0.CO;2-B DOI: https://doi.org/10.1002/(SICI)1097-0258(19981015)17:19<2265::AID-SIM918>3.0.CO;2-B

Thomas-Hawkins C, Flynn L, Clarke SP: Relationships between registered nurse staffing, processes of nursing care, and nurse-reported patient outcomes in chronic hemodialysis units. Nephrol Nurs J 2008; 35: 123-130.

Foley RN, Hakim RM. Why is the mortality of dialysis patients in the United States much higher than the rest of the world? J Am Soc Nephrol 2009; 20(7): 1432-1435 https://doi.org/10.1681/ASN.2009030282 DOI: https://doi.org/10.1681/ASN.2009030282

Stone PW, Pogorzelska M, Kunches L, Hirschhorn LR. Hospital staffing and healthcare-associated infections: a systematic review of the literature. Clin Infect Dis 2008; 47: 937-944. https://doi.org/10.1086/591696 DOI: https://doi.org/10.1086/591696

Aiken LH, Clarke SP, Sloane DM, Sochalski J, Silber JH: Hospital nurse staffing and patient mortality, nurse burnout and job dissatisfaction. JAMA 2002; 288(16): 1987-1993. https://doi.org/10.1001/jama.288.16.1987 DOI: https://doi.org/10.1001/jama.288.16.1987

Stratton KM, Blegen MA, Pepper G, Vaughn T. Reporting of medication errors by pediatric nurses. J Pediatr Nurs 2004; 19(6): 385-392. https://doi.org/10.1016/j.pedn.2004.11.007 DOI: https://doi.org/10.1016/j.pedn.2004.11.007

Dunton N, Gajewski B, Taunton RL, Moore J. Nurse staffing and patient falls in acute care hospital units. Nurs Outlook 2004; 52(1): 53-59. https://doi.org/10.1016/j.outlook.2003.11.006 DOI: https://doi.org/10.1016/j.outlook.2003.11.006

Thomas-Hawkins C, Denno M, Currier H, Wick G. Staff nurses’ perception of the work environment in freestanding hemodialysis facilities. Nephrol Nurs J 2003; 30(2): 169-178. https://doi.org/10.1016/j.ajic.2010.10.017

Patrician PA, Pryor E, Fridman M, et al. Needlestick injuries among nursing staff: association with shift-level staffing. Am J Infect Control 2011; 39(6): 477-482. https://doi.org/10.1053/j.ajkd.2011.03.027 DOI: https://doi.org/10.1016/j.ajic.2010.10.017

Wolfe WA.Adequacy of dialysis clinic staffing and quality of care: a review of evidence and areas of needed research. Am J Kidney Dis 2011; 58(2)166: 176. DOI: https://doi.org/10.1053/j.ajkd.2011.03.027

Wolfe WA. Is it possible to reduce hospitalizations through evidence-based clinic staffing? Nephrology News & Issues 2016; Epub July 6. https://www.healio.com/nephrology/ practice-management/news/online/%7b0dd2aa31-8528-4f99-b266-34c0f30df3df%7d/is-it-possible-to-reduce-hospital-admissions-through-evidence-based-clinic-staffing. Accessed September3, 2018.

Chan KE, Lazarus JM, Wingard RL, and Hakim RM. Association between repeat hospitalization and early intervention in dialysis patients following hospital discharge. Kidney International 2009; 76: 331-341. https://doi.org/10.1038/ki.2009.199 DOI: https://doi.org/10.1038/ki.2009.199

Carroll RJ, Ruppert D, Stefanski LA, Crainiceanu CM. Measurement error in nonlinear models: A modern perspective. Boca Raton: Chapman and Hall/CRC 2006. https://doi.org/10.1201/9781420010138 DOI: https://doi.org/10.1201/9781420010138

Firth D. Bias reduction of maximum likelihood estimates. Biometrika 1993; 80: 27-38. https://doi.org/10.1093/biomet/80.1.27 DOI: https://doi.org/10.1093/biomet/80.1.27

Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Statistics in Medicine 2002; 21: 2409-2419. https://doi.org/10.1002/sim.1047 DOI: https://doi.org/10.1002/sim.1047

Aitkin M, Longford N. Statistical modelling issues in school effectiveness studies (with discussion). J R Statist Soc A 1986; 149: 1-42. https://doi.org/10.2307/2981882 DOI: https://doi.org/10.2307/2981882

Goldstein H, Thomas S. Using examination results as indicators of school and college performance. J R Statist Soc A 1996; 159: 149-163. https://doi.org/10.2307/2983475 DOI: https://doi.org/10.2307/2983475

New York State Department of Health. Adult Cardiac Surgery in New York State 1998–2000. Albany: New York State Department of Health 2004.

Marshall C, Best N, Bottle A, Aylin P. Statistical issues in the prospective monitoring of health outcomes across multiple units. J R Statist Soc A 2004; 167: 541-559. https://doi.org/10.1111/j.1467-985X.2004.apm10.x DOI: https://doi.org/10.1111/j.1467-985X.2004.apm10.x

SAS Institute Inc 2013. SAS/STAT® 13.1 User’s Guide. The GLIMMIX Procedure. Cary, NC: SAS Institute Inc. Available at https://support.sas.com/documentation/onlinedoc/ stat/131/glimmix.pdf

Bates D, Maechler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. Journal of Statistical Software 2015; 67(1): 1-48. https://doi.org/10.18637/jss.v067.i01 DOI: https://doi.org/10.18637/jss.v067.i01