Validation of the antifungal national antimicrobial prescribing survey (AF-NAPS) quality assessment tools.

Abstract

Background: The AF-NAPS was developed to undertake streamlined quality audits of antifungal prescribing. The validity and reliability of such tools is not well characterised. Aim: To assess the validity and reliability of the AF-NAPS quality assessment tools for the classification of antifungal prescribing appropriateness. Methods: Case vignettes describing antifungal prescribing were prepared in four domains: haematology, intensive care, solid organ transplant and paediatrics. A steering group was assembled to determine gold standard appropriateness classifications. Infectious diseases physicians, antimicrobial stewardship (AMS) and specialist pharmacists were recruited to undertake a survey to classify appropriateness of antifungal prescribing utilising AF NAPS tools. Validity was measured as accuracy, sensitivity, and specificity compared to gold standard. Inter-rater reliability was measured using Fleiss’ Kappa statistics. Assessors’ responses and comments were thematically analysed to determine reasons for incorrect classification. Results: Twenty-eight clinicians assessed 59 antifungal prescriptions. Accuracy of appropriateness assessment by all assessors was 77.0% (95% CI 73.0% - 80.0%) with a sensitivity of 85.3% (95% CI 81.1% - 88.8%) and specificity of 68.8% (95% CI 62.7% - 73.0%). Highest accuracy was seen amongst specialist and AMS pharmacists (81% and 79% respectively). Prescriptions with lowest accuracy were in the haematology setting (69%), echinocandins (73%), mould active azoles (75%) and for prophylaxis (71%). Inter-rater reliability was fair (0.3906), with moderate reliability amongst specialist pharmacists (0.5304). Key barriers to accurate classification were incorrect use of the AF NAPS assessment matrix, knowledge gaps, lack of guidelines and familiarity. Discussion: The AF-NAPS tool provides high accuracy, with assessors able to correctly classify appropriate prescriptions (high sensitivity) and less commonly able to classify inappropriate prescriptions (low specificity). Specialist and AMS pharmacists had similar performance, providing support for AMS pharmacists to perform the AF-NAPS despite lower confidence in speciality areas. Identified reasons for incorrect classification will be targeted in AF NAPS online tool and educational materials.