Accuracy of Point-of-Care Antigen Vs Molecular-Based Tests for Diagnosis of SARS-Cov-2 Infection. A Multicentre Study in the Emergency Department

Abstract

To prospectively evaluate the accuracy of a Point-Of-Care (POC) antigen test for SARS-CoV-2 in the Emergency Department (ED). 

From 8 November 2020 to 27 January 2021, a convenience sample of adult patients presenting to four EDs of the central area of Tuscany was considered. Exclusion criteria were recent diagnosis of COVID-19 and known positivity for SARS-CoV-2. Nasopharyngeal swabs were obtained from all included patients for both POC fluorescent immunoassay (AFIAS-1) antigen test and laboratory-based nucleic acid amplification test as the reference standard. 

We included 1165 patients, among whom 583 (50%) were females. The sensitivity and specificity of the POC antigen test were 59.9% (95% CI: 55.1-63.7%) and 97.9% (97-98.6%), respectively, with a Negative Predictive Value (NPV) of 92.9% (92.1-93.6%) and a Positive Predictive Value (PPV) of 83.8% (77.1-89.1%). In patients without clinical suspicion of COVID-19 (630, 54%), the NPV of the POC antigen test was 98.2% (97.5-98.8%). In patients suspected for COVID-19 (535, 46%), the PPV of the POC antigen test was 89.8% (83.1-94.4%). In this group, when the cut-off was elevated from >/= 1 to >/= 4, the PPV increased to 98.7% (93.8-100%), with an absolute increase of +8.9% (95% CI: 4.1-17%). 

In the overall ED population, the POC antigen test did not exclude or identify SARS-CoV-2 infection with acceptable accuracy. When combined with clinical presentation, i.e. using different cut-offs for suspected and not-suspected patients, the POC antigen test could identify in suspected or exclude in not-suspected patients SARS-CoV-2 infection with high precision.

Keywords

COVID-19; Diagnosis; Emergency Department; Point-of-care tests

Introduction

Severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) presents important diagnostic challenges in the emergency setting. Several diagnostic strategies are available; however, Point-Of-Care (POC) antigen tests have the potential to enable earlier detection and isolation of cases than laboratory-based diagnostic methods. 

A recent Cochrane systematic review [1] and the last interim guidance of the World Health Organization (WHO) [2] highlighted that data on antigen performance in the clinical setting are still limited, thus encouraging paired Nucleic Acid Amplification Tests (NAATs) and antigen validations in representative field studies. 

This prospective study aimed to evaluate the diagnostic accuracy of an antigen POC rapid test to detect SARS-COV-2, using NAAT as a reference standard, both in the whole Emergency Department (ED) population and in COVID-19 suspected or not-suspected patients. We also aimed to verify whether a different Cut-Off Index (COI) should be used in suspected and not-suspected patients.

Methods

Design and setting 

This prospective study was conducted in the EDs of four community (not academic), level II trauma centre hospitals (Empoli, Firenze, Prato, Borgo San Lorenzo) of the central area of Tuscany. 

Participants 

From 8 November 2020 to 27 January 2021, a convenience sample of adult patients triaged by trained nurses and classified as suspected or not-suspected for COVID-19 according to the WHO standard clinical criteria  [3] were considered potentially eligible. According to the WHO criteria, COVID-19 is suspected when a patient has at least one of the following symptoms: A) presence of fever above 37.5°C and cough or B) acute onset of any three or more of the following signs or symptoms: fever, cough, general weakness/fatigue, headache, myalgia, sore throat, coryza, dyspnoea, anorexia/nausea/vomiting, diarrhoea, or altered mental status with or without epidemiological criteria, or C) patients with recent onset anosmia or ageusia in the absence of any other identified cause. Patients who needed to be admitted to the hospital wards or who required a SARS-CoV-2 test for other clinical reasons were considered for the study. Not-suspected patients underwent testing because according to internal policy, a patient needs to be screened for SARS-CoV2 to be admitted to any hospital ward. Exclusion criteria were a confirmed diagnosis of COVID-19 or known SARS-CoV-2 positivity within the last two months or refusal to undergo SARS-CoV-2 test or participate in the study. During the initial medical evaluation, the included patients underwent two combined oropharyngeal and nasopharyngeal swabs: one for the POC antigen test and one for the laboratory-based NAAT as the reference standard. The study was approved by the Institutional Review Board of the Emergency Department of the Azienda USL Toscana Centro (n° 05220), and written informed consent was obtained from all study subjects. 

Patient and public involvement 

No patient involved. 

Index test 

The index test was performed using an automated fluorescent immunoassay system AFIAS-1 (Boditech Med Inc, Korea) with a time-resolved fluorescent lateral flow assay against a specific antigen of the nucleoprotein of SARS-CoV-2, a semi-quantitative method in which a result expressed in a COI from 0 to >100 was obtained after 12 minutes. The manufacturer suggests using >/= 1 COI for positivity, with a reported sensitivity of 87.5% and a specificity of 96.5%  [4]. For testing, eight drops (approximately 180 mL) of nasopharyngeal specimens in virus transport medium with an extraction buffer was dispensed into the specimen well cartridge containing the test strip. After loading the cartridge into the AFIAS-1 system, all procedures were conducted automatically. The test was performed by the attending nurse according to the manufacturer’s instructions after a training of 1 hour and after performing 3 proctored tests. The reader found the COI on the AFIAS-1 display. To measure the Limit Of Detection (LOD), a cultured and heat-inactivated (56°C for 30 minutes) wild-type SARS-CoV-2 at 2*10⁶ Median Tissue Culture Infectious Dose (TCID₅₀) stock was 10-fold serially diluted in 450 µl of AFIAS-1 or UTM® buffer (Copan, IT) and tested with AFIAS-1 or SARS-CoV-2 ELITe MGB® RT-PCR Kit (ELITechGroup, FR), respectively. The latter system provides a cycle threshold (Ct) value and a quantitative result expressed in copies/ml of SARS-CoV-2. 

The clinicians were aware of the AFIAS-1 results, but the test was run only for clinical research purposes and did not influence clinical decisions. 

Reference test 

Molecular tests, e.g. PCR and NAAT, are the most accurate tests for detecting SARS-CoV-2 infection. NAATs performed in this study were different among different hospitals (InGenius®, ELITech Group, FR; AllplexTM® SARS-COV-2 Assay, Seegene Inc, KO; Genexpert®, Cepheid, USA), and their performance was under continuous surveillance by the referral regional laboratory (affiliation 5, University of Florence). NAAT was positive when at least one target gene was amplified in less than 36 cycles, independent of the assay used. When the antigen test was positive and the first NAAT was negative, a second NAAT was performed to exclude SARS-CoV-2 infection. When the antigen test was negative and the NAAT was positive, the case was diagnosed as positive for SARS-CoV-2. The result of the index test was always ready before that of NAAT and was not available to the laboratory personnel. 

Statistical analysis 

To determine the diagnostic accuracy of the index test, we initially planned to use the manufacturer’s suggested COI. Accuracy was the sum of true positive and true negative tests divided by the number of patients included. Patients with indeterminate or unavailable tests were excluded from the study (see flow diagram). Sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively) were calculated for the whole study population. 

To obtain a narrow (<3%) 95% Confidence Interval (CI) of overall accuracy, considering a mean prevalence of the target condition of 10%, we planned to include at least 500 patients in each group (suspected and not-suspected for COVID-19). 

Because we expected different levels of prevalence of SARS-CoV-2 infection and thus different accuracy, PPV, and NPV of the index test in suspected and not-suspected patients, we also planned to calculate accuracy, sensitivity, specificity, PPV, and NPP in suspected and not-suspected patients separately and to assess the usefulness of different COIs in these two groups. Different COIs were chosen after performing a ROC curve analysis if a sufficiently high (>98%) and statistically significant increase in PPV or NPV was noted compared to the COI suggested by the manufacturer. Continuous variables between the groups were compared using t-test. Categorical variables were compared using the X²-test. Statistical significance was set at p </= 0.05.

Results

Among 1302 potentially eligible patients, 1165 (89.5%) were included in the analysis (Figure 1); 583 (50%) were females, with a mean age of 69.8 years (interquartile range 56-85 years). We found no differences in age or sex distribution between suspected and not-suspected patients. No adverse events were registered in the included population according to the index and reference tests.

Using a preparation of SARS-CoV-2 of known concentration, the LOD of AFIAS-1 was estimated to be 20 TCID₅₀ (approximately 7.6 × 10³ copies/ml of SARS-CoV-2) when using COI  >/= 1, corresponding to a Ct of 30.

The prevalence of true positives according to the reference standard (NAAT) was 15.6%. The overall accuracy of the index test was 91.9%, with a sensitivity of 59.9% and a specificity of 97.9% (Table 1).

Patients not suspected for COVID-19 accounted for 54% of the overall study population, with a prevalence of the target condition of 3.7%. In these patients, the accuracy of the index test was 96.7%, with a NPV of 98.2% (Table 1). 

The prevalence of the target condition was significantly higher in patients suspected for COVID-19, (+26%, 95% CI: 22.3-29.7) than in not-suspected patients (Table 1). The diagnostic accuracy of the index test was 86.4%, with a PPV of 89.8% (Table 1 and Figure 2). If we increased the COI for positivity from >/= 1 to >/= 4, PPV significantly increased from 89.8% to 98.7% (+ 8.9%; 95% CI: 4.1-17%).

Discussion

POC systems provide a potentially attractive route to reduce testing time, but their potential role in patient care also depends on their accuracy, particularly with respect to the clinical field in which they are used. In the ED, a POC test is expected to be rapid but also highly accurate because especially for contagious and potentially severe diseases such as COVID-19, an apparently small rate of false-positive or false-negative results could result in rapid diffusion and affect prognosis. For the first time, we reported the diagnostic performance of a POC fluorescent immunoassay in a large population of patients with suspected and not-suspected COVID-19. We found a sensitivity of approximately 60% and specificity higher than 90%. As a study limitation, we acknowledge that we only repeated two NAATs when antigen test was negative and NAAT was positive; this could potentially have led to a slight increase in the apparent specificity of the antigen test, while ensuring that sensitivity did not decrease by repeating NAAT. However, a Cochrane review compared eight different POC antigen test, showing a similar average sensitivity of 56% [5]. More recently, studies performed in different clinical fields suggested that the sensitivity of the POC antigen test was higher in symptomatic subjects than in asymptomatic subjects [6-8]. A possible explanation is that the SARS-CoV-2 RNA load was putatively higher in POC-positive/NAAT-positive specimens than in POC-negative/NAAT-positive samples [8]. In particular, POC antigen tests showed adequate sensitivity for nasopharyngeal swabs with lower Cts, i.e. 92% with Cts </= 29 vs 55% with >29 Cts [9]. Accordingly, our quantitative analysis showed that AFIAS-1 could detect the presence of SARS-CoV-2 at 20 TCID₅₀, corresponding to a Ct of 30. These results suggest that samples with a low viral load (Ct>30) could remain undetected. 

There is currently no consensus regarding the most effective strategy to diagnose and stop the spread of COVID-19 in the ED [1,2,9,10]. Based on our findings, we propose a diagnostic algorithm that leverages both types of tests. We suggest the preferential use of the rapid and inexpensive POC antigen test, if in accordance with the clinical criteria (Figure 3, grey boxes in the last lines), and reservation of the highly sensitive and time-consuming NAAT for when clinical signs and POC antigen test results are discordant (Figure 3, white boxes in the last lines).

Conclusion

By combining simple clinical criteria with the POC antigen test results, we could rapidly and accurately screen for SARS-CoV-2 in most patients (60%) in the ED.

Figures

Tables

References

1. Dinnes J, Deeks JJ, Adriano A, Berhane S, Davenport C, et al. (2020) Cochrane COVID-19 Diagnostic Test Accuracy Group. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev 8: CD013705.
2. https://apps.who.int/iris/handle/10665/334254
3. https://www.who.int/publications/i/item/WHO-2019-nCoV-Surveillance_Case_Definition-2020.2
4. https://www.afias.eu/wp-content/uploads/2020/12/Brochure-Rev00_AFIAS-COVID-19-Ag-test_2p_20200608_ss1.pdf
5. Mina MJ, Parker R, Larremore DB (2020) Rethinking covid-19 test sensitivity- a strategy for containment. N Engl J Med 383: e120.
6. Albert E, Torres I, Bueno F, Huntley D, Molla E, et al. (2020) Field evaluation of a rapid antigen test (Panbio COVID-19 Ag Rapid Test Device) for COVID-19 diagnosis in primary healthcare centres. Clin Microbiol Infect. 27: e7-e10.
7. Torres I, Poujois S, Albert E, Álvarez G, Colomina J, et al. (2021) Point-of-care evaluation of a rapid antigen test (CLINITEST-Rapid COVID-19 Antigen Test) for diagnosis of SARS-CoV-2 infection in symptomatic and asymptomatic individuals. J Infect 82: e11-e12.
8. Hayer J, Kapasic D, Zemmrich C (2020) Real-world clinical performance of commercial SARS-CoV-2 rapid antigen tests in suspected COVID-19: A systematic meta-analysis of available data as per November 20, 2020. medRxiv 108: 592-602.
9. Nalumansi A, Lutalo T, Kayiwa J, Watera C, Balinandi S, et al. (2020) Field evaluation of the performance of a SARS-CoV-2 antigen rapid diagnostic test in Uganda using nasopharyngeal samples. Int J Infect Dis 104: 282-286.
10. Krüger LJ, Gaeddert M, Köppel L, Brümmer LE, Gottschalk C, et al. (2020) Evaluation of the accuracy, ease of use and limit of detection of novel, rapid, antigen-detecting-point-of-care diagnostics for SARS-CoV-2. medRxiv