Primum Non Nocere: Is the Performance of Available SARS-CoV-2 RNA and Immunoassays Good Enough To Do No Harm?

We have appreciated the conversation of the last few days amongst the AACC Artery community. So many of you struggle to provide the COVID-19 testing that your constituencies desire and need. We have been on calls with several public health officials who tell us many of the problems you all have described.

The diagnostics industry is running as fast as they can to provide the systems and tests needed, but the pace is unprecedented. RNA and immunoassay tests began appearing on the market within 45 days of the announcement of the outbreak in Wuhan. At that speed, all of the tests can’t possibly perform as well as needed.  It appears that some of the immunoassays had been developed for SARS (SARS-CoV) and were simply relabeled for COVID-19 (SARS-CoV-2). Yet, the pressure from many sources to implement testing is almost unbearable, so tests that might not be good enough are used.

The reports of false negative results for rt-PCR assays are concerning. There have been reports of test results fluctuating between positive and negative over several days in one patient. Although under vigorous investigation, it will take some time to flesh out whether the false negative problem is due to poor sampling (training for nasopharyngeal swab collection, repurposed swabs), test design (gene selection, specific sequences employed, one vs multiple targets), processing errors and/or other issues such as changing viral reservoirs.  And what is the Gold Standard by which we judge test results? This is most concerning where the test is used as a diagnostic and a false negative result can lead to release of an infected and infectious person into an environment where they can transmit the virus to others. Even in New York, most people that are tested are negative for RNA. How many false negatives are slipping through the cracks? Like in other outbreaks, time will tell. But, are we careful enough with the release of new tests that communities are trusting to make critical decisions about quarantine and exposure of healthcare staff without appropriate PPE? Should we have regional or national validation programs to verify the test performance? South Korea and others have done just that. Is this happening but we don’t hear about it?

The thing that concerns us most right now about COVID-19 testing is serology assays. This is particularly concerning given the use cases under consideration. We all want to see our regions and countries return to some level of normalcy. We can tolerate shelter in place only so long psychologically and economically. One of the popular discussions of serology use cases is the return of individuals to the community at large based on the results of tests for IgM and IgG. The presence of IgM would suggest a recent infection (actual recency to be determined), but IgG alone would suggest an exposure some weeks or months ago. An IgG only positive test might suggest that an individual has at least temporary protective immunity. We will not get into the issues of IgG viral antigen specificity, subclasses, titers, neutralizing antibodies, etc. that might be required to prove protective immunity, but let’s pretend for now that such a test would do what is intended. What we’d like to address is the clinical performance ramifications.

In most situations for the foreseeable future, most people in a community will not have been exposed to SARS-CoV-2, even if we are far off on the estimate of total infected people due to low testing numbers. If we look at Wuhan as an example, less than 1% of the city’s population was infected. Let’s pretend that estimates were off by at least 5-fold and it was actually 5%. In that circumstance, if we were to test a population with a serology test that had 99% specificity, we would expect 5 true positive samples and 1 false positive out of 100 people tested. That would mean that one person out of 6 cleared to return to work would not have been exposed and is at substantial risk. Authorities might find this acceptable. But, some of the immunoassays that we are seeing have substantially poorer performance than 99% specificity, many around 90%. In that case, for every 1 true positive in our Wuhan example, we would find 2 false positives. That is a positive predictive value of 33%. As of Friday April 10, there have been 161,000 COVID-19 cases in NYC with a total of 18,804,000 people living in the five boroughs at the beginning of 2020, which is just under 1% of the population infected (again, probably a low estimate). Of course, we haven’t seen the end of infections in NYC, but most other regions in the US have had far fewer cases per capita. In California there have been ~20,000 cases in a population of ~40,000,00, so ~0.05% of the population infected to date. At 0.05% prevalence, the prospect of useful results from a test with even 99% specificity is problematic at best (~5% positive predictive value). Are we really going tell people they can go back to work with results like these? We have not addressed the impact of false negative results, but these deserve careful consideration as well.

We only addressed IgG testing of the population at large, where there is a low pre-test probability of a positive result. The results can be greatly improved by testing only people known to have been infected through RNA tests, but that would eliminate the testing of persons that have probably been infected, survived but never had an RNA test. That is probably the majority of previously infected people. Traditionally for other viral infections, blood-screening labs have employed re-testing algorithms or independent confirmation tests. Perhaps other testing algorithms have been considered as part of the back to work testing schemes. We have considered a few options. Let’s think this through. Primum non nocere?

You can find our discussion of use cases for COVID-19 and the implications for product performance at https://halteresassociates.com/covid-19-diagnostic-use-cases/  and https://halteresassociates.com/halteres-sars-cov-2-use-case-tables/

Respectfully,

Mickey Urdea and Rich Thayer, Halteres Associates, April 14, 2020