What Are the Odds?
Updated: Apr 5
How We Know What A Diagnostic Test Means
Mary, a healthy 30-year-old woman, wakes up one morning with a stuffy nose and a scratchy throat. She has been very careful about wearing her high-quality, N95 mask in public places and has had three shots of an mRNA vaccine against the virus that causes Covid-19. Still, she knows that even vaccinated people can become infected, so she does a home Covid-19 test on herself and it is negative.
Mary has heard, however, that the rate of false negatives for the home tests, which detect the presence of a protein (also called an antigen) on the surface of the virus, is very high. She worries that despite the negative test she may actually be infected with SARS-CoV-2, the virus that causes Covid-19, so she seeks out a PCR test. The PCR, or polymerase chain reaction, test for the virus, she has heard, is more accurate but can take several days to get a result. She goes to a local pharmacy, gets the nasal swab test, and goes home to wait for the result. Her symptoms continue to worsen.
Three days later Mary gets a text telling her that her PCR test is negative. She is pretty sure this means that she does not have Covid-19, but she now has a bad cough, some chest pain, and a very sore throat. She decides to visit her primary care physician. Her doctor agrees that it is unlikely she is suffering from Covid-19, asks questions about her symptoms and her medical history, and performs a physical examination. Her lungs sound clear, and the doctor finds no abnormalities on the examination. Blood tests, a throat culture, and a chest X-ray to rule out pneumonia are ordered.
The next day, Mary gets a call from the doctor that her X-ray was normal but one of the blood tests of liver function the doctor ordered has yielded a slightly elevated level. Mary is perplexed by this—she has no symptoms other than what now appears to be an upper respiratory viral infection. She does not drink excessive amounts of alcohol and has had only one sexual partner for many years. Why should she have an abnormal liver function test? Could it be a false positive? Should she see a liver specialist and have more tests?
How to Interpret Laboratory Tests
This not uncommon situation raises the issue of how we interpret the myriad of laboratory and imaging tests doctors now have at their disposal to help make diagnoses. We would like to believe that the results of these tests have easy, cut-and-dry interpretations, either positive or negative for a disease. In fact, no laboratory test is perfect, false negatives and false positives occur with all of them, and even the decision about whether to order a test in the first place depends on many factors.
Diagnostic test results may give “positive” or “negative” results, or said to be “normal” or abnormal,” but their meaning is usually not so black and white. Several considerations are important in deciding whether to order laboratory tests in the first place and, if they are ordered, in understanding what the results mean (image: Shutterstock).
An article in The New York Times last January highlighted an extreme version of the difficulty sometimes encountered interpreting diagnostic tests. Titled “When they warn of rare disorders, these prenatal tests are usually wrong,” the article details the perils of tests performed on blood samples from pregnant women that are intended to reveal whether a fetus will be born with a genetic disorder. The analysis of these tests by The New York Times investigative reporters showed that “the positive results on those tests are incorrect about 85 percent of the time.” The article tells the harrowing story of people who were told on the basis of these blood tests that their unborn child would suffer from severe genetic abnormalities and is rightly critical of the industry that performs and promotes these tests. “The Times reviewed 17 patient and doctor brochures from eight of the testing companies,” the reporters wrote. “Ten of the brochures never mention that a false positive can happen. Only one mentioned how often each test gets positive results wrong.”
While it is obviously unacceptable to withhold information from prospective parents shelling out hundreds or thousands of dollars for these tests, which give false positive results most of the time, what The New York Times article did not explain is that none of this is at all a surprise to people who understand how diagnostic testing works. The accuracy of any diagnostic test depends not only on the accuracy of the test itself, on multiple other factors including how common the illness is in any population (that is, its prevalence) and something called “pre-test probability.”
Specificity and Sensitivity
The accuracy of any diagnostic test can be measured in several different ways, most basically with metrics called specificity and sensitivity. Specificity refers to the ability of a test to detect someone who does not have the disease in question. A highly specific diagnostic test will yield very few false positive results. The specificity of the PCR test for Covid-19 is very high, greater than 95%, so that if you have a positive test you are very likely to really be infected. The sensitivity of a test refers to its ability to detect someone who does have the disease. A highly sensitive diagnostic test will yield very few false negatives. The sensitivity of the PCR test for Covid-19 is also very high, around 90%, so that if your test is negative you most likely aren’t infected. Home Covid-19 tests are also quite specific, but not as sensitive so that Mary was correct that her negative test could have been a false negative and she did the right thing to seek a PCR test to be sure.
When testing for very rare diseases, however, as is the case with the prenatal tests discussed in The New York Times story, even very specific tests can yield many false positives. An article in ProPublica noted by Critica Chief Medical Officer Dr. David Scales gives a good explanation of why this is the case. It gives the example of a disease that is present in 4% of the population. If 1000 people are screened and the test is 100% sensitive (note that almost no real tests are this sensitive), it will accurately detect all 40 people who truly have the disease. Let’s say that the test is 95% specific. That means that 5% of the remaining 960 people who don’t actually have the disease—48 people—will nevertheless test positive. The number of false positives thus exceeds the number of true positives in this example, even with a test with very high specificity and sensitivity. The genetic syndromes The New York Times article was concerned about are much less common than even 4%--occurring in some cases at rates of around 0.05% of the population or even less. It is obvious, then, that no matter how accurate the diagnostic test is, there will be more false positives than true positives with a screening test for diseases this rare. A negative result on one of these prenatal tests may be reassuring, but a positive test is hard to interpret and obviously will cause a huge amount of anxiety in the prospective parents, who will have to wrestle with decisions such as whether to continue the pregnancy, undergo more invasive diagnostic tests, or even just ignore the blood test result. Testing companies should be much more upfront informing people about the real-life accuracy of these prenatal tests, but no one should be surprised that positive results will usually be false.
One thing that can help sharpen diagnostic decision making is to consider the pre-test probability that a person has an illness. Before ordering any test, it is important to consider two things: how likely is it that the person has the illness in question and will the test result make any difference in the patient’s life. The first of these considerations is the pre-test probability. In Mary’s case, there was a reasonable pre-test probability that she had Covid-19 because she had typical symptoms of the illness: nasal congestion, cough, and sore throat. This makes her negative result on the home Covid-19 test more reassuring, because false negatives are more common in asymptomatic people.
On the other hand, Mary’s history and physical examination results make it very unlikely that she has liver disease. Were it not for the fact that she came to the doctor because of nasal stuffiness and cough, no blood tests would ever have been ordered. She has no risk factors for liver disease and on examination her liver was not enlarged. Many physicians would not have ordered liver function tests in this case in the first place.
Now faced with an abnormal laboratory result, Mary and her doctor can apply an algorithm that takes into account the pre-test probability of liver disease and the accuracy of the laboratory test. We won’t go through the mathematics of that here but suffice it to say that applying the algorithm yields a very low percentage likelihood that Mary actually has any problem with her liver. The post-test odds of disease, then, are very low and Mary and her doctor conclude the test result is most likely a false positive (the magnitude of the abnormality of the liver test, which was small, is also a factor here that we haven’t discussed). They decide to ignore it and repeat the test in a few months just to be sure. Other patients and doctors may choose a different course of action, but the important thing is that once again there is no black and white about the blood test result. A “positive” test result in this case does not mean that Mary has liver disease.
It was not that long ago that doctors had few diagnostic tests at their disposal and had to make diagnoses based mainly on history, physical examination, and intuition. There is no question that more sophisticated laboratory methods and new diagnostic tests have improved the accuracy of diagnoses and patient outcomes. But there is also recognition that many tests are ordered unnecessarily. Before a test is ordered, some considerations are:
1. How accurate is the test—what is its sensitivity and specificity? How likely is a false negative or false positive result?
2. What is the pre-test probability that an illness is actually present? What effect would a positive test result have on the post-test odds that an illness is present?
3. What difference will the test result make? Will it alter management of an illness, provide reassurance, change plans?
Before you ask your doctor to order a blood test or an MRI scan, then, ask them to discuss those three considerations with you. It might be that you and the doctor will decide that the test is unnecessary. At the very least, you’ll be in a better position to understand what the test results mean if you do go ahead with it. Had Mary known all of this, she might have asked her doctor what all the blood tests they ordered were, what they were for, and why they were being ordered. Perhaps, with that information, the liver test would never have been done at all.