Wondering How the CDC Collects Breakthrough Data? Get in Line

On November 11, 2021, MedPage Today reported on the CDC's lag in updating breakthrough infection data. As part of our review of the year's top stories, we follow up on what the agency has changed since our initial report -- and why their methods are still so confusing.

In mid-October, the CDC created a tracker for rates of COVID-19 cases and deaths by vaccination status, after announcing in May that it would only focus on collecting data from states on severe breakthrough cases, referring to those that resulted in hospitalization or death.

The need for clear and timely data on breakthrough cases has become ever more evident, but so have the challenges the CDC faces in compiling and presenting that data.

"There are a lot of things to say about what's going on with CDC data beyond the fact that it's late," said Jessica Malaty Rivera, MS, an infectious disease epidemiologist who led science communications for the COVID Tracking Project and is now working with the Rockefeller Center to track Omicron and breakthrough cases. "It's the type of granularity that has kind of always been missing, and that we needed."

What Has Changed

Since MedPage Today reported an almost 10-week lag in the CDC data, the agency has made a number of changes to its tracker page. Case and death data were updated a few times: Although the date changes were made isn't listed, by late November, data was available through Oct. 2 instead of Sept. 4; and by Dec. 18, they had added cases through Nov. 20 and deaths through Oct. 30.

More states have also been added since October. On Dec. 18, the CDC changed the tracker page to state that 27 jurisdictions now are providing their data, covering "over half" of the country, which is up from 16 states, or "over a third" of the U.S. population.

However, Malaty Rivera was skeptical. "That's just not how the distribution of COVID-19 has ever gone. You cannot have representative data using only 27 jurisdictions," she said. "You just can't."

The agency added language about its methodology, noted that updates will be monthly, and explained that deaths are counted for the date of testing positive, not the date of death.

It also modified the tracker to reflect breakthrough infections after booster shots as Omicron takes hold. It notes that those with booster shots had the lowest case rates. This comes as the CDC faces new challenges in communicating the need for booster shots and confusion about whether vaccines actually work.

But Malaty Rivera said that, without being based on representative data, "it's not useful data."

CDC Response

MedPage Today reached out to five current and former CDC employees and members of the COVID-NET Team for more detail on the breakthrough infection data. None were able to comment.

A public affairs representative for the CDC simply referred MedPage to the tracking tool and to COVID-NET for breakthrough hospitalizations and provided information already available on agency websites.

But at a Dec. 10 press briefing, Serena Marshall with NowThis News, who also hosts the "Track the Vax" podcast from MedPage Today and Everyday Health, asked CDC Director Rochelle Walensky, MD, MPH, whether the agency planned on changing how it would be managing and tracking breakthrough cases, noting that there had been a push from the medical community to provide data in real time.

While Walensky did not answer the question, she explained that breakthrough cases are tracked by "passive reporting," (or voluntary reporting), acknowledging that this does not provide a full picture. She followed by saying that the CDC has been following "many different cohorts" to monitor breakthrough infections, including groups of healthcare workers, patients in long-term care facilities, and in healthcare systems like Kaiser Permanente and Intermountain Health.

These cohorts likely include a handful of networks that function essentially as long-term, observational studies, some with as few as eight to 11 sites (VISION, COVID-NET, IVY, NHSN, and HEROES/RECOVER).

"The major problem overall is basically selection bias," Stephen Morse, PhD, of Columbia University Medical Center in New York City, wrote in an email to MedPage Today. "Essentially all of these data come from hospitalized patients or those seeking medical attention (such as in the ER), so it will be biased towards the symptomatic individuals, but not necessarily in a systematic way."

Walensky said that the more than 20 public health departments providing data (likely referring to states) gives "a really accurate view" of breakthrough cases. But beyond the acknowledgement that 27 state health departments who, according to the CDC "regularly link their case surveillance to immunization information system data," are included in the tracking tool, much remains unclear.

The CDC hasn't made it easy to understand whether or not it has a single automated system to compile state breakthrough data or how it works.

"It's not like there's even going to be one person who can tell you all that within a public health department," said David Dowdy, MD, PhD, an epidemiologist at the Johns Hopkins School of Public Health in Baltimore. "There is no system is what I'm trying to say."

Guidance for Public Health Departments

Following the CDC's online trail provides some insight. Guidance to state health departments and other public health actors asks them to use multiple data collection entities. In a document from April 2021 (a more recent version is less detailed), the CDC clarified its May shift to only reporting breakthrough hospitalizations and deaths, rather than all breakthrough cases.

Complicating matters is an apparent shift in data collection altogether. The CDC explained in its guidance from April that it developed a national vaccine breakthrough database via REDCap, but "ultimately" will use the National Notifiable Diseases Surveillance System (NNDSS) instead. When the switch happens, state health departments would stop reporting cases into REDCap directly, submit to the NNDSS instead, and the CDC will enter the data into REDCap.

On the NNDSS website, a number of guidance pages and PowerPoint webinars do describe what seems to be a new way to report COVID-19 case data that is meant to "streamline case surveillance." One PowerPoint from November does mention breakthrough cases and REDCap.

The guide itself contains over 100 data elements that public health departments should include with each reported COVID-19 case. On their "onboarding map," four states are marked as having moved past onboarding and into "production," seemingly for using the new standards. On a page promoting onboarding speeds, the NNDSS office says it cut the onboarding process to 2.5 months.

Of the transition to NNDSS from REDCap, Malaty Rivera said she has many unanswered questions.

Remaining Challenges

Standardizing data from 50 different states is no easy feat. "I think that we have a more fragmented public health system than people realize," said Dowdy. "People want to think that there is this nationwide registry that all states kind of buy into, but each state has its own system."

The challenges are many. For example, though the CDC has a standard definition of a "breakthrough" case, it might not line up with each states' definition, Malaty Rivera said. Many states do record breakthrough data, whereas others may just send along immunization data and COVID-19 test data for the CDC to sort through.

Kansas, for example, is listed as a participating jurisdiction for the CDC's breakthrough tracker. However, Matthew Lara, of the Kansas Department of Health and Environment, told MedPage Today that they were not reporting breakthrough rates.

States that do track breakthroughs may do so at different time intervals. There are multiple components that the CDC needs to have: immunization records, hospitalization records, testing information, and death records to verify the information. And each state has different ways of collecting each of these elements.

"You're still dealing with very unstandardized data, and that's what's made this very difficult," said Malaty Rivera. "Without federal standards, you're basically looking at 50 disparate systems to describe one thing."

And public health departments may not all have the resources to maintain good systems, let alone the staff time to devote to standardizing data for the CDC. Some have extensive networks for hospitals to share electronic health records, and others don't. "They don't have the money to update their information systems on a regular basis. They don't have a lot of people. And suddenly they're being asked to do this coordination," said Dowdy.

According to the CDC, all states have electronic lab reporting, though not all have automatic sharing of electronic health records with public health agencies -- something the data modernization portion of the CARES Act seeks to address. Each state even has a slightly different system for storing immunization records.

For now, the U.S. looks to data from countries like the U.K. and Israel for guidance on vaccine effectiveness. These countries have fewer people to monitor, but also more centralized and robust systems for tracking COVID spread.

Others, like the Pandemic Tracking Collective Malaty leads, have undertaken efforts to compile state data themselves. "I don't have a lot of confidence in the CDC's ability to track this, to be completely honest," Malaty said. "That's why our team is working really hard to kind of help add some more data transparency to the public."

Comment