Less than a month after Twitter started its Birdwatch fact-checking program, a study has unearthed some troubling issues with the crowdsourced initiative. The Poynter Institute for Media Studies analyzed almost 2,700 contributions and found that less than half of the program’s participants submitted reliable sources and citations to back up their fact checks. The institute also determined that there’s a partisan slant to some of the submitted notes.
Birdwatch, which is currently a pilot project, has around 1,000 users. They can flag tweets as misleading and, in a note, they can cite a source that disputes claims made in a tweet or explain why it may not be accurate. The notes are displayed under the tweet and other Birdwatch users can rate how helpful they are. They can be filtered by recency or helpfulness.
Among the notes that Poynter analyzed earlier this week, only around seven percent were rated as helpful. More than a third of those have source links that are just another tweet or nothing at all. The cited tweets may also be inaccurate, which amplifies the problem.
The report notes that a tweet from YouTuber Tim Pool, who wrote that the US presidential election was “rigged,” was flagged as having a disputed claim. Some Birdwatch users disagreed and it backed up Pool’s assertion, with others rating such notes as helpful. Twitter this week removed the “helpful” label from the notes and tweaked the Birdwatch algorithm.
A note now needs at least five ratings (up from three) before the “rated helpful” label is applied. Twitter also increased the helpfulness rating threshold from 0.5 to 0.84. Under five percent of Birdwatch notes meet the updated requirements for the label, according to Poynter, and three quarters of those link to a source away from Twitter.
The institute also found that there’s a degree of partisanship among Birdhouse participants. “A majority of the most prolific Birdwatch user’s notes mark tweets critical of the right as ‘misleading’ and those critical of the left as ‘not misleading,'” according to the report. It determined that the five most active users account for more than a tenth of Birdhouse notes and four of them “have similar activity as the most prolific user.” Just one of the top five users, the second most-frequent contributor, includes a corroborating source link with all of their notes.
The Twitter handles of contributors are included alongside their notes, as Poynter’s Alex Mahadevan points out, which could make them a target of harassment or abuse. Participants can register for Birdwatch under a pseudonym, according to Twitter.
Birdwatch is a work in progress, but fact-checking isn’t easy. Leading fact-checkers suggested to Poynter that Twitter should offer training and incentives to Birdwatch users while employing professionals to assess high-ranking notes.
Twitter’s vice president of product, Keith Coleman, told the publication that the company doesn’t have a timeline for rolling out Birdwatch more broadly and that it’s “working to learn as much as possible and iterate while the pilot is small.” He noted that Twitter values openness in terms of who can contribute to the project and that “through input from a diverse group, the most helpful notes can be elevated.”
Share This on Facebook