Project Description
This project investigates how disinformation and its corrections are structured as narratives, and whether an AI system used to measure that structure is itself biased. Drawing on a folklore-based coding framework, the research applies a fixed rubric to matched pairs of false claims and the corrections that answer them, drawn from both independent fact-checking organizations and state-run counter-disinformation programs. Each claim and correction is scored on a set of narrative elements by an AI system, using a consistent, unedited prompt so the coding is reproducible. The central question is whether the AI system reads more story into a text than a trained human reader would, and if so, where that tendency concentrates. To answer this, every AI-generated score is independently checked by two trained student readers. The methodology deliberately separates two distinct questions: whether two humans agree with each other, which tests whether the coding rubric itself is usable, and whether the AI agrees with humans, which tests the AI system for bias. Keeping these separate, rather than treating agreement with the model as success, is what allows the research to detect and measure bias rather than simply confirm the system's output.
Tasks and Responsibilites
Two students will serve as independent human auditors of the AI system's narrative coding. For each case, a student reads the claim and its correction, forms an independent judgment using the project's coding rubric, and only then compares that judgment against the AI system's score. Where the two agree, the student confirms it. Where they disagree, the student records their own score and a brief reason for the disagreement. Students work independently of one another so that their judgments serve as a genuine check rather than a shared opinion. Over the course of the semester, they apply this process across an initial set of materials and then a second, independently gathered set, allowing the research to test whether any patterns of disagreement hold up on new material rather than being specific to the first batch. Students also help compile basic summary figures from their own audit work, such as how often they agreed with the AI system and where disagreements tended to cluster, and contribute their observations to the eventual written report.