UNC-Chapel Hill leaders share facts on Willingham dataset, findings
Friday, Jan. 17, 2014
Details:
In a regular meeting of the University of North Carolina at Chapel Hill Faculty Council, Chancellor Carol L. Folt and Executive Vice Chancellor and Provost James W. Dean, Jr. shared their initial findings from an evaluation of the dataset used by Mary Willingham, a student adviser. Folt and Dean shared facts that laid out a range of serious mistakes made by Willingham in the University’s analysis of her data.
“We take these allegations seriously,” Folt said. “Our goals are to be proactive in our analysis and solutions, to protect the privacy rights of individual students, and to apply the rigorous standards of assessment expected here at Carolina.”
The University’s Academic Support Program for Student Athletes (ASPSA), which serves Carolina’s 800 student-athletes with tutoring services and academic support and reports directly to the provost, used the Scholastic Abilities Test for Adults (SATA) as a tool to screen some student-athletes for potential learning differences/learning disabilities until 2012-2013. [The SATA is not affiliated with the SAT administered by the College Board.] Willingham has made claims regarding student-athletes’ reading grade equivalency based on SATA test results. Based on an initial University review, there is sufficient evidence to take issue with these claims, which is why Folt and Dean chose today to update the Faculty Council.
Key Points:
IRB Review Triggered by Federal Rules Regarding Use of Protected Data
Student Reading Levels Were Assessed by a Simple Vocabulary Test
Test Was Not Designed to Measure Reading Comprehension
Wrong Data From the Test Apparently Used To Predict Grade Equivalency
University Seeking Independent Analysis of Dataset
Background:
Last week, CNN reported about Willingham’s observations regarding reading levels for some football and basketball players based on SATA Reading Vocabulary Subtest results between 2004 and 2012. The findings neither represented the University’s position on the reading abilities of its student-athletes, nor reflected official data from the Office of Undergraduate Admissions. That office’s analysis found that between 2004 and 2012, the same time period examined by CNN, more than 97 percent of UNC-Chapel Hill first-year student-athletes met or exceeded the threshold for “college literacy” defined by CNN’s own experts. The 2012–2013 first-year cohorts fared even better, with only two of 321 student-athletes who entered through UNC’s special talent provision not meeting the threshold; all of the 2013 cohort met or exceeded the threshold.
Folt and Dean shared the following initial findings with faculty leaders:
1. Why the Institutional Review Board (IRB) rescinded its earlier determination that Willingham’s study was not “human subjects research:”
Every research university is required by the federal government to have a campus body, such as an IRB, to oversee human subjects research. UNC-Chapel Hill’s IRB is an independent governing body comprised of faculty researchers, knowledgeable staff and community members. Willingham’s analysis contradicted information she provided in her formal application to the federally regulated IRB in 2008. At that time, based on information she provided, the IRB determined that their oversight was not needed because 1) the data were represented as secondary (not collected directly from subjects) and 2) individuals could not be identified. De-identified means that the researchers themselves do not know the identities of subjects—not that the researchers do not disclose their subjects’ identities as part of their work. As a result, the IRB allowed Willingham to proceed without direct IRB oversight. Willingham collected and retained identified data, as her public comments over the past two weeks have confirmed, and this was the basis of the IRB’s decision to rescind its earlier determination.
2. Did the University have the dataset in question?
Since last August, Willingham had shared general information with statistics about enrollment patterns in a specific course and some additional information that she said reflected her own personal analysis about academic data and student-athletes.
Until January 13th, Willingham had not provided University officials with more detailed information about her dataset. A representative of the Provost’s Office and the chair of the Faculty Athletics Committee had previously unsuccessfully requested that data in the context of academic reform efforts underway on campus. Contrary to Willingham’s public statements, no full datasets had been delivered previously to the University.
University officials understand that a subset of data was provided to Baker Tilly, a consulting firm, and former N.C. Gov. James Martin by Willingham during her interview. Their 2012 investigation was completely independent, and the University did not receive the investigative materials from their work, including this dataset until recently. University officials also understand that Baker Tilly did take Willingham’s information into consideration in their final report and findings.
The Office of University Counsel also received information in 2010, but it was not the data. In fact, a handwritten note from Willingham accompanying the flash drive she provided stated she could not include the dataset until it was de-identified.
3. What was the test used to determine reading grade levels?
The test that Willingham used as the underpinning of her analysis and findings was the Scholastic Abilities Test for Adults (SATA), a Reading Vocabulary Subtest (not comprehension). The actual test instrument was one page with 25 questions, and typically requires 10 minutes to complete. The test instruction reads as follows: “Read the four words and identify one of three possible relations among the words. One possibility is that two words have the same meaning [or] they have opposite meanings [or] none of the words are related in meaning.”
According to the SATA Manual: “any standardized test purporting to provide a comprehensive measure of reading that does not assess sentence or passage comprehension should be considered inadequate.” (Wiederhold & Bryant, 1987, p. 96, quoted in SATA Manual on p. 28)
4. What data were used to determine grade levels?
Results of the SATA Reading Vocabulary Subtest can be expressed as raw scores, standard scores, percentiles, or grade equivalents. The dataset Willingham provided to the University included only standard scores, but the findings she has reported publicly misrepresent these standard scores as grade equivalents. This misrepresentation leads to serious errors in the findings. The initial University evaluation found Willingham’s calculations to be off by a factor of 10, which would dramatically change the findings she has shared publicly.
5. Do the data support claims that have been made?
The University’s initial evaluation has revealed a basic mistake in Willingham’s apparent methodology that led her to make serious accusations about the literacy levels of UNC-Chapel Hill student-athletes.
According to Dean, “Our judgment at this point is that claims made based on this dataset are nearly meaningless and grossly unfair to our students.”
According to Chancellor Folt, “Carolina has a world-renowned reputation for our research, and the work we have just reviewed does not reflect the quality and excellence found throughout the Carolina community.”
“I’m proud of the dedicated and talented students – and the work of our faculty and staff – who make us so proud of Carolina every day.”
6. What comes next?
The University will have Willingham’s raw dataset and apparent methodology independently evaluated and reported back to the University as soon as that process has been completed.
Related Links:
Admissions Backgrounder,
http://uncnews.unc.edu/2014/01/16/97-un ... ews-story/
Chancellor Folt’s Statement to the Campus Community,
http://www.unc.edu/campus-updates/chanc ... to-campus/