PIILO: An open-source system for personally identifiable information labeling and obfuscation

Abstract

Education is increasingly taking place in technologically mediated settings, making it easier to collect data for learning analytics. However, much of this data is not available to the research community due to concerns about protecting student privacy, and deidentification remains difficult for unstructured data such as student-generated text. This study reports on an automatic deidentification system for personally identifiable information labeling and obfuscating (PIILO) in student-generated text. The system labels student names using a fine-tuned large language model and pattern-matching for other identifier types. The model recalled 84% of student names on a held-out testing set. A combined labeling system automatically detected 75% of direct identifiers in a second dataset of 2,118 classroom discussion board posts. The identifiers in the second dataset were obfuscated using a replacement strategy called hiding-in-plain-sight (HIPS, Carrell et al., 2013, 2019), which replaces labeled identifiers with artificially generated surrogates of the same type, making it difficult to distinguish them from any residual identifiers. In a simulated reidentification attack, experts recovered less than 25% of residual identifiers in HIPS-obfuscated data. The automatic approaches to text deidentification developed present a low-cost alternative to manual deidentification.