The Readability Paradox: Can We Trust Decisions on AI Detectors?

Abstract

Artificial Intelligence (AI) detectors are increasingly used in various domains, including healthcare, finance, criminal justice, and more, to make critical decisions. These systems are designed to identify patterns, anomalies, or specific features within data to assist or automate decision-making processes. However, the trustworthiness of AI detectors is a growing concern, particularly as these systems can exhibit biases, errors, and lack of transparency. This study evaluates the effectiveness of AI-detection tools and analyzes the linguistic characteristics distinguishing AI-generated and human-written academic texts across six disciplines. A total of 200 research papers were examined, 100 generated using large language and 100 peer-reviewed human-authored articles. Five AI detection tools were assessed for accuracy in classification. The study also conducted a comparative readability analysis using five established indices which include Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning Fog Index, SMOG Index, and total Word Count. Results indicate that while human-written papers were correctly identified by most tools with over 80% accuracy, AI-generated papers were frequently misclassified, especially after paraphrasing. Also from the result, the AI-generated texts were significantly shorter and exhibited higher syntactic complexity, with lower readability scores across all indices and disciplines. These findings underscore limitations in current detection tools and highlight notable stylistic differences in how AI and humans generate academic content, with implications for academic integrity policies and future AI writing systems.