Abstract
As Large Language Models (LLMs) become increasingly sophisticated, distinguishing human-written text from AI-generated content has become a critical challenge for academic integrity and information security. This paper presents a robust AI text detection system fine-tuned on the DeBERTa-v3 architecture, trained specifically to distinguish human-authored Wikipedia articles from text generated by GPT-Neo (1.3B). The model achieves 99.4% accuracy and an F1-score of 0.99 on in-distribution test data. However, I demonstrate a significant "generalization gap" when evaluating the model on out-of-distribution text generated by state-of-the-art (SOTA) models such as GPT-4, where confidence scores drop significantly (0.54–0.74). These findings suggest that current detection methodologies are model-specific and that "universal" detection remains an elusive goal without continuous retraining on evolving model architectures.
1. Introduction
Problem Statement: The rapid democratization of LLMs (ChatGPT, Claude, Gemini) has led to a flood of AI-generated content, raising concerns about plagiarism, misinformation, and the dilution of human discourse.
The Challenge: While earlier detection systems were effective at identifying statistical anomalies in older models (such as GPT-2), modern LLMs closely mimic human perplexity and burstiness, significantly increasing the difficulty of reliable detection.
Contribution:
- A domain-controlled dataset is engineered to minimize topic bias.
- A random chunking preprocessing pipeline is introduced to eliminate structural artifacts.
- Empirical evidence is provided demonstrating that detectors fine-tuned on open-source models (GPT-Neo) struggle to generalize to proprietary SOTA models (GPT-4).
2. Related Work
Transformer Approaches: Previous studies have explored transformer-based classifiers such as RoBERTa and DeBERTa for AI text detection. He et al. (2021) demonstrated that DeBERTa’s disentangled attention mechanism improves contextual representation, making it a strong baseline for stylistic analysis.
The Arms Race: Recent work highlights an ongoing arms race between text detectors and increasingly capable LLMs. Sadasivan et al. (2023) show that paraphrasing tools and advanced generation models can easily bypass existing detection systems, raising doubts about the long-term viability of static detectors.
3. Methodology
3.1 Data Curation & Generation:
Human Data: Over 2,000 Wikipedia articles were scraped across diverse domains, including science, history, and technology.
AI Data: More than 1,600 samples were generated using GPT-Neo-1.3B across the same domains to ensure consistency.
Prompt Engineering: Explicit “Wikipedia-style” prompts (e.g., “Write a formal encyclopedia entry about...”) were used to enforce stylistic alignment and prevent the model from learning trivial cues such as stereotypical section endings.
3.2 Preprocessing Pipeline:
The Artifact Problem: Initial experiments resulted in near-perfect accuracy (≈100%) due to unintended structural leakage, such as human text beginning mid-sentence while AI-generated text consistently started with capitalized sentences.
The Solution: To address this issue, a random chunking strategy was implemented. Both human and AI texts were generated at approximately 400 words and then randomly sliced into ~300-word segments. This ensured consistent sentence fragmentation and eliminated positional biases.
3.3 Model Architecture:
Base Model: microsoft/deberta-v3-base.
Reasoning: DeBERTa improves upon BERT and RoBERTa through disentangled attention and enhanced positional encoding, which are essential for capturing subtle stylistic and syntactic dependencies.
3.4 Training Configuration:
Hyperparameters: Learning Rate: 2e-5, Batch Size: 8, Epochs: 3
Optimization: AdamW optimizer with linear learning rate decay.
Regularization: Early stopping was applied at epoch 2 to mitigate overfitting.
4. Experiments and Results
4.1 In-Distribution Performance (GPT-Neo)
Metrics from the best checkpoint (Epoch 2):
Interpretation: The model effectively learned the statistical signature of GPT-Neo-generated text, achieving zero false negatives within the training distribution.
4.2 Out-of-Distribution Performance (The "Generalization Gap")
Test Set: Samples generated by GPT-4 and Gemini.
Result: The detector frequently misclassified SOTA AI-generated text as human-authored, with confidence scores falling within the 50–70% range, indicating high uncertainty.
5. Discussion
Quality Hypothesis: GPT-Neo exhibits subtle weaknesses such as repetition and simpler sentence structures, which the detector learns to exploit. GPT-4 largely eliminates these artifacts, leading the detector to incorrectly classify such text as human-written.
Domain Sensitivity: The model is highly specialized for Wikipedia-style prose. Informal or creative AI-generated text shifts the decision boundary and reduces detection reliability.
Implications: These findings suggest that static detection systems are not sustainable. Effective AI text detection must function as a continuously updated system, retrained frequently on outputs from newly released generative models.
6. Limitations
Although the proposed DeBERTa-based detection system demonstrates strong performance within its training distribution, several limitations hinder its use as a universal detector:
6.1 Rapid Evolution: Training was conducted on GPT-Neo-1.3B (a 2021-era architecture), while evaluation on GPT-4 and Claude 3 revealed a substantial drop in sensitivity. Newer models exhibit lower burstiness and more human-like perplexity profiles.
6.2 Domain Rigidity: Fine-tuned exclusively on Wikipedia text to control for topic bias. Consequently, it struggles with informal, conversational, or creative writing styles (emails, fiction, social media).
6.3 Adversarial Vulnerability: Does not assess robustness against paraphrasing attacks or prompt injection tricks designed to increase burstiness.
6.4 Short Inputs: Detection reliability decreases sharply for inputs under 50 words where the attention mechanism lacks sufficient context.
7. Conclusion and Future Work
Summary: This study demonstrates that while highly accurate detectors can be built for specific generative models, transferability across architectures remains fragile, highlighting a fundamental limitation of current AI text detection approaches.
Future Work: Incorporate multi-domain datasets (e.g., Reddit, social media) using HC3.
References
- [1] He, P., et al. (2021). DeBERTa: Decoding-enhanced BERT with Disentangled Attention.
- [2] Black, S., et al. (2021). GPT-Neo: Large Scale Autoregressive Language Modeling.
- [3] Zhang, J., et al. (2024). Transformers: State-of-the-Art Natural Language Processing.
- [4] OpenAI (2023). GPT-4 Technical Report.