Detecting LLM-Generated Texts with “Classical” Machine Learning
Points and comments are a snapshot, not live.
Traditional ML (SVM+TF-IDF) detects LLM-generated web fiction with up to 85% sentence-level accuracy.
The author built a classifier using scikit-learn's LinearSVC and TF-IDF on Chinese web fiction. Training data: 10,000 human-written texts from 2010-2022, plus texts generated by 7 LLMs (Gemini, Qwen, GLM, Kimi, Deepseek, Doubao). Sentence-level accuracy hits ~85% per model; 7-model majority voting reduces false positives. A JS browser demo runs TF-IDF+SVM entirely client-side with a 107MB JSON model. Tests on unseen models (Claude, GPT-5.2) still detect ~70% of AI text. False positive rate on 10,000 pre-2022 fanfics is 0.04% at 60% threshold, under 0.01% at 70%.
Of 1,800+ trending Lofter articles, 32% scored >50% AI-none declared AI-generated. The author notes translation roundtrip and prompt-based rewriting only slightly reduce detection scores (from 89.9% to ~83-86%).
What commenters are saying
Commenters split on detectability. Some argue LLM text has inherent statistical patterns beyond style, citing the author's 0.04% false positive rate as evidence. Others contend detection is a cat-and-mouse game: adversarial fine-tuning (GAN-style) or high-temperature sampling can defeat it, and language drift from heavy LLM use will blur human-AI boundaries. One commenter built a similar browser extension using a quantized MiniLM model, reporting 0.9944 AUC.
Several note that current commercial models' bland, engagement-optimized style is the real tell-not technical impossibility. A few stress that false positives harm real writers, especially those with formal or technical prose.