https://jsl5710.github.io/tag/text-detection/Text DetectionHugoBlox Kit (https://hugoblox.com)en-usTue, 24 Jun 2025 14:00:00 +0000https://jsl5710.github.io/media/icon_hu_1b2044c02ce09a43.pnghttps://jsl5710.github.io/tag/text-detection/https://jsl5710.github.io/event/talk-munich_invited/Tue, 24 Jun 2025 14:00:00 +0000https://jsl5710.github.io/event/talk-munich_invited/<div class="callout flex px-4 py-3 mb-6 rounded-md border-l-4 bg-blue-100 dark:bg-blue-900 border-blue-500" data-callout="note" data-callout-metadata=""> <span class="callout-icon pr-3 pt-1 text-blue-600 dark:text-blue-300"> <svg height="24" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" d="m16.862 4.487l1.687-1.688a1.875 1.875 0 1 1 2.652 2.652L6.832 19.82a4.5 4.5 0 0 1-1.897 1.13l-2.685.8l.8-2.685a4.5 4.5 0 0 1 1.13-1.897zm0 0L19.5 7.125"/></svg> </span> <div class="callout-content dark:text-neutral-300"> <div class="callout-title font-semibold mb-1">Note</div> <div class="callout-body">Click on the <strong>Video</strong> link above to watch the full presentation on YouTube.</div> </div> </div> <h2 id="talk-overview">Talk Overview</h2> <p>This presentation introduces <strong>Beemo</strong>, a groundbreaking benchmark that addresses a critical gap in machine-generated text (MGT) detection research. Unlike traditional benchmarks that only consider single-author scenarios, Beemo captures the reality of human-AI collaboration in text creation.</p> <h2 id="key-contributions">Key Contributions</h2> <h3 id="1-multi-author-benchmark-design">1. Multi-Author Benchmark Design</h3> <ul> <li><strong>19.6k total texts</strong> across five use cases: open-ended generation, rewriting, summarization, open QA, and closed QA</li> <li><strong>Expert-edited content</strong>: 2,187 machine-generated texts refined by professional editors</li> <li><strong>LLM-edited variants</strong>: 13.1k texts edited by GPT-4o and Llama3.1-70B-Instruct using diverse prompts</li> </ul> <h3 id="2-comprehensive-evaluation">2. Comprehensive Evaluation</h3> <ul> <li><strong>33 MGT detector configurations</strong> tested across multiple scenarios</li> <li><strong>Zero-shot and pretrained detectors</strong> including Binoculars, DetectGPT, RADAR, and MAGE</li> <li><strong>Novel task formulations</strong> examining detection performance on edited content</li> </ul> <h3 id="3-critical-findings">3. Critical Findings</h3> <ul> <li><strong>Expert editing evades detection</strong>: AUROC scores drop by up to 22% for edited content</li> <li><strong>LLM-edited texts remain detectable</strong>: Less likely to be classified as human-written</li> <li><strong>Category-specific challenges</strong>: Detection performance varies significantly across text types</li> </ul> <h2 id="implications-for-ai-safety">Implications for AI Safety</h2> <p>This research reveals significant vulnerabilities in current MGT detection systems, particularly relevant for:</p> <ul> <li><strong>Content moderation</strong> and misinformation detection</li> <li><strong>Academic integrity</strong> in educational settings</li> <li><strong>Publishing and journalism</strong> authenticity verification</li> <li><strong>Legal and regulatory</strong> compliance for AI-generated content</li> </ul> <h2 id="future-directions">Future Directions</h2> <p>Beemo opens new research avenues in developing more robust detection methods that account for the collaborative nature of human-AI text creation in real-world applications.</p> <hr> <p><em>This work was conducted at The Pennsylvania State University in collaboration with Toloka AI, MIT Lincoln Laboratory, and University of Oslo.</em></p>