NLP | Jason S. Lucas

DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects

Wed, 08 Apr 2026 00:00:00 +0000

🏆 Social Impact Paper Award — ACL 2026, San Diego, July 2–7, 2026.

DIA-HARM investigates the robustness of harmful content detection systems across 50 English dialects, addressing critical equity gaps in automated content moderation.

Key contributions include:

Dialect-Diverse Detection (D3) Corpus: Over 195K samples derived from benchmark harmful content datasets, transformed using 189 morphosyntactic rules from eWAVE covering 50 dialects
Comprehensive Model Evaluation: 16 detection models tested — 10 fine-tuned, 5 zero-shot, and 1 in-context learning — revealing systematic performance disparities across dialect groups
D-PURIFY Validation: Quality filtering pipeline ensuring linguistic validity of dialect transformations with 97.2% average F1 for the best model (mDeBERTa)
Key Finding: Detection degradation correlates with density of morphosyntactic transformations rather than specific dialect features, with 2,450 dialect pairs analyzed

Resources:

Area Chair — ACL 2026

Thu, 05 Mar 2026 00:00:00 +0000

Serving as Area Chair (OpenReview) for ACL 2026 — the 64th Annual Meeting of the Association for Computational Linguistics, taking place July 7–12, 2026 in San Diego, California.

Coordinate review and meta-review process for submissions in the assigned track
Manage reviewer assignments and discussion to ensure rigorous and fair evaluation
Provide meta-reviews and recommendations for the program committee

Learn more:

Program Committee Member — LoResLM 2026 Workshop (EACL)

Thu, 15 Jan 2026 00:00:00 +0000

Serving as Program Committee Member for the , co-located with EACL in Rabat, Morocco. ( )

Reviewed up to four submitted papers, providing rigorous and constructive feedback on research quality
Contributed to ensuring a strong review process for high-quality workshop proceedings
Promoted the workshop within research circles to increase community engagement and participation

Organizing Committee Member — 12th MASC-SLL

Fri, 01 Aug 2025 00:00:00 +0000

Serving as Organizing Committee Member for the at University Park, PA. ( )

Assisted Conference Chairs with abstracts and paper submission
Participated in the presentation, abstract, and manuscript review processes
Assisted with organization and logistics of conference

Beemo - Benchmark of Expert-edited Machine-generated Outputs

Tue, 24 Jun 2025 14:00:00 +0000

Note

Click on the Video link above to watch the full presentation on YouTube.

Talk Overview

This presentation introduces Beemo, a groundbreaking benchmark that addresses a critical gap in machine-generated text (MGT) detection research. Unlike traditional benchmarks that only consider single-author scenarios, Beemo captures the reality of human-AI collaboration in text creation.

Key Contributions

1. Multi-Author Benchmark Design

19.6k total texts across five use cases: open-ended generation, rewriting, summarization, open QA, and closed QA
Expert-edited content: 2,187 machine-generated texts refined by professional editors
LLM-edited variants: 13.1k texts edited by GPT-4o and Llama3.1-70B-Instruct using diverse prompts

2. Comprehensive Evaluation

33 MGT detector configurations tested across multiple scenarios
Zero-shot and pretrained detectors including Binoculars, DetectGPT, RADAR, and MAGE
Novel task formulations examining detection performance on edited content

3. Critical Findings

Expert editing evades detection: AUROC scores drop by up to 22% for edited content
LLM-edited texts remain detectable: Less likely to be classified as human-written
Category-specific challenges: Detection performance varies significantly across text types

Implications for AI Safety

This research reveals significant vulnerabilities in current MGT detection systems, particularly relevant for:

Content moderation and misinformation detection
Academic integrity in educational settings
Publishing and journalism authenticity verification
Legal and regulatory compliance for AI-generated content

Future Directions

Beemo opens new research avenues in developing more robust detection methods that account for the collaborative nature of human-AI text creation in real-world applications.

This work was conducted at The Pennsylvania State University in collaboration with Toloka AI, MIT Lincoln Laboratory, and University of Oslo.

Conference Reviewer — ACL, EACL, EMNLP

Mon, 15 Jan 2024 00:00:00 +0000

Served as Conference Reviewer for leading NLP venues (2024–2025):

ACL conference review
EACL conference review
EMNLP conference review

Fighting Fire with Fire - EMNLP 2023

Wed, 06 Dec 2023 00:00:00 +0000

Fighting Fire with Fire

LLMs in Crafting and Detecting Disinformation

Jason Lucas¹, Adaku Uchendu¹,², Michiharu Yamashita¹, Jooyoung Lee¹, Shaurya Rohatgi¹, Dongwon Lee¹

¹Penn State University, ²MIT Lincoln Laboratory

EMNLP 2023 Main Conference

Problem & Motivation

Challenge: LLMs generate realistic but harmful disinformation

Persuasive texts indistinguishable from human content
Large-scale disinformation potential
Limited LLM-generated content detection

Core Question: Can LLMs detect their own disinformation?

Research Questions

**RQ1**: Can LLMs efficiently generate disinformation via prompt engineering?

RQ2: How proficient are LLMs at detecting disinformation?

5 Evaluation Dimensions:

Human vs. LLM-generated
Self vs. externally-generated
Posts vs. articles
In vs. out-of-distribution
Zero-shot LLMs vs. fine-tuned detectors

F3 Framework

**Fighting Fire with Fire (F3)** - 5-step approach:

Human data collection
Prompt engineering generation
PURIFY hallucination filtering
Cloze-prompt detection
Zero-shot evaluation

RQ1: Bypassing Alignment

**Key Discovery**: Impersonator roles override safety measures Without role: “Sorry, I can’t assist…” With role (“You are an AI news curator”): ✅ Generates disinformation Finding: Impersonator prompts successfully bypass GPT-3.5 protections

Generation Strategies

**Perturbation-Based** (Fake):

Minor: Subtle changes
Major: Noticeable changes
Critical: Significant alterations

Paraphrase-Based (Real):

Minor: Light summary
Major: Moderate rewording
Critical: Full rephrasing

Output: 43K+ synthetic samples

PURIFY Framework

**Problem**: 38% hallucinated misalignments

PURIFY filters using 4 metrics:

Natural Language Inference
AlignScore
BERTScore
Semantic Distance

Result: 43,272 → 27,667 quality samples

RQ2: Detection Results

**Human vs. LLM Content**:

Human-authored: 55-66% accuracy
LLM-generated: 60-85% accuracy

Self vs. External:

GPT-3.5: Strong self-detection
LLaMA-GPT: Best external detector
Challenge: Minor disinformation detection

Key Findings

Content Type: Articles > Social media posts Distribution: In-distribution > Out-of-distribution Model Type: Fine-tuned > GPT-3.5 > Domain-specific Critical: Subtle disinformation challenges all detectors

Technical Contributions

1. Novel prompting for disinformation generation 2. PURIFY hallucination filtering framework 3. Cloze-prompt detection strategies 4. Comprehensive SOTA benchmark 5. F3 dataset for research community

Dataset & Evaluation

**Models**: GPT-3.5, LLaMA-2, Palm-2, Dolly-2

Data: CoAID, FakeNewsNet, F3 (27,667 samples)

Languages: 11 languages, Pre/Post-GPT splits

Metrics: Macro-F1 across human/AI datasets

Impact & Implications

Dual-Use Reality: LLMs both create and detect disinformation Detection Promise: Zero-shot capabilities show potential Security Concern: Easy alignment bypass requires safeguards Research Direction: Focus on subtle disinformation detection

“Fighting Fire with Fire”

Re-purposing LLMs as countermeasures against disinformation

Questions & Resources

**Code**: https://github.com/mickeymst/F3 **Paper**: EMNLP 2023 Main Conference **Contact**: jsl5710@psu.edu

Penn State University | PIKE Research Lab

Thank You!

Conference Volunteer — EMNLP 2023

Tue, 01 Aug 2023 00:00:00 +0000

Served as Conference Volunteer at the Empirical Methods in Natural Language Processing (EMNLP) 2023.

Provided technical support for the main conference
Provided poster presentations support and coordination
Assisted with participant registration and check-in