On February 10, 2025, Proofpoint encountered a production incident that impacted email delivery services. The core issue stemmed from a flawed detection policy rule that erroneously classified email messages containing URLs as threats. This resulted in the unintended quarantine of legitimate emails that contained URL's. Additionally, repeated remediation of a message-more than 10 times —also led to the condemnation of the associated domain.
The incident was not related to a cyberattack. Once identified, Proofpoint removed the defective rule, and implemented remediation measures to restore normal email delivery. To prevent future occurrences, Proofpoint is enhancing its procedures for rule generation, change management, and communications
Customer Experience
Between 22:35 UTC (February 10, 2025) and 11:00 UTC (February 11, 2025), customers reported an unexpected increase in false positives in their quarantine folders. The issue stemmed from a system error that incorrectly flagged all incoming emails containing URLs as phishing attempts, leading to their quarantine. As a result, any URL flagged during this period caused both the initial and subsequent emails containing these URLs to be automatically quarantined. These emails were then handled according to the specific disposition rules set by each customer.
Future Actions
Proofpoint has implemented immediate controls to prevent an incident such as this from recurring. We have deployed a software update to the rules service, which now verifies the integrity of all rules in a package before deployment.
Furthermore, to ensure resiliency and faster resolution during incidents, Proofpoint is conducting independent reviews on:
- End-to-end quality processes of our rules generation and change management processes
- Current recovery processes and identify areas to streamline
- Communication processes with customers to ensure we communicate the impact in a timely manner with accurate information
Download the Incident Root Cause Analysis | Download our CEO's note
