I first read about the “death of email” in the early 2000s. Nearly two decades later, message traffic has grown exponentially. For many organizations, this represents both opportunity and risk.
It’s easier and faster to send customer information electronically. However, it can also lead to sending too much information like personal identifiable information (PII), confidential company material or inappropriate communications. Many companies use supervisory review tools such as Proofpoint Intelligent Supervision to help mitigate this risk and meet industry or regulatory requirements.
What does a supervision system do? It evaluates electronic communication, flagging content such as unauthorized, illegal or unprofessional messages. Suspect content can be identified using rules, sampling and artificial intelligence (AI). A supervision system evaluates content from various data sources—email, texts, social media, instant messaging, collaboration tools—anything that’s supported by a capture and/or archiving solution.
Now, let’s focus the rest of this blog on the heart of a robust supervision system: its rules.
“Hotwords” not always so hot
Rules are used to flag messages of interest for human review. Reviewers can act or clear a flagged message. A cleared message is a “false positive,” also referred to as “noise” or low-risk content. Common causes of false positives include disclaimer text, newsletters or overly broad rule criteria.
There is an art to creating rule criteria. Well-written rules will flag questionable messages without flagging too many false positives.
Legacy systems commonly use simple “hotword” lists to match specific keywords that could indicate sensitive or questionable content. While this may work well for simple risk categories, such as profanity, it would result in a high number of false positives for other categories, such as detecting data loss or monitoring dealer/broker messaging, where key hotwords would be expected in the normal course of business.
A focus on capturing the “true positives”
At Proofpoint, we take a different approach. We provide an extensive library of prebuilt rules based on our experience with customers across industries, including financial, broker/dealer, insurance, healthcare and more. These prebuilt rules can help our customers get started with developing a robust supervision system that can flag suspect content for review.
Implementing rules from the rule library is only the first step. Rules need to be customized and evaluated for performance. The goal should be to have rules that capture all “true positives”—messages that require action—while keeping the false positives to a manageable rate.
This is the rule refinement process. And Proofpoint uses a risk-based approach to rule refinement, based on reports and analytics gathered from flagged messages and reviewer action.
Dramatic reduction in flagging rates and false positives
Using several reports, including the “Flagging Rule Effectiveness Report,” as well as the “Flagging Match” and “Flagging False Positive” reports, we can identify rules with high rates of false positives and drill down to the individual content criteria or message snippets responsible.
From there, we can decide what corrective adjustments are needed. Our options are rule modification, new disclaimers or exclusion rule updates.
In the table below, you’ll see how rule refinement for an internal lab test helped reduce the flagging rate by more than 77%, and false positives by more than 81%.
Figure 1. Proofpoint risk-based approach to rule refinement
Rule refinement in practice
Following is a Proofpoint example of how the rule refinement process can help increase efficiency and reduce risk in the message review process:
On a recent engagement, our team assisted a large financial organization with the initial conversion to our Intelligent Supervision solution from a legacy supervision system. The customer opted to implement the Proofpoint rule library, replacing their existing single word/phrase list.
The organization’s IT department drove the initial implementation, and the business managers were consulted toward the end of it. Although we discussed the benefits of the rule library, several managers told us they were more comfortable with word lists, similar to what their legacy product provided.
We implemented several word lists on a Friday afternoon. By Monday, we found one word list had flagged 37,000 messages. We learned the reviewers expected this level of flagging. They conducted a manual inspection of a handful of items before “bulk” clearing all the flagged messages.
Meanwhile, another organization was overwhelmed with false positives, and the reviewers fell further behind. In this instance, they wanted to address the problem by “sampling” the rule criteria—or randomly selecting a small percentage of messages matching the simple keyword criteria.
Both choices could result in clearing suspect content and also violating Financial Industry Regulatory Authority (FINRA) or other regulatory agency review requirements. But there is a better, less risky way.
We worked with our customer to produce and analyze the report data and demonstrated to the business managers a better method of identifying content using context. Instead of matching on hotwords, we expanded the criteria to include additional nearby/related terms.
This approached help reduce the false positive rates by 73%. It also improved the quality of the messages flagged for review, leading to an increase of the true positive rate from <.01% to .03%. The result was significantly fewer review hours spent evaluating irrelevant or compliant messages.
Support for your supervision journey
Message review is an increasingly important task for many businesses. With the proper technology and training, it can be an efficient risk mitigation process. And rule refinement is an essential part of maintaining a functional supervision system.
Proofpoint Professional Services can help support you at every stage of your supervision journey. Contact your Proofpoint sales team for more information.