Engineering Insights is an ongoing blog series that gives a behind-the-scenes look into the technical challenges, lessons and advances that help our customers protect people and defend data every day. Each post is a firsthand account by one of our engineers about the process that led up to a Proofpoint innovation.
Inspired by the rapid rise of generative artificial intelligence (GenAI), we recently kicked off several internal initiatives at Proofpoint that focused on using it within our products. One of our leadership team’s goals was to find a tool to help increase developer productivity and satisfaction. The timing was perfect to explore options, as the market had become flush with AI-assisted coding tools.
Our project was to analyze the available tools on the market in-depth. We wanted to choose an AI assistant that would provide the best productivity results while also conforming to data governance policies. We set an aggressive timeline to analyze the tools, collaborate with key stakeholders from legal, procurement, finance and the business side, and then deploy the tool across our teams.
In the end, we selected GitHub Copilot, a code completion tool developed by GitHub and OpenAI, as our AI coding assistant. In this post, we walk through how we arrived at this decision. We also share the qualitative and quantitative results that we’ve seen since we’ve introduced it.
Our analysis: approach and criteria
When you want to buy a race car—or any car for that matter—it is unlikely that you’ll look at just one car before making a final decision. As engineers, we are wired to conduct analyses that dive deeply into all the possible best options as well as list all the pros and cons of each. And that’s what we did here, which led us to a final four list that included GitHub Copilot.
These are the criteria that we considered:
- Languages supported
- IDEs supported
- Code ownership
- Stability
- AI models used
- Protection for intellectual property (IP)
- Licensing terms
- Security
- Service-level agreements
- Chat interface
- Innovation
- Special powers
- Pricing
- Data governance
- Support for a broad set of code repositories
We took each of the four products on our shortlist for a test drive using a specific set of standard use cases. These use cases were solicited from several engineering teams. They covered a wide range of tasks that we anticipated would be exercised with an AI assistant.
For example, we needed the tool to assist not just developers, but also document writers and automation engineers. We had multiple conversations and in-depth demos from the vendors. And when possible, we did customer reference checks as well.
Execution: a global rollout
Once we selected a vendor, we rolled out the tool to all Proofpoint developers across the globe. We use different code repos, programming languages and IDEs—so, we’re talking about a lot of permutations and combinations.
Our initial rollout covered approximately 50% of our team from various business units and roles for about 30 days. We offered training sessions internally to share best practices and address challenges. We also built an internal community of experts to answer questions.
Many issues that came up were ironed out during this pilot phase so that when we went live, it was a smooth process. We only had a few issues. All stakeholders were aware of the progress, from our operations/IT team to our procurement and finance teams.
Our journey from start to finish was about 100 days. This might seem like a long time, but we wanted to be sure of our choice. After all, it is difficult to hit “rewind” on an important initiative of this magnitude.
Monitoring and measuring results
We have been using GitHub Copilot for more than 150 days and during that period we’ve been collecting telemetry data from the tool and correlating it with several productivity and quality metrics. Our results have been impressive.
When it comes to quantitative results, we have seen a general increase in velocity that’s upwards of 20%! We have also seen an increase in quality, as measured by the number of defects reported. One business unit—where we have a consistent measurement of defect rates over time—has seen a decrease in reported defects of over 50%!
There are also qualitative benefits. Many significant users of GitHub Copilot have indicated that their job satisfaction has increased because they can focus more on the “enjoyable parts” of the software development lifecycle. That’s because they rely on the AI code assist tool to automate or accelerate some of the more mundane aspects of their workflow.
Interestingly, we have observed different levels of user engagement. Many of our engineering staff use it extensively, while others use it sporadically or not at all. We are in the process of assessing what factors could lead to increased use. As we see this capability more widely embraced, we anticipate productivity and quality numbers will increase further.
As we engage with family, colleagues and others, we sometimes hear the question, “Will my job be replaced by AI?” At some point, it may be a question to consider more deeply. But based on our experience so far using AI to assist with an intensive knowledge workflow—software engineering—we would answer, “It probably won’t replace your job. But if you aren’t figuring out how to use AI to make yourself more productive, you should be concerned about being replaced by someone who is.”
Join the team
At Proofpoint, our people—and the diversity of their lived experiences and backgrounds—are the driving force behind our success. We have a passion for protecting people, data and brands from today’s advanced threats and compliance risks.
We hire the best people in the business to:
- Build and enhance our proven security platform
- Blend innovation and speed in a constantly evolving cloud architecture
- Analyze new threats and offer deep insight through data-driven intelligence
- Collaborate with our customers to help solve their toughest cybersecurity challenges
If you’re interested in learning more about career opportunities at Proofpoint, visit the careers page.
About the authors
Vaish Krishnamurthy is a vice president of engineering at Proofpoint. Before her current role, she held various technical leadership roles at machine learning and AI startups. She also founded her startup, CleanRobotics, in 2015, where she and other engineers created an automated smart trashcan that separates recyclables from landfill waste. Vaish was named by the Pittsburgh Business Times as a “Fast Tracker” (40 Under 40) in 2015. She started her Ph.D. in electrical engineering when she was 20 but left that track to pursue industry roles. Vaish holds a master’s degree in electrical engineering from the University of California, Riverside. She is a strong advocate for increasing the number of women in computer science; she gave a talk at the University of Pittsburgh WiCS group and has blogged about it.
As group vice president of AI at Proofpoint, Dan Rapp spearheads the adoption of cutting-edge AI technologies into the company’s suite of security, protection and compliance solutions. His extensive background in research and development, AI, and machine learning informs his leadership of Proofpoint Labs, the company’s innovation hub. The team at Proofpoint Labs boasts a diverse range of expertise in key areas like cybersecurity, natural language processing and big data, ensuring that Proofpoint continues to lead in the information security and protection field.
As the senior director of engineering operations, Prasanna Basavapatna is responsible for driving the CORE organization’s financial planning, learning, tooling and productivity, security engineering, and compliance and standards functions. Before joining Proofpoint about four years ago, Prasanna spent nearly 20 years with McAfee and Sun Microsystems where he led the governance, risk and compliance product engineering teams.
Subscribe to the Proofpoint Blog