What Is Prompt Engineering?

Prompt engineering is the practice of fine-tuning and structuring inputs, or “prompts”, to shape how large language models (LLMs) behave and the outputs they produce. LLMs are the backbone of today’s artificial intelligence systems, and prompts form the instructions, context, and limitations of the model’s outputs. Prompt engineering directly affects AI governance and security, as negligent or poorly designed prompts can result in data leaks, output bias, prompt-injection attacks, and a slew of other attacks.

In LLMs, prompts are what drive the heavy lifting. The model’s output is influenced by how the instructions are framed, the role context it frames, and the examples it includes. For IT and AI teams, that relationship between input and output is the foundation of reliable performance.

Developers or administrators set system prompts to define the model’s overall behaviour and guardrails. User prompts aren’t always the same; they change with each session. Security teams pay close attention here because open-ended user inputs can cause a model to produce outputs or data that weren’t meant to be produced or visible to certain users.

Role framing changes how the model sees things. Giving it a persona, like “You are a security analyst looking at a suspicious email”, helps it choose its tone, words, and perception. Context and examples establish framing, and anchor responses are based on expectations specific to a field.

Temperature settings adjust the output of randomness and creativity to control how much the output changes. Higher settings produce more creative and less predictable responses. When compliance leaders want to ensure outputs are consistent and follow the rules, they usually prefer lower settings because they make outputs tighter and more predictable.

The mechanisms that bridge the gap between governance and practice are guardrails and constraints. Well-structured system prompts serve as a policy layer, restricting the topics a model can address or disclose. For businesses that are required to protect people’s privacy, that structure is a useful way to control data.

Prompt engineering has surpassed the experimental phase. Today, it’s become an integral component of the tools, platforms, and operational processes that drive enterprise-wide security, IT, HR, and other departments.

For CISOs, the ubiquity of structured prompts demands governance and usage policies because they are currently used to drive decision-making, surface intelligence, and summarise sensitive information within enterprise systems.

Below is an example of how prompt engineering is applied in the day-to-day operation of enterprises:

• Customer support automation: Structured prompts are used to determine how AI assistants respond to inbound requests, provide tone guidance, and facilitate escalation procedures. As such, the end-user receives a consistent, on-brand experience that scales without a manual review of each interaction.
• Internal knowledge retrieval: Employees use natural language processing (NLP) prompts to access answers to common questions from policy documents, onboarding guides, and internal wikis in under five seconds. This reduces the frustration for HR and operations teams, who receive direct, summarised answers to their questions quickly.
• Code generation: Developers rely on prompt-driven tools to generate code snippets, identify potential vulnerabilities, and refactor logic in real time. Using a well-defined prompt will limit the amount of unnecessary output (i.e., insecure/hallucinated code) that reaches a production environment.
• Security investigation assistance: SOC analysts use structured prompts to analyse phishing emails, collect relevant data from threat intelligence feeds, and collect indicators of compromise from high-volume alert queues. Precise prompt framing expedites triage time and surfaces the threats that truly matter.
• Fraud analysis: Fraud teams use prompt-based queries to analyse transaction data, identify behavioural anomalies, and generate plain-language summaries of suspicious activity. This methodology enables faster decision-making and produces cleaner documentation for regulatory/audit review.

Prompt engineering unlocks LLM productivity, but this flexibility creates vulnerabilities. When prompts are poorly designed, not properly governed, or left open to inputs that can’t be trusted, they become an attack surface that traditional security measures weren’t meant to stop.

Prompt Injection Attacks

A prompt-injection attack is when malicious instructions are hidden within content that the LLM is supposed to process. That content could be an email, a document, or a support ticket. The user may never see the added instructions, yet the model still executes them.

This kind of attack lets you steal data by changing models, giving yourself more access, and, in agentic environments, cross-domain prompt contamination, where one injected instruction can spread through connected tools and workflows. OWASP says that prompt injection is the most dangerous flaw in LLM apps, with an 84% success rate for attacks in agentic deployments.

For security teams, that means input validation and real-time output monitoring are not optional extras; they are must-have controls. For CISOs, prompt injection is a new attack surface that requires its own governance strategy rather than being added to existing ones.

Prompt Leakage and Data Exposure

Users often add private information in prompts without knowing how risky it is. A single request to an external LLM API can include private customer information, proprietary code, legal strategy, or internal financial information. Most workers don’t know where that prompt data goes, how long it stays there, or if it helps train future models.

API integrations make the problem worse. When enterprise tools connect to third-party LLM services, the data that goes through those connections is often not protected by the organisation’s current data loss prevention controls. That gap is big for compliance leaders. Data classification rules should cover more than just files and endpoints; they should also cover prompt content. And for legal teams, the risk of exposure to rules such as GDPR, HIPAA, and CCPA is very real if protected information appears in prompt logs.

Bias and Hallucination Risks

If the prompts are poorly made, they can exacerbate biases in a model’s training data. If a prompt doesn’t provide enough context, detail, or limits, the model fills in the blanks with patterns it learned from its training. These patterns could show past unfairness or wrong ideas.

Hallucinated outputs are different but just as problematic. LLMs give answers that sound confident, even when the information they are based on is incorrect or fabricated. In regulated sectors such as finance, healthcare, and law, executives who use AI-generated summaries need human oversight before making decisions based on them. Hallucination rates and prompt quality should be part of AI governance oversight for risk officers, along with access controls and data handling policies.

Jailbreaking

Jailbreaking attacks use carefully crafted inputs to bypass a model’s safety guardrails. Prompt injection usually hides instructions in external data, whereas jailbreaking works through the user interface by exploiting how a model processes and prioritises instructions.

The goal is to get the model to make outputs that it was told not to, such as harmful content, private system information, or limited functional behaviour. For security teams, jailbreaking is a constant threat that changes as the models it targets change.

System Prompt Extraction

System prompt extraction is an attack in which a user tricks an LLM into giving up its own configuration instructions. This is different from data leakage through user inputs. System prompts often include rules for security, business logic, persona settings, and operational limits that apply only to that system.

Attackers can see exactly how a deployed AI system is supposed to work and where its limits are because of their exposure. For developers making AI-powered products or tools for their own use, enforced instruction hierarchies and model isolation are the best ways to protect against this type of attack.

These two terms share a common concept of using “prompts” to function; however, they represent polar extremes in security.

• Prompt engineering is an intentional, deliberate process by which developers, security experts, and AI development teams create well-defined, structured input formats for a model to follow. In turn, this engineering ensures that the model produces the desired output in line with the developer’s intent and the organisation’s policies. When executed properly, this can be a valid discipline and a form of defence against unintended use of the model.
• Prompt injection is an attack vector against a model. In this case, an attacker will create or embed specific input formats into a model to either manipulate the model’s behaviour (i.e., cause it to produce unauthorised output) or circumvent its inherent guardrails and constraints.

This distinction is important because the same technical expertise that enables a team to execute prompt engineering effectively can also be used to design and detect prompt-injection attacks against a model. As a core aspect of developing AI systems that are capable and resilient, security teams should understand both the positive and negative aspects of the model’s instruction hierarchy.

Counterintuitive to most cybersecurity protocols, technology doesn’t come first when it comes to prompt security. It all starts with policy. When CISOs are building an AI risk management framework, the first step is to identify which prompts are being used, who is using them, on which systems, and for what purposes. Controls at the technical layer don’t have anything solid to stand on without that visibility.

Governance Controls

These are the decisions that the company makes that affect how AI tools are used throughout the business.

• Policies for using AI: Specify which AI tools are allowed, what data can be used in prompts, and what use cases need extra review or approval.
• Prompt documentation standards: Teams must write down the system prompts they use in production workflows, along with their intended use and any known problems.
• Role-based access to AI tools: Limit access to AI features that are high-risk or sensitive based on the person’s job and the level of data classification.
• Logging and auditing prompts: Keep a full record of all prompt activity across all enterprise AI deployments. This is not just a best practice for compliance teams; it’s becoming a legal requirement under laws like the EU AI Act.

Technical Controls

Technical controls operate at the model and integration layers, where governance policy can be implemented.

• Input validation: Check all user-provided inputs for known attack signatures, attempts to override instructions, and unusual query patterns before they get to the model.
• Output filtering: Continuously monitor model responses in real time to make sure they don’t share sensitive data, break the rules, or include prohibited content.
• Retrieval isolation: In retrieval-augmented generation (RAG) architectures, limit model access to pre-approved, regularly checked data sources to reduce the risk of hallucinations and cross-domain contamination.
• API monitoring: Keep an eye on the number, types, and unusual behaviour of all LLM API calls to find possible misuse or attempts to steal data. For SOC teams, this is a natural next step in their current workflows for monitoring networks and applications.
• Data loss prevention integration: Link DLP security controls directly to AI tool traffic so that sensitive data in prompts is flagged or blocked before it leaves the company’s network.

Human Controls

Organisations planning to scale their AI usage need to ensure their employees are properly trained and prepared for responsible use.

• Educate employees on AI usage best practices: Provide operational teams with clear, useful instructions on the type of information that should never be included in prompts and what tools are permitted for use in a responsible way.
• Awareness of prompt injection: Train employees to be aware that the materials they request AI to process may contain unknown instructions or injected details that alter how the model functions.
• Escalation protocols: Streamlines reporting of abnormal AI behaviour, such as suspicious outputs or potential prompt-based incidents, to security teams.

There are now automated optimisation tools for prompts, which have become a standard component of enterprise AI development. Tools for testing, versioning, and life cycle management of prompts are giving AI teams the capacity to systematically assess and optimise prompts as opposed to using a “hit-or-miss” approach based on experience. For many AI teams that support production deployments, a standardised prompt library and version control are now infrastructure requirements.

Organisations are also integrating prompt templates into SaaS platforms. Many CRMs, productivity suites, human resource systems, and security solutions include pre-built prompt configurations that the average user may never see or review. As such, security architects need to consider the impact of these hidden prompts on their organisation.

The emergence of autonomous AI agents (i.e., systems that generate and execute their own prompts across connected tools without needing to receive instructions from humans) raises the bar even higher. The autonomy inherent in these systems expands an organisation’s AI attack surface, as one compromised agent can initiate a series of unintended actions across multiple systems.

Regulatory bodies are starting to take notice. The EU AI Act and SEC Cyber Disclosure Rules introduce legal obligations around AI transparency, audit trail integrity, and adversarial robustness. Simultaneously, research on adversarial prompt attacks is increasing at a rapid pace, with red teams stress-testing enterprise AI systems in much the same way they test applications and networks. Organisations that view prompt security as a research-grade discipline will be well-positioned for regulatory compliance requirements once they come into full force.

Prompt engineering is becoming a permanent part of business processes, and the associated risks require a security plan that focuses on people, data, and behaviour. Proofpoint helps businesses protect themselves against AI-generated phishing, detect threats that use prompt injection to bypass email and collaboration channels, and prevent sensitive data from leaving the company through unauthorised AI interactions. Proofpoint gives security teams the visibility and control they need to use AI safely and responsibly by combining identity threat detection and data loss prevention with a human-centred approach to AI risk management. Contact Proofpoint to learn more.