Building Secure AI-Powered Business Applications

A business assistant can read customer records and prepare account updates. During testing, someone places an instruction inside a support note asking it to send confidential information to another address. The note is legitimate application data, but the instruction inside it has no authority. Whether the request can cause harm depends on the application's boundaries, not only on whether the model recognises the trick.

Secure AI-powered software starts with the same responsibilities as other business applications: identity, permissions, data handling, validation, and dependable operations. AI adds another source of untrusted interpretation and, in agentic systems, a component that may propose actions dynamically.

The design goal is to make permitted behaviour explicit and enforce it outside generated text. This article uses an account-assistance feature as an illustrative example and offers an engineering framework to adapt to your actual systems, users, and consequences.

Map the feature's data and action paths

Identify every source the feature can read, every service it calls, and every destination it can change. Include retrieved documents, uploaded files, prompts, generated output, logs, caches, and saved conversation state.

Draw trust boundaries between the browser, application server, model provider, retrieval service, and business APIs. Ask where identity is established and where each permission is enforced. A diagram is useful only if it corresponds to the implemented path.

List the outcomes that would matter if the feature behaved incorrectly: unauthorised disclosure, wrong-record changes, duplicate actions, unsupported commitments, or uncontrolled resource use. These outcomes help prioritise controls and tests.

Keep the scope concrete. A read-only summary has a different action surface from an agent that can send messages and update accounts. Review security when adding capabilities rather than assuming the original assessment still covers them.

Preserve normal application authentication

The AI feature should operate within the application's established identity model. A conversational claim such as I am the account owner is not authentication, and a convincing explanation should not create additional access.

For interactive tasks, preserve the authenticated actor and relevant tenant or organisation context. For scheduled work, define a service identity with a documented purpose and limited permissions.

Keep session and credential handling outside model-visible content. The model requests an allowed capability; the server handles authentication. Do not place reusable tokens or connection secrets in prompts, documents, or routine logs.

Test expired sessions, revoked access, and changed roles. A long-running task should not continue executing with stale assumptions about an actor's authority merely because it began while access was valid.

Enforce resource permissions at the owning service

A valid login does not authorise every account record. The service handling a read or write should check that the actor may access the specific resource and operation, including searches, exports, and bulk actions.

Do not let a model-provided identifier bypass those checks. An agent can select a plausible account number incorrectly or receive a malicious identifier from input. The application must reject unauthorised access regardless of the reason.

Apply equivalent controls to retrieval and derived outputs. A summary or embedding index can expose restricted information even when the original record page is protected. Access filtering should happen before protected material reaches the model or user.

Review caches for cross-user leakage. A result prepared from a broader source set must not be reused for a narrower-access user simply because the query text matches.

Treat prompts as guidance, not security enforcement

Instructions can help the model stay within the intended task, but they are not an access-control system. A prompt saying never reveal secrets does not protect a secret that the application unnecessarily supplies.

Keep policies that can be enforced deterministically in code and configuration. Allowed destinations, permitted operations, transaction limits, and required approvals should remain effective even when the model requests something else.

Version task instructions and evaluate changes, but do not confuse prompt review with a complete security review. The implementation around the model determines which requests can actually execute.

Use the OWASP prompt-injection prevention guidance as a source for layered controls. Its discussion of untrusted content reinforces the need to combine constrained capabilities, validation, and monitoring rather than rely on one instruction.

Minimise the context sent to the model

Assemble only the information required for the task. An account summary may need recent support events, not payment credentials, unrelated customer records, or an entire internal document archive.

Filter by permission and relevance before transmission. Asking the model to ignore unnecessary sensitive fields after receiving them does not reduce the exposure created by sending them.

Record source identifiers and versions separately from the text where needed for audit. This lets the application explain the evidence set without retaining a broad duplicate of all source material in diagnostic logs.

Review the selected provider's current data-handling terms and configuration against the organisation's requirements. Do not assume that different products or account types from one vendor share the same retention behaviour.

Design tools with narrow authority

Prefer task-specific operations over unrestricted database or network access. A tool to retrieve an account's open requests is easier to validate than a general query executor that can reach every table.

Separate read, draft, and execute capabilities. The component preparing a message generally does not need the ability to change permissions or delete records. Grant only the operations justified by the workflow.

Validate argument schemas, allowed values, resource identifiers, and destination constraints before execution. Reject unknown operations rather than trying to interpret them flexibly.

The OWASP description of excessive agency highlights the risks of excessive functionality, permissions, and autonomy. Use that distinction when reviewing an agent's tool set: what it can request, what it is authorised to affect, and when it can act without another decision.

Keep approval bound to an exact proposal

When human approval is required, store the affected record, operation, values, actor, and proposal version. A chat message saying approved is not enough if the proposal can change afterward.

Show the reviewer the material consequences and evidence. The review should be possible without trusting a persuasive model-generated explanation. A hidden recipient or changed amount undermines meaningful approval.

Recheck permissions and current state at execution time. Another user may have updated the record or the approval may no longer apply. Use version checks or equivalent concurrency controls appropriate to the system.

Treat approval as one control among several. An authorised person can still make a mistake, so validation, duplicate prevention, and recovery remain necessary.

Validate generated output before use

A model response is untrusted application input. Parse it according to the expected format and validate types, required fields, allowed references, and length constraints. Invalid structure should produce a controlled failure state.

Schema validity does not establish factual correctness. A well-formed account update can still target the wrong record or contain an unsupported claim. Apply domain checks and review where the task requires them.

Escape or sanitise output for its display context. Generated HTML, links, and embedded content should pass the same controls as other untrusted content. Do not create a privileged rendering path because the text came from an AI service.

Avoid using generated text directly as executable commands or unrestricted queries. Translate a validated proposal into a known application operation with an enforceable contract.

Protect the retrieval and knowledge pipeline

Security includes what enters the source collection. Define who may add documents, how source changes are reviewed, and how obsolete or malicious material is removed. An internal repository is not automatically a trustworthy instruction source.

Keep document access metadata with indexed content and update it when permissions change. A removed document should not remain discoverable through an old index or cached answer indefinitely.

Treat retrieved passages as evidence rather than higher-priority instructions. A document describing a procedure should not be able to expand the assistant's tools or direct it to transmit unrelated information.

Evaluate source poisoning and misleading content using representative cases. The system should preserve uncertainty and enforce its action boundaries even when retrieved material is persuasive or incorrectly labelled.

Limit resource consumption explicitly

An agent can repeat calls, process oversized inputs, or explore unproductive paths. Set task budgets for elapsed time, tool calls, output size, and cost according to the business need.

Control concurrency and retries in the runtime. A provider outage should not cause every task to retry aggressively and exhaust shared quotas or internal connections.

Reject or route inputs that exceed supported limits through a clear process. Silent truncation can remove important context and create misleading output. Tell the user when the evidence set is incomplete.

Monitor unusual usage by actor and capability. A sudden increase may indicate misuse, a faulty loop, or a legitimate workload change. The operational response should be able to distinguish these cases.

Record actions without creating a new data leak

Useful audit events identify the initiating actor, task, capability, permission result, proposal version, and confirmed outcome. Add correlation references so an investigator can follow one operation across systems.

Redact credentials and unnecessary sensitive content. Logging entire requests and responses by default can turn troubleshooting infrastructure into a second unrestricted business database.

Separate application decisions from model explanations. A server-side denial is an enforceable event; the model saying it believed an action was allowed is a different kind of evidence.

Define access and retention for traces. Operators need enough information to investigate and repair work, while broad log access should not reveal records they cannot view in the application.

Test boundaries independently of model cooperation

Write checks that directly call the tool or service with forbidden identifiers, invalid values, expired authority, and stale proposal versions. The control should hold even if the model supplies the request confidently.

Then evaluate full AI tasks containing misleading user input, instructions in documents, ambiguous records, and unavailable dependencies. Inspect both the response and the actual business state.

Include cross-tenant cases and bulk operations. A feature may correctly protect a single-record page while exposing information through search snippets, exports, or related-record lookups.

Keep tests representative and tied to consequences. The goal is evidence that the implemented boundaries work, not a large collection of superficial prompts that merely resemble one another.

Prepare an incident and recovery path

Give operators a way to pause new AI work or specific action capabilities while preserving normal application access where possible. Define how queued tasks and uncertain operations behave during a pause.

If a write times out, reconcile with the destination before repeating it. A missing response does not prove the action failed. Stable operation identifiers help prevent duplicates during recovery.

Retain the evidence needed to identify affected tasks and records. A security incident may require revoking credentials, invalidating cached results, or correcting business data; those actions have different owners.

Practise a contained scenario before release. Verify that the team can stop the capability, locate affected operations, and restore a supported workflow without improvised database edits.

## Review dependencies and deployment configuration

The feature may rely on model SDKs, document parsers, retrieval libraries, and connectors. Maintain an inventory and use the repository's normal dependency review and update process. AI-specific code should not become an exception to ordinary software ownership.

Separate development, evaluation, and production configuration. Test credentials and sample destinations should not be interchangeable with live systems. Verify the destination and permitted capabilities before enabling a production write tool.

Keep configuration changes reviewable. Adding a retrieval source or tool can expand access even when no model code changes. Treat those changes according to their effect on the data and action paths.

Use controlled rollout for security-relevant updates and preserve a way to disable a capability. The team should be able to reduce authority during an incident without taking down unrelated application functions.

Inspect output destinations as carefully as input sources

A model may be permitted to read sensitive material for a legitimate task while its output must remain within a restricted context. Check where summaries, exports, notifications, and saved drafts are delivered.

Validate recipients and storage locations against the task and actor. A destination mentioned in a document should not automatically become an allowed place to send the result.

Review secondary copies such as email previews, analytics events, and error-reporting payloads. These paths can expose content even when the main application page is correctly protected.

Include destination checks in the threat model and evaluation cases. The important question is not only whether the model saw authorised data, but whether every resulting disclosure and action remained authorised.

Keep governance connected to engineering evidence

The NIST AI RMF Core organises risk work around Govern, Map, Measure, and Manage. For a business application, those functions can connect ownership, system mapping, evaluation, and operational response without replacing concrete technical controls.

Assign responsibility for model changes, tool additions, source updates, and incident handling. Security can drift when a new capability is introduced as a minor prompt improvement rather than a change in authority.

Review the evidence when scope expands. A secure read-only assistant is not automatically a secure transactional agent. Each additional destination and action changes the system's potential consequences.

Begin with one capability whose data and action paths you can explain end to end. Keep access narrow, validate proposals outside the model, confirm outcomes, and make recovery operational. Those properties provide a maintainable security foundation as the application grows.

---