Building Safer AI Agents: 8 Essential Concepts for Tool Execution Safety and Reliability

Views: 2

AI agents are evolving beyond simple question-answering systems. They can now interact with APIs, execute code, retrieve information, communicate with external services, and perform actions on behalf of users.

This capability makes AI agents useful, but it also introduces a new set of engineering challenges.

Consider an agent that can send emails, update customer records, initiate payments, or interact with business applications. What happens if it misunderstands a request? What if it selects the wrong tool? What if it repeats an operation after a timeout? What if a malicious instruction attempts to manipulate its behavior?

These are not merely language-model problems. They are application architecture, security, and reliability problems.

While developing a tool-using AI agent in Python, I have been implementing and testing several mechanisms designed to address these challenges.

Here are eight concepts I believe every developer should consider when building an AI agent that interacts with external tools.

1. System Prompt Hardening

The system prompt is one of the first places to establish the agent’s behavioral boundaries.

It should explain:

  • Which tools are available.
  • What each tool is intended to do.
  • When the agent should use a tool.
  • When it should avoid using a tool.
  • What information it must obtain before requesting an action.
  • How it should communicate tool results and failures.

For example, an agent with access to a messaging tool should understand that it must not send a message merely because a user mentions someone or discusses a possible message.

It should distinguish between drafting a message and actually sending it.

However, there is an important limitation: system prompts are not security controls by themselves.

An LLM can misunderstand or fail to follow instructions. Therefore, important restrictions must also be enforced in application code.

Think of the system prompt as behavioral guidance, not your authorization mechanism.

2. Precise Tool Descriptions

Tool descriptions directly influence how an LLM selects and uses available tools.

A vague description such as:

“Send a message.”

does not provide enough context.

A better description would explain:

  • The tool’s intended purpose.
  • The required recipient and message content.
  • The circumstances in which it should be used.
  • Important restrictions.
  • An example of an appropriate use case.

For example:

“Send a message to a specified recipient after the user has explicitly requested the action. Requires a recipient and message body. Do not use this tool to draft messages or send messages based on assumptions.”

Good tool descriptions reduce ambiguity and help the model select the appropriate functionality.

But descriptions alone cannot guarantee safe behavior. The application must still validate arguments, check permissions, and enforce approval requirements.

3. Schema Validation: Never Blindly Trust LLM-Generated Arguments

When an LLM requests a tool, it also generates the arguments needed to execute that tool.

Those arguments should be treated as untrusted input.

For example, a calculator might expect:

  • expression: a mathematical expression represented as a string.

A messaging tool might expect:

  • recipient: a recipient identifier.
  • message: the message content.

What happens if the model sends an unexpected field, omits a required field, or provides an invalid value?

This is where structured validation becomes important.

In my implementation, I use Pydantic to define strict argument schemas and validate tool inputs before execution.

A simplified example:

from pydantic import BaseModel, ConfigDict

class MessageArguments(BaseModel):
    model_config = ConfigDict(extra="forbid")

    recipient: str
    message: str

The extra="forbid" setting rejects unexpected fields instead of silently accepting them.

Schema validation creates a predictable boundary between model-generated arguments and application execution.

It does not establish whether an action is authorized, but it ensures that the application receives arguments in an expected format.

4. Permission-Based Access Control

An AI agent should not automatically have access to every capability available in the application.

This is where the principle of least privilege becomes important.

Each tool should have an explicitly defined permission requirement.

For example:

ToolRequired permission
Calculatormath:calculate
Read current timesystem:read_time
Analyze texttext:analyze
Send messagemessage:send

The application checks whether the active policy allows the required permission before proceeding with execution.

This is an application-level enforcement mechanism.

It is particularly important because the LLM should not be responsible for deciding whether it is allowed to perform an action.

The model can request an operation, but the application makes the authorization decision.

5. Human-in-the-Loop Approval

Some operations are consequential enough that they should require explicit human approval.

Examples include:

  • Sending emails or messages.
  • Initiating payments.
  • Deleting records.
  • Publishing content.
  • Modifying important business data.

A useful implementation pattern is to separate tool selection from authorization and approval.

The agent requests a tool operation. The application validates the arguments, checks permissions, and determines whether human approval is required.

Only after approval should the sensitive action proceed.

This also means approval should be associated with the actual validated operation, not merely a vague conversational confirmation.

In my implementation, sensitive tools have a requires_approval property and use an application-level approval callback.

One important detail: approval should happen only once for a particular idempotent operation. Repeated requests should not create multiple approval prompts or execute the same operation again.

6. Timeouts and Controlled Retries

External tools and APIs can fail for many reasons:

  • Network interruptions.
  • Temporary service unavailability.
  • Rate limits.
  • Slow responses.
  • Connection timeouts.

A reliable agent should not wait indefinitely for a tool to respond.

Timeouts establish how long an operation is allowed to wait for a response.

Retries can help recover from temporary failures, but they must be implemented carefully.

For example, a temporary failure when retrieving information may be safe to retry.

A timeout while initiating a payment is different. The payment might have succeeded even if the application never received the response.

Blindly retrying such an operation could initiate the payment twice.

A better approach is to:

  1. Set appropriate timeouts.
  2. Distinguish transient failures from permanent failures.
  3. Use bounded retries.
  4. Apply exponential backoff where appropriate.
  5. Combine retries with idempotency for operations that may produce side effects.

Retries improve resilience, but they do not independently guarantee safe execution.

7. Idempotency and Concurrency Control

This is one of the most important concepts when building agents that perform consequential actions.

Imagine a user asks an agent to initiate a payment.

The application sends the request, but the response times out.

The agent tries again.

Without an appropriate mechanism, the same payment might be initiated twice.

Idempotency addresses this problem.

An idempotency key identifies a logical operation. When the same operation is received again with the same key, the application can return its previously recorded result rather than executing it again.

A simplified operation lifecycle might look like this:

New Request
     |
     v
Check Idempotency Key
     |
     +---- Existing Completed Operation
     |          |
     |          v
     |     Return Cached Result
     |
     +---- Existing In-Progress Operation
     |          |
     |          v
     |     Reject Duplicate Execution
     |
     +---- New Operation
                |
                v
          Claim Operation
                |
                v
          Execute Tool
                |
                v
          Store Result

But idempotency alone is not enough when multiple workers or concurrent requests are involved.

This is why operation ownership and leases are also useful.

A lease allows a worker to hold temporary ownership of an operation. Other workers should not execute the same operation while a valid lease is held.

If the worker crashes and the lease expires, the operation can be marked for recovery and inspection.

An important distinction: an expired lease does not prove that the external action failed. The action may have succeeded before the worker crashed.

For this reason, automatically repeating an uncertain external side effect can be dangerous.

Exactly-once execution across external systems is not something an application can simply guarantee through a local database. It requires cooperation from the external service or a reliable reconciliation mechanism.

8. Audit Logging and Recovery

A reliable agent should provide visibility into what happened during execution.

When something goes wrong, developers should be able to answer:

  • Which tool was requested?
  • What arguments were supplied?
  • Was the operation authorized?
  • Was human approval required?
  • Was approval granted or rejected?
  • How many execution attempts occurred?
  • Did the operation complete?
  • Was the result stored?
  • Does the operation require recovery?

Structured audit logging helps answer these questions.

In my implementation, execution events are recorded in JSON Lines format, while SQLite maintains operation state for idempotency and recovery.

These are related but different responsibilities:

  • Audit logging provides a record of important events.
  • Operation state tracks the lifecycle of an individual operation.
  • Recovery mechanisms identify incomplete or uncertain operations that need further attention.

This observability becomes particularly valuable when debugging failures in production systems.

Putting Everything Together

These concepts should not be implemented as isolated features.

They work together as layers of an execution architecture.

A simplified tool-execution flow looks like this:

User Request
     |
     v
LLM / Agent
     |
     v
Tool Selection
     |
     v
Tool Registry
     |
     v
Permission Check
     |
     v
Argument Validation
     |
     v
Idempotency Check and Operation Claim
     |
     v
Human Approval (if required)
     |
     v
Tool Execution with Timeout / Retry
     |
     v
Operation Finalization
     |
     v
Audit Logging and Recovery

The LLM proposes an action, but the application controls whether and how that action is executed.

That separation is fundamental to building safer and more reliable tool-using agents.

What I Have Implemented So Far

I have been developing these concepts incrementally in a Python-based tool-using AI agent.

The implementation currently includes:

  • A provider-independent LLM interface.
  • Multiple tools and a centralized tool registry.
  • Pydantic-based argument validation.
  • Permission-based access control.
  • Human approval for sensitive tool operations.
  • Timeout and retry handling.
  • SQLite-backed idempotency.
  • Operation leases and recovery states.
  • Structured audit logging.
  • Automated tests covering duplicate requests, approval rejection, lease expiration, and execution failures.

The implementation is still an evolving learning and experimentation project, rather than a production-ready payment or messaging system.

The next step is to expand its adversarial testing and evaluate how reliably these mechanisms behave under different failure conditions.

Final Thoughts

Building an AI agent is not simply a matter of connecting an LLM to a collection of tools.

The real engineering challenge is controlling what happens when the model makes an incorrect decision, when an external service fails, when requests are repeated, or when someone attempts to exploit the system.

A useful principle to remember is:

Let the LLM decide what action to propose. Let the application decide whether that action is permitted, approved, and safe to execute.

These eight concepts provide a practical starting point for developers building more reliable tool-using AI agents.

What other mechanisms would you include in this architecture?

Further Reading

  1. OWASP: Excessive Agency — risks associated with excessive agent permissions and autonomy.
  2. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses — adversarial evaluation of tool-using LLM agents.
  3. AWS Builders’ Library: Making Retries Safe with Idempotent APIs — practical idempotency and retry design.
  4. Pydantic Documentation — structured validation of application data.
  5. Python SQLite Documentation — persistent operation-state management.
  6. LiteLLM Documentation — provider-independent LLM integration.

Leave a Reply

Your email address will not be published. Required fields are marked *

Search