Tag: AI

A step-by-step guide

OpenAI Assistant API

OpenAI has recently introduced new features that showcase an agent-like architecture, such as the Assistant API. According to OpenAI:

The Assistants API allows you to build AI assistants within your own applications. An Assistant has instructions and can leverage models, tools, and files to respond to user queries. The Assistants API currently supports three types of tools: Code Interpreter, File Search, and Function calling.

While these advancements are promising, they still lag behind LangChain. LangChain enables the creation of agent-like systems powered by LLMs with greater flexibility in processing natural language input and executing context-based actions.

However, this is only the beginning.

At a high level, interaction with the Assistant API can be envisioned as a loop:

• Given a user input, an LLM is called to determine whether to provide a response or take specific actions.
• If the LLM’s decision suffices to answer the query, the loop ends.
• If an action leads to a new observation, this observation is included in the prompt, and the LLM is called again.
• The loop then restarts.

Unfortunately, despite the announced advantages, I found the documentation for the API to be poorly done, especially regarding interactions with custom function calls and building apps using frameworks like Streamlit.

In this blog post, I will guide you through building an AI assistant using the OpenAI Assistant API with custom function calls, paired with a Streamlit interface, to help those interested in effectively using the Assistant API.

Use case: Tax Computation Assistant

In this blog post, I will demonstrate a simple example: an AI assistant capable of calculating tax based on a given revenue. Langchain users can easily come into mind implementing this by creating an agent with a “tax computation” tool.

This tool would include the necessary computation steps and a well-designed prompt to ensure the LLM knows when to call the tool whenever a question involves revenue or tax.

However, this process is not exactly the same with the OpenAI Assistant API. While the code interpreter and file search tools can be used directly in a straightforward manner according to OpenAI’s documentation, custom tools require a slightly different approach.

assistant = client.beta.assistants.create(
  name="Data visualizer",
  description="You are great at creating beautiful data visualizations. You analyze data present in .csv files, understand trends, and come up with data visualizations relevant to those trends. You also share a brief text summary of the trends observed.",
  model="gpt-4o",
  tools=[{"type": "code_interpreter"}],
)

Let’s break it down step by step. We aim to:

1. Define a function that computes tax based on given revenue.
2. Develop a tool using this function.
3. Create an assistant that can access this tool and call it whenever tax computation is needed.

Tax Computation Function for Assistant Integration

Please note that the tax computation tool described in the following paragraph is designed as a toy example to demonstrate how to use the API discussed in the post. It should not be used for actual tax calculations.

Consider the following piecewise function, which returns the tax value for a given revenue. Note that the input is set as a string for simpler parsing:

def calculate_tax(revenue: str):
    try:
        revenue = float(revenue)
    except ValueError:
        raise ValueError("The revenue should be a string representation of a number.")

    if revenue <= 10000:
        tax = 0
    elif revenue <= 30000:
        tax = 0.10 * (revenue - 10000)
    elif revenue <= 70000:
        tax = 2000 + 0.20 * (revenue - 30000)
    elif revenue <= 150000:
        tax = 10000 + 0.30 * (revenue - 70000)
    else:
        tax = 34000 + 0.40 * (revenue - 150000)

    return tax

Next, we define the assistant:

function_tools = [
    {
        "type": "function",
        "function": {
            "name": "calculate_tax",
            "description": "Get the tax for given revenue in euro",
            "parameters": {
                "type": "object",
                "properties": {
                    "revenue": {
                        "type": "string",
                        "description": "Annual revenue in euro"
                    }
                },
                "required": ["revenue"]
            }
        }
    }
]
# Define the assistant
assistant = client.beta.assistants.create(
    name="Assistant",
    instructions="",
    tools=function_tools,
    model="gpt-4o",
)

Now, the essential point:

How does the assistant use the function when “calculate_tax” is called? This part is poorly documented in the OpenAI assistant, and many users might get confused the first time using it. To handle this, we need to define an EventHandler to manage different events in the response stream, specifically how to handle the event when the “calculate_tax” tool is called.

    def handle_requires_action(self, data, run_id):
        tool_outputs = []

        for tool in data.required_action.submit_tool_outputs.tool_calls:
            if tool.function.name == "calculate_tax":
                try:
                    # Extract revenue from tool parameters
                    revenue = ast.literal_eval(tool.function.arguments)["revenue"]
                    # Call your calculate_tax function to get the tax
                    tax_result = calculate_tax(revenue)
                    # Append tool output in the required format
                    tool_outputs.append({"tool_call_id": tool.id, "output": f"{tax_result}"})
                except ValueError as e:
                    # Handle any errors when calculating tax
                    tool_outputs.append({"tool_call_id": tool.id, "error": str(e)})
        # Submit all tool_outputs at the same time
        self.submit_tool_outputs(tool_outputs)

The code above works as follows: For each tool call that requires action:

• Check if the function name is “calculate_tax”.
• Extract the revenue value from the tool parameters.
• Call the calculate_tax function with the revenue to compute the tax. (This is where the real interaction happens.)
• After processing all tool calls, submit the collected results.

Talking to the assistant

You can now interact with the assistant following these standard steps documented by OpenAI (for that reason, I will not provide many details in this section):

1. Create a thread: This represents a conversation between a user and the assistant.
2. Add user messages: These can include both text and files, which are added to the thread.
3. Create a run: Utilize the model and tools associated with the assistant to generate a response. This response is then added back to the thread.

The code snippet below demonstrates how to run the assistant in my specific use case: The code sets up a streaming interaction with an assistant using specific parameters, including a thread ID and an assistant ID. An EventHandler instance manages events during the stream. The stream.until_done() method keeps the stream active until all interactions are complete. The with statement ensures that the stream is properly closed afterward.

  with client.beta.threads.runs.stream(thread_id=st.session_state.thread_id,
                                         assistant_id=assistant.id,
                                         event_handler=EventHandler(),
                                         temperature=0) as stream:
        stream.until_done()

Streamlit interface

While my post could end here, I’ve noticed numerous inquiries on the Streamlit forum (like this one) where users struggle to get streaming to work on the interface, even though it functions perfectly in the terminal. This prompted me to delve deeper.

To successfully integrate streaming into your app, you’ll need to extend the functionality of the EventHandler class mentioned earlier, specifically focusing on handling text creation, text deltas, and text completion. Here are the three key steps required to display text in the Streamlit interface while managing chat history:

1. Handling Text Creation (on_text_created): Initiates and displays a new text box for each response from the assistant, updating the UI to reflect the status of preceding actions.
2. Handling Text Delta (on_text_delta): Dynamically updates the current text box as the assistant generates text, enabling incremental changes without refreshing the entire UI.
3. Handling Text Completion (on_text_done): Finalizes each interaction segment by adding a new empty text box, preparing for the next interaction. Additionally, it records completed conversation segments in chat_history.

For instance, consider the following code snippet for managing text deltas:

def on_text_delta(self, delta: TextDelta, snapshot: Text):
    """
    Handler for when a text delta is created
    """
    # Clear the latest text box
    st.session_state.text_boxes[-1].empty()
    
    # If there is new text, append it to the latest element in the assistant text list
    if delta.value:
        st.session_state.assistant_text[-1] += delta.value
    
    # Re-display the updated assistant text in the latest text box
    st.session_state.text_boxes[-1].info("".join(st.session_state["assistant_text"][-1]))

This code accomplishes three main tasks:

• Clearing the Latest Text Box: Empties the content of the most recent text box (st.session_state.text_boxes[-1]) to prepare it for new input.
• Appending Delta Value to Assistant Text: If new text (delta.value) is present, it appends this to the ongoing assistant text stored in st.session_state.assistant_text[-1].
• Re-displaying Updated Assistant Text: Updates the content of the latest text box to reflect the combined content of all assistant text accumulated so far (st.session_state[“assistant_text”][-1]).

Conclusion

This blog post demonstrated how to use the OpenAI Assistant API and Streamlit to build an AI assistant capable of calculating tax.

I did this simple project to highlight the capabilities of the Assistant API, despite its less-than-clear documentation. My goal was to clarify ambiguities and provide some guidance for those interested in using the Assistant API. I hope this post has been helpful and encourages you to explore further possibilities with this powerful tool.

Due to space constraints, I have tried to avoid including unnecessary code snippets. However, if needed, please visit my Github repository to view the complete implementation.

---

Creating an Assistant with OpenAI Assistant API and Streamlit was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introducing a high-level capabilities engineering framework for AI Agents

Introduction

In my recent article ‘From Prompt Engineering to Agent Engineering’ I proposed a framework for AI Agent Engineering that introduces a mental model for approaching the design and creation of AI agents. To recap the framework proposes the following structure:

• AI agents are given Job(s)
• Job(s) require Action(s) to complete
• Performing Action(s) requires Capabilities
• Capabilities have a Required Level of Proficiency
• The Required Level of Proficiency requires Technologies and & Techniques
• Technologies and Techniques require Orchestration

If you missed that article or need to refer back to it, you can find it here.

Although straightforward, on a deeper level, the framework tackles expansive topics and ideas. Drilling into the concepts surfaced by the broader framework is a substantial endeavor, and in this article, we continue our work by focusing on an AI Agent Capabilities Engineering Framework. The approach to this framework relies on a taxonomically oriented mindset, that extends concepts primarily rooted in cognitive and behavioral sciences.

Cognitive and Behavioral Science Foundations

As I have mentioned in other writings, throughout the history of human tool & technology development we have often used ourselves as the inspiration or model for what we are trying to build. A topical example of this in AI itself is the neural network which was inspired by the human brain. In an effort to build a framework for AI Agent Capabilities it seems natural then to turn to cognitive and behavioral sciences for inspiration, guidance and extension of useful concepts. Let’s first get a high-level grasp on what these sciences entail.

Cognitive Science

Cognitive science is the interdisciplinary study of the mind and its processes, encompassing areas such as psychology, neuroscience, linguistics, and artificial intelligence. It provides critical insights into how humans perceive, think, learn, and remember.

Behavioral Science

Behavioral science is an interdisciplinary field that studies cognitive processes and actions, often considering the behavioral interaction between individuals and their environments. It includes disciplines such as psychology, sociology, anthropology, and economics.

As the expectations for what AI agents can accomplish continue to reach new heights, grounding our capabilities framework in cognitive and behavioral theories should give us a solid foundation to begin to meet those expectations and help us unlock a future where AI agents are equipped to perform complex jobs with human-like proficiency.

AI Agent Capabilities Framework

Before we dive into the minutiae let’s consider on a high-level how we might categorize the so-called ‘capabilities’ that power the ‘actions’ our agents need to take in an effort to perform their ‘jobs’. I propose that in general they fall into the categories of Perceiving, Thinking, Doing and Adapting. From there we can move on to identifying example capabilities in these categories on a more granular level. Although the resulting framework is categorically cohesive, bear in mind that the implied relationships between granular capabilities and categories are approximate. In reality the capabilities are heavily intertwined throughout the framework and trying to model this multi-dimensionality does not feel particularly useful at this stage. Below is a visual representation of the major categories and sub-categories that make up the framework without the categorical alignments that you will see shortly.

While our primary focus is driven by LLM-centered AI Agent Engineering, to future-proof and allow for the expansion of these frameworks into the realm of embodied AI and robots, we incorporate concepts that would be applicable in these settings as well.

Finally we do not deal with autonomy explicitly in the framework as it is more appropriately an overarching characteristic for a given agent or one of more of its capabilities. That said, autonomy is not necessarily a requirement that must be met for an agent to be effective in its given job(s).

With that foundation in place, let’s expand out the entire framework.

Perceiving

Encompasses the capabilities through which Agents acquire, interpret, and organize sensory information from the environment. It involves the detection, recognition and understanding of the appropriate stimuli, enabling Agents to perform as expected. Examples of granular capabilities include:

• Visual Processing: Image and object recognition and processing.
• Textual Data Processing: Text recognition and processing
• Auditory Processing: Speech and sound recognition and processing
• Haptic Processing: Touch recognition and processing.
• Olfactory and Gustatory Processing: Scent recognition and processing.
• Sensory Integration: Combining data from different sensory inputs for cohesive understanding

Thinking

Refers to the capabilities that enable Agents to process information, form concepts, solve problems, make decisions, and apply knowledge. Examples of granular capabilities include:

Contextual Understanding and Awareness

• Contextual Awareness and Understanding: Recognizing and comprehending situational, environmental, spatial and temporal context.
• Self-Awareness and Metacognition: Self-awareness, self-monitoring, self-evaluation, metacognitive knowledge

Attention and Executive Functions

• Selective Attention: Focusing on relevant data while filtering out irrelevant information
• Divided Attention: Managing and processing multiple tasks or sources of information simultaneously
• Sustained Attention: Maintaining focus and concentration over prolonged periods
• Planning: Formulating a sequence of actions or strategies to achieve a specific goal.
• Decision Making: Analyzing information, assessing options, and choosing the best course of action.
• Inhibitory Control: Suppression of inappropriate or unwanted behaviors or actions.
• Cognitive Flexibility: Switching between thinking about two different concepts or thinking about multiple concepts simultaneously
• Emotional Regulation: Managing and responding to emotional experiences with appropriate emotions

Memory

• Short-Term Memory: Holding and manipulating information temporarily
• Working Memory: Actively processing and manipulating information
• Long-Term Memory: Storing and retrieving information over extended periods

Reasoning and Analysis

• Logical Reasoning: Drawing conclusions based on formal logic and structured rules
• Probabilistic Reasoning: Making predictions and decisions based on probability and statistical models
• Heuristic Reasoning: Applying rules of thumb or shortcuts to find solutions
• Inductive Reasoning: Making generalizations from specific observations
• Deductive Reasoning: Drawing specific conclusions from general principles or premises
• Abductive Reasoning: Forming hypotheses to explain observations
• Analogical Reasoning: Solving problems by finding similarities to previously encountered situations
• Spatial Reasoning: Understanding and reasoning about spatial relationships

Knowledge Utilization and Application

• Semantic Knowledge: Acquiring and applying general world knowledge and features that make up concepts
• Episodic Knowledge: Acquiring and using knowledge of specific events and experiences
• Procedural Knowledge: Knowing how to perform tasks and actions efficiently
• Declarative Knowledge: Acquiring and using factual information
• Language Comprehension: Understanding and interpreting language

Social and Emotional Intelligence

• Emotion Recognition: Detecting and interpreting emotions
• Social Interaction: Engaging with humans or other agents in socially appropriate ways
• Empathy: Understanding and responding to the emotional states of others
• Theory of Mind: Inferring and understanding mental states, intentions, and beliefs
• Social Perception: Recognizing and understanding social cues and context
• Relationship Management: Managing and nurturing long-term relationships

Creativity and Imagination

• Idea Generation: Producing new and innovative ideas
• Artistic Creation: Creating original artistic works such as music, visual art, and literature
• Imaginative Thinking: Envisioning and articulating new possibilities and scenarios beyond current reality

Doing

Description: Involves the capabilities through which Agents interact with the environment and perform tasks. It includes both digital and physical actions. This category of capabilities also covers communication and interaction, enabling the Agent to engage meaningfully with users and other systems. Examples of granular capabilities include:

• Digital Action Execution: Performing specific digital actions, including output generation, automation, problem-solving actions, decision implementation, and response actions.
• Physical Action Execution: Planning, initiating, and adjusting movements, integrating sensory information with motor actions, grasping and handling objects, and learning and adapting new motor skills.
• Human Communication and Interaction: Engaging in meaningful dialogues with users, handling multiple languages, and maintaining the context of conversations.
• Agent and Systems Communication and Interaction: Effectively communicating and coordinating with other AI agents and systems, using protocols and interfaces to exchange information, synchronize actions, and maintain interaction context across platforms.

Adapting

Description: Refers to the capabilities that allow Agents to adjust and evolve their behaviors, processes, and emotional responses based on new information, experiences, and feedback. To be clear, we are focused here on adaptation and learning capabilities of the agent in its operative state and not learning that happens within the context of enabling its foundational capabilities. In our framework that will be the domain of Tools & Techniques. Examples of granular capabilities include:

Learning

• Cognitive Learning: Acquiring knowledge through cognitive processes
• Imitation Learning: Acquiring new skills and behaviors by observing and replicating actions
• Experiential Learning: Learning through experience and reflection

Adaptation and Evolution

• Behavioral Adaptation: Adjusting behaviors in response to feedback or environmental changes
• Cognitive Adaptation: Modifying cognitive processes based on new information
• Emotional Adaptation: Adjusting emotional responses based on experiences and context
• Motor Adaptation: Adapting motor skills through practice and feedback
• Social Adaptation: Modifying social behaviors based on social cues and interactions
• Evolution: Long-term changes and improvements in behaviors and cognitive processes over time

Since this is intended to be an article and not a book, we won’t go into a detailed discussion on each of these example granular level capabilities. As much as I would like to believe that this is exhaustive, it’s at best a good start. Through iteration and feedback we will surely revise it, improve it and move towards a stable framework that might then be suitable for broader adoption.

Let’s turn now to some examples that illustrate the practical application of the framework and how it can be valuable in an agent engineering setting.

The AI Agent Capabilities Framework in Practice

The practical application of the AI Agent Capabilities Framework involves leveraging its structured concepts, rooted in cognitive and behavioral science, to facilitate the design thinking process. Given the diversity in how we will envision and articulate desired capabilities for our agents, this framework helps establish a common ground, fostering consistency and comprehensiveness in capability design and engineering. This will be particularly valuable as the expectation for the sophistication level of our AI Agent’s capabilities continues to grow. Let’s explore an example:

AI Agent for Customer Support

Let’s consider an AI agent whose job is to provide customer support and personalized product recommendations. Armed with the framework, let’s aim for a higher fidelity job and scenario description that paints a more vivid picture.

Job: Deliver exceptional and empathetic customer support and product recommendations, while proactively predicting sales trends and incorporating granular contextual elements for highly personalized interactions.

Scenario: It is a bustling online customer service environment, and our AI agent is tasked with not only resolving customer queries and making product recommendations but also enhancing the overall customer experience by anticipating needs and personalizing interactions. It is a job that encompasses a broad spectrum of actions and capabilities. A few years back, building some of these capabilities would have been completely out of reach. Can the capabilities for this job be effectively articulated using our AI Agent Capabilities Framework in an effort to ascertain its feasibility? Let’s take a closer look while bearing in mind that the below outline is not intended to be comprehensive:

Actions Required:

• Understand and interpret customer queries.
• Provide accurate and helpful responses.
• Escalate issues when appropriate.
• Predict sales trends based on customer interactions.
• Make product recommendations.

Capabilities Required:

1. Perception

• Textual Data Processing: Recognize and understand written customer queries, including complex sentences and slang.
• Auditory Processing: Transcribe and comprehend spoken queries, even in noisy environments.
• Visual Processing: Interpret visual cues and body language during video support sessions.

2. Cognition

Contextual Understanding and Awareness:

• Temporal Awareness: Recognize seasonal trends and peak periods.
• Location Awareness: Understand geolocation data.
• Personal Context Awareness: Understand individual customer, their history and preferences.

Memory:

• Short-Term Memory: Retain recent interactions to maintain context.
• Long-Term Memory: Utilize past interactions for context.

Reasoning and Analysis:

• Probabilistic Reasoning: Identify patterns in customer interactions to predict future behavior.
• Deductive Logic: Apply logical frameworks to troubleshoot issues.
• Behavioral Analysis: Understand and interpret patterns in customer behavior.
• Trend Analysis: Understand current market trends and seasonal data.

Knowledge Utilization and Application

• Semantic Knowledge: Apply general world knowledge to understand and respond to queries.
• Episodic Knowledge: Use specific events and past experiences for relevant support.
• Declarative Knowledge: Access factual information for accurate responses.

Social and Emotional Intelligence

• Emotion Recognition: Detect and interpret customer emotions.
• Social Interaction: Engage with customers in a socially appropriate manner.
• Theory of Mind: Infer customer needs and preemptively offer solutions.
• Relationship Management: Build rapport with customers to foster loyalty.

Creativity and Imagination

• Imaginative Thinking: Envision new possibilities beyond current issues.

Action

Digital Interactions:

• Output Generation: Produce quick, accurate, and contextually appropriate responses.
• Product Recommendation Generation: Suggest products based on customer preferences, and other relevant analyses.

Human Communication and Interaction:

• Conversation Continuity: Maintain context over multiple interactions.

Agent and Systems Communication:

• Inter-Agent Coordination: Communicate with other AI systems to synchronize actions and share insights.

Adaptation

Learning:

• Experiential Learning: Continuously improve understanding of customer behavior.

Adaptation:

• Behavioral Adaptation: Adjust interaction style based on feedback.
• Cognitive Adaptation: Update knowledge with new information.
• Emotional Adaptation: Modify emotional responses.

Some of these insights might be a bit surprising. For example, should AI Agents have relationship management as a capability? Or how about AI Agents that are pseudo-embodied on screen and are capable of observing and responding to a whole new array of data points they can “observe” via video? For certain, there are a plethora of privacy concerns and issues to contend with, but not a concept that we should rule out entirely.

Creating Capabilities Through Technologies and Techniques

Although this article will not focus on an evaluation of Technologies and Techniques to enable capabilities we should address the question that naturally emerges after going through the above exercise. Don’t LLMs give us the tools for most of these capabilities right out of the box?

Although LLMs have certainly advanced the state-of-the-art by leaps and bounds, the simple answer is, no. And in cases like the capabilities for reasoning and analysis, even though LLMs can simulate what looks like reasoning or analysis quite impressively, it falls far short of the human capabilities for such. In short, LLMs provide a not entirely reliable but powerful shortcut to enabling many of these capabilities. They represent a very consequential evolutionary step in intelligence and interaction technologies and their unprecedented adoption helps explain why there is so much excitement around the idea of Artificial General Intelligence (AGI). Although the definition of what it actually entails is the subject debate, if achieved, it could be the go to technology solution for enabling many of the cognitive/behavioral capabilities described above.

Conclusion

I hope you find the AI Agent Capabilities Engineering framework to be an insightful approach for defining your AI agents’ capabilities. By integrating concepts from cognitive and behavioral sciences, this framework aims to guide the development of the capabilities needed for AI agents to perform complex tasks. The framework is relatively dense and will surely evolve over time. The key takeaway at this stage is the mental model centered around Perceiving, Thinking, Doing, and Adapting. These four high-level concepts on their own provide a very robust foundation for organizing and developing Agent capabilities effectively.

Thanks for reading and stay tuned for future refinements of this framework and extension of other aspects of the AI Agenting Engineering framework. If you would like to discuss the framework or other topics I have written about further, do not hesitate to connect with me on LinkedIn.

Unless otherwise noted, all images in this article are by the author.

---

AI Agent Capabilities Engineering was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post presents a solution that uses a generative artificial intelligence (AI) to standardize air quality data from low-cost sensors in Africa, specifically addressing the air quality data integration problem of low-cost sensors. The solution harnesses the capabilities of generative AI, specifically Large Language Models (LLMs), to address the challenges posed by diverse sensor data and automatically generate Python functions based on various data formats. The fundamental objective is to build a manufacturer-agnostic database, leveraging generative AI’s ability to standardize sensor outputs, synchronize data, and facilitate precise corrections.

Name entity recognition (NER) is the process of extracting information of interest, called entities, from structured or unstructured text. Manually identifying all mentions of specific types of information in documents is extremely time-consuming and labor-intensive. Some examples include extracting players and positions in an NFL game summary, products mentioned in an AWS keynote transcript, or […]

In this post, we explore a comprehensive solution for addressing the challenges of securing a virtual travel agent powered by generative AI. We provide an end-to-end example and its accompanying code to demonstrate how to implement prompt engineering techniques, content moderation, and various guardrails to make sure the assistant operates within predefined boundaries by relying on Guardrails for Amazon Bedrock. Additionally, we delve into monitoring strategies to track the activation of these safeguards, enabling proactive identification and mitigation of potential issues.