Webhooks and GitHub Automations for Prompt Deployments​

11 March 2026

v0.94.0

You can now trigger automations when a prompt deployment happens in Agenta. Send the event to any HTTPS endpoint, or call GitHub directly with repository_dispatch or workflow_dispatch.

This makes it easier to connect prompt deployments to CI, repository sync jobs, and pull request workflows. If your GitHub workflow needs the latest prompt content, fetch it from Agenta during the run and commit the result back to your repo.

Learn more: Webhooks | GitHub Automations

Tool Integrations in the Playground​

27 February 2026

v0.87.0

You can now connect 150+ external tools to your prompts directly from the playground. Browse integrations like Gmail, Slack, Notion, Google Sheets, and GitHub. Authenticate with OAuth, attach tool actions to your prompt, and execute tool calls with one click. Use Google Sheets or Notion as data sources for RAG, send emails from your prompt, or automate developer workflows.

AI-Powered Prompt Refinement in the Playground​

25 February 2026

v0.84.0

You can now refine prompts with AI directly in the playground. Click the wand icon on any prompt, describe what you want to improve in plain English, and get back a refined version with a summary of changes. Each refinement builds on the last, so you can iterate. Toggle diff view to see exactly what changed, edit the result before applying, or use the quick "Optimize using best practices" shortcut.

Enterprise Compliance Features​

17 February 2026

v0.83.0

Agenta has new enterprise features. You can now create separate organizations for different teams or clients, each with its own billing, projects, and roles. We added SSO with any OIDC provider (Okta, Azure AD, Auth0, OneLogin, Google Workspace). You can enforce SSO-only for an org and disable password login. Domain verification lets you claim your company domain so new users with matching emails join automatically. We also launched a US region for customers who need their data to stay in the United States.

SSO, and domain verification are on Business and Enterprise plans. The US region is on all plans.

Folders for Prompt Organization​

4 February 2026

v0.82.0

Prompts multiply fast when you're building agents or managing multiple use cases. Finding the right one becomes guesswork.

You can now create folders and subfolders to organize prompts. Drag prompts between folders, create nested hierarchies, and search across everything. Folder URLs are shareable.

Onboarding Widget and Guided Walkthroughs​

29 January 2026

v0.81.1

New users now get an onboarding widget with guided walkthroughs. The widget appears in the sidebar and walks you through key features like the playground, evaluations, and observability. Each tour highlights UI elements as you go, so you learn by doing. Track your progress and revisit walkthroughs anytime.

Navigation Links from Traces to App/Environment/Variant​

28 January 2026

v0.81.0

You can now click directly from any trace to the application, variant, or environment that generated it. Links appear in both the trace table and drawer view. This makes debugging faster since you can jump straight to the configuration that produced a specific output.

To enable navigation links, store references in your traces using the Python SDK (ag.tracing.store_refs()) or OpenTelemetry span attributes. See the reference prompt versions guide for details.

Test Set Versioning and New Test Set UI​

20 January 2026

v0.74.0

Test sets now have versioning. Every edit, upload, or programmatic update creates a new version. Evaluations link to specific versions, so you can compare results knowing they used the same test data.

The test set UI is completely rebuilt. It handles hundreds of thousands of rows without slowing down. Editing is much easier, especially for chat messages. You can view and edit complex JSON directly, toggle between raw and formatted views, and choose whether columns store strings or JSON.

Playground UX Improvements​

13 January 2026

v0.73.0

Three quality-of-life improvements to the Playground: You can now see provider costs per million tokens directly in the model selection dropdown. You can run evaluations directly from the Playground without navigating to the evaluation menu. And you can collapse test cases to navigate large test sets more easily.

Chat Sessions in Observability​

9 January 2026

v0.73.0

You can now track multi-turn conversations with chat sessions. All traces with the same session ID are automatically grouped together, letting you analyze complete conversations instead of individual requests.

The new session browser shows key metrics like total cost, latency, and token usage per conversation. Open any session to see all traces with their parent-child relationships. This makes debugging chatbots and AI assistants much easier. Add session tracking with one line of code using either our Python SDK or OpenTelemetry.

Minor improvements:

• Added time filtering to the analytics dashboard. You can now view metrics for the last 6 hours, 24 hours, 7 days, or 30 days.
• Added the ability to batch delete multiple traces at once. Select traces using checkboxes and delete them in a single operation.

JSON Multi-Field Match Evaluator​

31 December 2025

v0.73.0

The new JSON Multi-Field Match evaluator validates multiple fields between JSON objects. Configure any number of field paths using dot notation, JSON Path, or JSON Pointer formats. Each field gets its own score (0 or 1), and an aggregate score shows the percentage of matching fields. This evaluator is ideal for entity extraction tasks like validating extracted names, emails, and addresses. The UI automatically detects fields from your test data for quick setup. This replaces the old JSON Field Match evaluator, which only supported single fields.

PDF Support in the Playground​

17 December 2025

v0.69.0

The Playground now supports PDF attachments for chat applications. You can attach PDFs by uploading files, providing URLs, or using file IDs from provider APIs. This works with vision-capable models and extends to evaluations and observability. You can now build and test document processing applications like invoice analysis or contract review.

Agenta Documentation MCP Server​

14 December 2025

v0.68.3

AI coding agents like Cursor, Claude Code, and VS Code Copilot can now access Agenta documentation directly through the Agenta MCP server. Connect your IDE to get instant answers about Agenta features, APIs, and code examples without leaving your editor. The server supports multiple clients and requires no authentication.

Provider Built-in Tools in the Playground​

11 December 2025

v0.66.0

You can now use provider built-in tools in the Playground. Add web search, code execution, file search, and Bash scripting tools directly to your prompts. Supported providers include OpenAI, Anthropic, and Gemini. Tools are saved with your prompt configuration and automatically used when you invoke prompts through the LLM gateway.

Projects within Organizations​

4 December 2025

v0.65.0

You can now create projects within an organization. This lets you divide your work between different AI products. Each project scopes its prompts, traces, and evaluations. Create a new project or navigate between projects directly from the sidebar.

Reasoning Effort Support in the Playground​

18 November 2025

v0.62.5

You can now configure reasoning effort for models that support this parameter, such as OpenAI's o1 series and Google's Gemini 2.5 Pro. The reasoning effort setting is part of your prompt template, making it available when you fetch prompts via the SDK or invoke them through Agenta as an LLM gateway.

Jinja2 Template Support in the Playground​

17 November 2025

v0.62.3

You can now use Jinja2 templates in your prompts. Jinja2 is available in both the Playground and in prompt management.

Learn more in our blog post or check the documentation.

Agenta Core is Now Open Source​

13 November 2025

We're open sourcing the core of Agenta under the MIT license. All functional features are now available to the community. This includes the evaluation system, prompt playground and management, observability, and all core workflows.

Development moves back to the public repository. We're building in public again. Only enterprise collaboration features like RBAC, SSO, and audit logs remain under a separate license.

Get started with the self-hosting guide. View the code and contribute on GitHub. Read why we made this decision at agenta.ai/blog/commercial-open-source-is-hard-our-journey.

Evaluation SDK​

12 November 2025

v0.62.0

You can now run programmatic evaluations of complex AI agents and workflows directly from code. The Evaluation SDK gives you full control over test data and evaluation logic. It works with agents built using any framework.

The SDK lets you create test sets in code or fetch them from Agenta. You can use built-in evaluators like LLM-as-a-Judge, semantic similarity, or regex matching. You can also write custom Python evaluators. The SDK evaluates end-to-end workflows or specific spans in execution traces. Evaluations run on your own infrastructure; results display in the Agenta dashboard.

Check out the Evaluation SDK documentation to get started.

Online Evaluation​

11 November 2025

v0.62.0

You can now automatically evaluate every request to your LLM application in production. Online Evaluation helps you catch hallucinations and off-brand responses as they happen. You no longer need to discover problems through user complaints.

You can configure evaluators like LLM-as-a-Judge with custom prompts. Set sampling rates to control costs. Create evaluations with filters for specific spans in your traces. All evaluated requests appear in one dashboard. You can filter traces by evaluation scores to understand issues. You can also add problematic cases to test sets for continuous improvement.

Setting up online evaluation takes just a couple of minutes. It provides immediate visibility into production quality.

Customize LLM-as-a-Judge Output Schemas​

10 November 2025

v0.62.0

The LLM-as-a-Judge evaluator now supports custom output schemas. Create multiple feedback outputs per evaluator with any structure you need.

You can configure output types (binary, multiclass), include reasoning to improve prediction quality, or provide a raw JSON schema with any structure you define. Use these custom schemas in your evaluations to capture exactly the feedback you need.

Learn more in the LLM-as-a-Judge documentation.

Documentation Overhaul​

3 November 2025

v0.59.10

We've completely rewritten and restructured our documentation with a new architecture. This is one of the largest updates we've made, involving a near-complete rewrite of existing content.

Key improvements include:

• Diataxis Framework: Organized content into Tutorials, How-to Guides, Reference, and Explanation sections for better discoverability
• Expanded Observability Docs: Added missing documentation for tracing, annotations, and observability features
• JavaScript/TypeScript Support: Added code examples and documentation for JavaScript developers alongside Python
• Ask AI Feature: Ask questions directly to the documentation for instant answers

Vertex AI Provider Support​

24 October 2025

v0.59.6

We've added support for Google Cloud's Vertex AI platform. You can now use Gemini models and other Vertex AI partner models in the playground, configure them in the Model Hub, and access them through the Gateway using InVoke endpoints.

Check out the documentation for configuring Vertex AI models.

Filtering Traces by Annotation​

14 October 2025

v0.58.0

You can now filter and search traces based on their annotations. This helps you find traces with low scores or bad feedback quickly.

We rebuilt the filtering system in observability with a simpler dropdown and more options. You can now filter by span status, input keys, app or environment references, and any key within your span.

The new annotation filtering lets you find:

• Spans evaluated by a specific evaluator
• Spans with user feedback like success=True

This enables powerful workflows: capture user feedback from your app, filter to find traces with bad feedback, add them to test sets, and improve your prompts based on real user data.

New Evaluation Results Dashboard​

26 September 2025

v0.54.0

We've completely redesigned the evaluation results dashboard. You can analyse your evaluation results more easily and understand performance across different metrics.

Here's what's new:

• Metrics plots: We've added plots for all the evaluator metrics. You can not see the distribution of the results and easily spot outliers.
• Side-by-side comparison: You can now compare multiple evaluations simultaneously. You can compare the plots but also the single outputs.
• Improved test cases view: The results are now displayed in a tabular format works both for small and large datasets.
• Focused detail view: A new focused drawer lets you examine individual data points in more details. It's very helpful if your data is large.
• Configuration view: See exactly which configurations were used in each evaluation
• Evaluation Run naming and descriptions: Add names and descriptions to your evaluation runs to organize things better.

Deep URL Support for Sharable Links​

24 September 2025

v0.53.0

URLs across Agenta now include workspace context, making them fully shareable between team members. Previously, URLs would always point to the default workspace, causing issues when refreshing pages or sharing links.

Now you can deep link to almost anything in the platform - prompts, evaluations, and more - in any workspace. Share links directly with team members and they'll see exactly what you intended, regardless of their default workspace settings.

Major Speed Improvements and Bug Fixes​

19 September 2025

v0.52.5

We rewrote most of Agenta's frontend. You'll see much faster speeds when you create prompts or use the playground.

We also made many improvements and fixed bugs:

Improvements:

• LLM-as-a-judge now uses double curly braces {{}} instead of single curly braces { and }. This matches how normal prompts work. Old LLM-as-a-judge prompts with single curly braces still work. We updated the LLM-as-a-judge playground to make editing prompts easier.

Self-hosting:

• You can now use an external Redis instance for caching by setting it as an environment variable

Bug fixes:

• Fixed the custom workflow quick start tutorial and examples
• Fixed SDK compatibility issues with Python 3.9
• Fixed default filtering in observability dashboard
• Fixed error handling in the evaluator playground
• Fixed the Tracing SDK to allow instrumenting streaming responses and overriding OTEL environment variables

Multiple Metrics in Human Evaluation​

9 September 2025

v0.51.0

We rebuilt the human evaluation workflow from scratch. Now you can set multiple evaluators and metrics and use them to score the outputs.

This lets you evaluate the same output on different metrics like relevance or completeness. You can also create binary, numerical scores, or even use strings for comments or expected answer.

Watch the video below and read the post for more details. Or check out the docs to learn how to use the new human evaluation workflow.

DSPy Integration​

29 August 2025

We've added DSPy integration to Agenta. You can now trace and debug your DSPy applications with Agenta.

View the full DSPy integration →

Open-sourcing our Product Roadmap​

12 August 2025

We've made our product roadmap completely transparent and community-driven.

You can now see exactly what we're building, what's shipped, and what's coming next. Plus vote on features that matter most to you.

Why we're doing this: We believe open-source startups succeed when they create the most value possible, and the best way to do that is by building with our community, not in isolation. Up until now, we've been secretive with our roadmap, but we're losing something important: your feedback and the ability to let you shape our direction. Today we're open-sourcing our roadmap because we want to build a community of owners, not just passive users.

View the full roadmap →

Major Playground Improvements and Enhancements​

7 August 2025 v0.50.5

We've made significant improvements to the playground. Key features include:

• Improving the error handling in JSON editor for structured output
• Preventing the JSON field order from being changed
• Visual diff when committing changes
• Markdown and text view toggle
• Collapsible interface elements
• Collapsible test cases for large sets

Support for Images in the Playground​

29 July 2025 v0.50.0

Agenta now supports images in the playground, test sets, and evaluations. Click above for more details.

LlamaIndex Integration​

17 June 2025 v0.48.4

We're excited to announce observability support for LlamaIndex applications.

If you're using LlamaIndex, you can now see detailed traces in Agenta to debug your application.

The integration is auto-instrumentation - just add one line of code and you'll start seeing all your LlamaIndex operations traced.

This helps when you need to understand what's happening inside your RAG pipeline, track performance bottlenecks, or debug issues in production.

We've put together a Jupyter notebook and tutorial to get you started. Links are in the comments.

Annotate Your LLM Response (preview)​

15 May 2025 v0.45.0

One of the major feature requests we had was the ability to capture user feedback and annotations (e.g. scores) to LLM responses traced in Agenta.

Today we're previewing one of a family of features around this topic.

As of today you can use the annotation API to add annotations to LLM responses traced in Agenta.

This is useful to:

• Collect user feedback on LLM responses
• Run custom evaluation workflows
• Measure application performance in real-time

Check out the how to annotate traces from API for more details. Or try our new tutorial (available as jupyter notebook) here.

Other stuff:

• We have cut our migration process to take a couple of minutes instead of an hour.

Tool Support in the Playground​

10 May 2025 v0.43.1

We released tool usage in the Agenta playground - a key feature for anyone building agents with LLMs.

Agents need tools to access external data, perform calculations, or call APIs.

Now you can:

• Define tools directly in the playground using JSON schema
• Test how your prompt generates tool calls in real-time
• Preview how your agent handles tool responses
• Verify tool call correctness with custom evaluators

The tool schema is saved with your prompt configuration, making integration easy when you fetch configs through the API.

Documentation Overhaul, New Models, and Platform Improvements​

2 May 2025

v0.43.0

We've made significant improvements across Agenta with a major documentation overhaul, new model support, self-hosting enhancements, and UI improvements.

Revamped Prompt Engineering Documentation:

We've completely rewritten our prompt management and prompt engineering documentation.

Start exploring the new documentation in our updated Quick Start Guide.

New Model Support:

Our platform now supports several new LLM models:

• Google's Gemini 2.5 Pro and Flash
• Alibaba Cloud's Qwen 3
• OpenAI's GPT-4.1

These models are available in both the playground and through the API.

Playground Enhancements:

We've added a draft state to the playground, providing a better editing experience. Changes are now clearly marked as drafts until committed.

Self-Hosting Improvements:

We've significantly simplified the self-hosting experience by changing how environment variables are handled in the frontend:

• No more rebuilding images to change ports or domains
• Dynamic configuration through environment variables at runtime

Check out our updated self-hosting documentation for details.

Bug Fixes and Optimizations:

• Fixed OpenTelemetry integration edge cases
• Resolved edge cases in the API that affected certain workflow configurations
• Improved UI responsiveness and fixed minor visual inconsistencies
• Added chat support in cloud

We are SOC 2 Type 2 Certified​

18 April 2025 v0.42.1

We are SOC 2 Type 2 Certified. This means that our platform is audited and certified by an independent third party to meet the highest standards of security and compliance.

Structured Output Support in the Playground​

15 April 2025

v0.42.0

We now support structured output support in the playground. You can define the expected output format and validate the output against it.

With Agenta's playground, implementing structured outputs is straightforward:

• Open any prompt

• Switch the Response format dropdown from text to JSON mode or JSON Schema

• Paste or write your schema (Agenta supports the full JSON Schema specification)

• Run the prompt - the response panel will show the response beautified

• Commit the changes - the schema will be saved with your prompt, so when your SDK fetches the prompt, it will include the schema information

Check out the blog post for more detail https://agenta.ai/blog/structured-outputs-playground

New Feature: Prompt and Deployment Registry​

7 April 2025

v0.38.0

We've introduced the Prompt and Deployment Registry, giving you a centralized place to manage all variants and versions of your prompts and deployments.

Key capabilities:

• View all variants and revisions in a single table
• Access all commits made to a variant
• Use older versions of variants directly in the playground

Learn more in our blog post.

Bug Fixes

• Fixed minor UI issues with dots in sidebar menu
• Fixed minor playground UI issues
• Fixed playground reset default model name
• Fixed project_id issue on testset detail page
• Fixed breaking issues with old variants encountered during QA
• Fixed variant naming logic

Improvements to the Playground and Custom Workflows​

19 March 2025 v0.36.0

We've made several improvements to the playground, including:

• Improved scrolling behavior
• Increased discoverability of variants creation and comparison
• Implemented stop functionality in the playground

As for custom workflows, now they work with sub-routes. This means you can have multiple routes in one file and create multiple custom workflows from the same file.

OpenTelemetry Compliance and Custom workflows from API​

11 March 2025

v0.35.0

We've introduced major improvements to Agenta, focusing on OpenTelemetry compliance and simplified custom workflow debugging.

OpenTelemetry (OTel) Support:

Agenta is now fully OpenTelemetry-compliant. This means you can seamlessly integrate Agenta with thousands of OTel-compatible services using existing SDKs. To integrate your application with Agenta, simply configure an OTel exporter pointing to your Agenta endpoint—no additional setup required.

We've enhanced distributed tracing capabilities to better debug complex distributed agent systems. All HTTP interactions between agents—whether running within Agenta's SDK or externally—are automatically traced, making troubleshooting and monitoring easier.

Detailed instructions and examples are available in our distributed tracing documentation.

Improved Custom Workflows:

Based on your feedback, we've streamlined debugging and running custom workflows:

• Run workflows from your environments: You no longer need the Agenta CLI to manage custom workflows. Setting up custom workflows now involves simply adding the Agenta SDK to your code, creating an endpoint, and connecting it to Agenta via the web UI. You can check how it's done in the quick start guide.

• Custom Workflows in the new playground: Custom workflows are now fully compatible with the new playground. You can now nest configurations, run side-by-side comparisons, and debug your agents and complex workflows very easily.

New Playground​

4 February 2025

v0.33.0

We've rebuilt our playground from scratch to make prompt engineering faster and more intuitive. The old playground took 20 seconds to create a prompt - now it's instant.

Key improvements:

• Create prompts with multiple messages using our new template system
• Format variables easily with curly bracket syntax and a built-in validator
• Switch between chat and completion prompts in one interface
• Load test sets directly in the playground to iterate faster
• Save successful outputs as test cases with one click
• Compare different prompts side-by-side
• Deploy changes straight to production

For developers, now you create prompts programmatically through our API.

You can explore these features in our updated playground documentation.

Quality of life improvements​

27 January 2025

v0.32.0

Small release today with quality of life improvements, while we're preparing the huge release coming up in the next days:

• Added a collapsible side menu for better space management
• Enhanced frontend performance and responsiveness
• Implemented a confirmation modal when deleting test sets
• Improved permission handling across the platform
• Improved frontend test coverage

Agenta is SOC 2 Type 1 Certified​

15 January 2025

v0.31.0

We've achieved SOC 2 Type 1 certification, validating our security controls for protecting sensitive LLM development data. This certification covers our entire platform, including prompt management, evaluation frameworks, and observability tools.

Key security features and improvements:

• Data encryption in transit and at rest
• Enhanced access control and authentication
• Comprehensive security monitoring
• Regular third-party security assessments
• Backup and disaster recovery protocols

This certification represents a significant milestone for teams using Agenta in production environments. Whether you're using our open-source platform or cloud offering, you can now build LLM applications with enterprise-grade security confidence.

We've also updated our trust center with detailed information about our security practices and compliance standards. For teams interested in learning more about our security controls or requesting our SOC 2 report, please contact team@agenta.ai.

New Onboarding Flow​

4 January 2025

v0.30.0

We've redesigned our platform's onboarding to make getting started simpler and more intuitive. Key improvements include:

• Streamlined tracing setup process
• Added a demo RAG playground project showcasing custom workflows
• Enhanced frontend performance
• Fixed scroll behavior in trace view

Add Spans to Test Sets​

11 December 2024

v0.29.0

This release introduces the ability to add spans to test sets, making it easier to bootstrap your evaluation data from production. The new feature lets you:

• Add individual or batch spans to test sets
• Create custom mappings between spans and test sets
• Preview test set changes before committing them

Additional improvements:

• Fixed CSV test set upload issues
• Prevented viewing of incomplete evaluations
• Added mobile compatibility warning
• Added support for custom ports in self-hosted installations

Viewing Traces in the Playground and Authentication for Deployed Applications​

29 November 2024

v0.28.0

Viewing traces in the playground:​

You can now see traces directly in the playground. For simple applications, this means you can view the prompts sent to LLMs. For custom workflows, you get an overview of intermediate steps and outputs. This makes it easier to understand what’s happening under the hood and debug your applications.

Authentication improvements:​

We’ve strengthened authentication for deployed applications. As you know, Agenta lets you either fetch the app’s config or call it with Agenta acting as a proxy. Now, we’ve added authentication to the second method. The APIs we create are now protected and can be called using an API key. You can find code snippets for calling the application in the overview page.

Documentation improvements:​

We’ve added new cookbooks and updated existing documentation:

• New cookbook for observability with LangChain
• Updated the custom workflows documentation and added reference
• Updated the reference for the observability SDK and for the prompt management SDK

Bug fixes:​

• Fixed an issue with the observability SDK not being compatible with LiteLLM.
• Fixed an issue where cost and token usage were not correctly computed for all calls.

Observability and Prompt Management​

6 November 2024

v0.27.0

This release is one of our biggest yet—one changelog hardly does it justice.

First up: Observability

We’ve had observability in beta for a while, but now it’s been completely rewritten, with a brand-new UI and fully open-source code.

The new Observability SDK is compatible with OpenTelemetry (Otel) and gen-ai semantic conventions. This means you get a lot of integrations right out of the box, like LangChain, OpenAI, and more.

We’ll publish a full blog post soon, but here’s a quick look at what the new observability offers:

• A redesigned UI that lets you visualize nested traces, making it easier to understand what’s happening behind the scenes.

• The web UI lets you filter traces by name, cost, and other attributes—you can even search through them easily.

• The SDK is Otel-compatible, and we’ve already tested integrations for OpenAI, LangChain, LiteLLM, and Instructor, with guides available for each. In most cases, adding a few lines of code will have you seeing traces directly in Agenta.

Next: Prompt Management

We’ve completely rewritten the prompt management SDK, giving you full CRUD capabilities for prompts and configurations. This includes creating, updating, reading history, deploying new versions, and deleting old ones. You can find a first tutorial for this here.

And finally: LLM-as-a-Judge Overhaul

We've made significant upgrades to the LLM-as-a-Judge evaluator. It now supports prompts with multiple messages and has access to all variables in a test case. You can also switch models (currently supporting OpenAI and Anthropic). These changes make the evaluator much more flexible, and we're seeing better results with it.

New Application Management View and Various Improvements​

22 October 2024

v0.26.0

We updated the Application Management View to improve the UI. Many users struggled to find their applications when they had a large number, so we've improved the view and added a search bar for quick filtering. Additionally, we are moving towards a new project structure for the application. We moved test sets and evaluators outside of the application scope. So now, you can use the same test set and evaluators in multiple applications.

Bug Fixes

• Added an export button in the evaluation view to export results from the main view.
• Eliminated Pydantic warnings in the CLI.
• Improved error messages when fetch_config is called with wrong arguments.
• Enhanced the custom code evaluation sandbox and removed the limitation that results need to be between 0 and 1

Evaluator Testing Playground and a New Evaluation View​

22 September 2024

v0.25.0

Many users faced challenges configuring evaluators in the web UI. Some evaluators, such as LLM as a judge, custom code, or RAG evaluators can be tricky to set up correctly on the first try. Until now, users needed to setup, run an evaluation, check the errors, then do it again.

To address this, we've introduced a new evaluator test/debug playground. This feature allows you to test the evaluator live on real data, helping you test the configuration before committing to it and using it for evaluations.

Additionally, we have improved and redesigned the evaluation view. Both automatic and human evaluations are now within the same view but in different tabs. We're moving towards unifying all evaluator results and consolidating them in one view, allowing you to quickly get an overview of what's working.

UI Redesign and Configuration Management and Overview View​

22 August 2024

v0.24.0

We've completely redesigned the platform's UI. Additionally we have introduced a new overview view for your applications. This is part of a series of upcoming improvements slated for the next few weeks.

The new overview view offers:

• A dashboard displaying key metrics of your application
• A table with all the variants of your applications
• A summary of your application's most recent evaluations

We've also added a new JSON Diff evaluator. This evaluator compares two JSON objects and provides a similarity score.

Lastly, we've updated the UI of our documentation.

New Alpha Version of the SDK for Creating Custom Applications​

20 August 2024

v0.23.0

We've released a new version of the SDK for creating custom applications. This Pydantic-based SDK significantly simplifies the process of building custom applications. It's fully backward compatible, so your existing code will continue to work seamlessly. We'll soon be rolling out comprehensive documentation and examples for the new SDK.

In the meantime, here's a quick example of how to use it:

import agenta as ag
from agenta import Agenta
from pydantic import BaseModel, Field

ag.init()

# Define the configuration of the application (that will be shown in the playground )
class MyConfig(BaseModel):
    temperature: float = Field(default=0.2)
    prompt_template: str = Field(default="What is the capital of {country}?")

# Creates an endpoint for the entrypoint of the application
@ag.route("/", config_schema=MyConfig)
def generate(country: str) -> str:
    # Fetch the config from the request
    config: MyConfig = ag.ConfigManager.get_from_route(schema=MyConfig)
    prompt = config.prompt_template.format(country=country)
    chat_completion = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": prompt}],
        temperature=config.temperature,
    )
    return chat_completion.choices[0].message.content

RAGAS Evaluators and Traces in the Playground​

12 August 2024

v0.22.0

We're excited to announce two major features this week:

1. We've integrated RAGAS evaluators into agenta. Two new evaluators have been added: RAG Faithfulness (measuring how consistent the LLM output is with the context) and Context Relevancy (assessing how relevant the retrieved context is to the question). Both evaluators use intermediate outputs within the trace to calculate the final score.

   Check out the tutorial to learn how to use RAG evaluators.

2. You can now view traces directly in the playground. This feature enables you to debug your application while configuring it—for example, by examining the prompts sent to the LLM or reviewing intermediate outputs.

Migration from MongoDB to Postgres​

9 July 2024

v0.19.0

We have migrated the Agenta database from MongoDB to Postgres. As a result, the platform is much more faster (up to 10x in some use cases).

However, if you are self-hosting agenta, note that this is a breaking change that requires you to manually migrate your data from MongoDB to Postgres.

If you are using the cloud version of Agenta, there is nothing you need to do (other than enjoying the new performance improvements).

More Reliable Evaluations​

5 July 2024

v0.18.0

We have worked extensively on improving the reliability of evaluations. Specifically:

• We improved the status for evaluations and added a new Queued status
• We improved the error handling in evaluations. Now we show the exact error message that caused the evaluation to fail.
• We fixed issues that caused evaluations to run infinitely
• We fixed issues in the calculation of scores in human evaluations.
• We fixed small UI issues with large output in human evaluations.
• We have added a new export button in the evaluation view to export the results as a CSV file.

In observability:

• We have added a new integration with Litellm to automatically trace all LLM calls done through it.
• Now we automatically propagate cost and token usage from spans to traces.

Evaluators can access all columns​

4 June 2024

v0.17.0

Evaluators now can access all columns in the test set. Previously, you were limited to using only the correct_answer column for the ground truth / reference answer in evaluation. Now you can configure your evaluator to use any column in the test set as the ground truth. To do that, open the collapsable Advanced Settings when configuring the evaluator, and define the Expected Answer Column to the name of the columns containing the reference answer you want to use.

In addition to this:

• We've upgraded the SDK to pydantic v2.
• We have improved by 10x the speed for the get config endpoint
• We have add documentation for observability

New LLM Provider: Welcome Gemini!​

25 May 2024

v0.14.14

We are excited to announce the addition of Google's Gemini to our list of supported LLM providers, bringing the total number to 12.

Playground Improvements​

24 May 2024

v0.14.1-13

• We've improved the workflow for adding outputs to a dataset in the playground. In the past, you had to select the name of the test set each time. Now, the last used test set is selected by default..
• We have significantly improved the debugging experience when creating applications from code. Now, if an application fails, you can view the logs to understand the reason behind the failure.
• We moved the copy message button in the playground to the output text area.
• We now hide the cost and usage in the playground when they aren't specified
• We've made improvements to error messages in the playground

Bug Fixes

• Fixed the order of the arguments when running a custom code evaluator
• Fixed the timestamp in the Testset view (previous stamps was droppping the trailing 0)
• Fixed the creation of application from code in the self-hosted version when using Windows

Prompt and Configuration Registry​

1 May 2024

v0.14.0

We've introduced a feature that allows you to use Agenta as a prompt registry or management system. In the deployment view, we now provide an endpoint to directly fetch the latest version of your prompt. Here is how it looks like:

from agenta import Agenta
agenta = Agenta()
config = agenta.get_config(base_id="xxxxx", environment="production", cache_timeout=200) # Fetches the configuration with caching

You can find additional documentation here.

Improvements

• Previously, publishing a variant from the playground to an environment was a manual process., from now on we are publishing by default to the production environment.

Miscellaneous Improvements​

28 April 2024

v0.13.8

• The total cost of an evaluation is now displayed in the evaluation table. This allows you to understand how much evaluations are costing you and track your expenses.

Bug Fixes

• Fixed sidebar focus in automatic evaluation results view
• Fix the incorrect URLs shown when running agenta variant serve

Evaluation Speed Increase and Numerous Quality of Life Improvements​

23rd April 2024

v0.13.1-5

• We've improved the speed of evaluations by 3x through the use of asynchronous batching of calls.
• We've added Groq as a new provider along with Llama3 to our playground.

Bug Fixes

• Resolved a rendering UI bug in Testset view.
• Fixed incorrect URLs displayed when running the 'agenta variant serve' command.
• Corrected timestamps in the configuration.
• Resolved errors when using the chat template with empty input.
• Fixed latency format in evaluation view.
• Added a spinner to the Human Evaluation results table.
• Resolved an issue where the gitignore was being overwritten when running 'agenta init'.

Observability (beta)​

14th April 2024

v0.13.0

You can now monitor your application usage in production. We've added a new observability feature (currently in beta), which allows you to:

• Monitor cost, latency, and the number of calls to your applications in real-time.
• View the logs of your LLM calls, including inputs, outputs, and used configurations. You can also add any interesting logs to your test set.
• Trace your more complex LLM applications to understand the logic within and debug it.

As of now, all new applications created will include observability by default. We are working towards a GA version in the next weeks, which will be scalable and better integrated with your applications. We will also be adding tutorials and documentation about it.

Find examples of LLM apps created from code with observability here.

Compare latency and costs​

1st April 2024

v0.12.6

You can now compare the latency and cost of different variants in the evaluation view.

Minor improvements​

31st March 2024

v0.12.5

Toggle variants in comparison view

You can now toggle the visibility of variants in the comparison view, allowing you to compare a multitude of variants side-by-side at the same time.

Improvements

• You can now add a datapoint from the playground to the test set even if there is a column mismatch

Bug fixes

• Resolved issue with "Start Evaluation" button in Testset view
• Fixed bug in CLI causing variant not to serve

New evaluators​

25th March 2024

v0.12.4

We have added some more evaluators, a new string matching and a Levenshtein distance evaluation.

Improvements

• Updated documentation for human evaluation
• Made improvements to Human evaluation card view
• Added dialog to indicate testset being saved in UI

Bug fixes

• Fixed issue with viewing the full output value during evaluation
• Enhanced error boundary logic to unblock user interface
• Improved logic to save and retrieve multiple LLM provider keys
• Fixed Modal instances to support dark mode

Minor improvements​

11th March 2024

v0.12.3

• Improved the logic of the Webhook evaluator
• Made the inputs in the Human evaluation view non-editable
• Added an option to save a test set in the Single model evaluation view
• Included the evaluator name in the "Configure your evaluator" modal

Bug fixes

• Fixed column resize in comparison view
• Resolved a bug affecting the evaluation output in the CSV file
• Corrected the path to the Evaluators view when navigating from Evaluations

Highlight ouput difference when comparing evaluations​

4th March 2024

v0.12.2

We have improved the evaluation comparison view to show the difference to the expected output.

Improvements

• Improved the error messages when invoking LLM applications
• Improved "Add new evaluation" modal
• Upgraded Sidemenu to display Configure evaluator and run evaluator under Evaluations section
• Changed cursor to pointer when hovering over evaluation results

Deployment Versioning and RBAC​

14th February 2024

v0.12.0

Deployment versioning

You now have access to a history of prompts deployed to our three environments. This feature allows you to roll back to previous versions if needed.

Role-Based Access Control

You can now invite team members and assign them fine-grained roles in agenta.

Improvements

• We now prevent the deletion of test sets that are used in evaluations

Bug fixes

• Fixed bug in custom code evaluation aggregation. Up until know the aggregated result for custom code evalution where not computed correctly.

• Fixed bug with Evaluation results not being exported correctly

• Updated documentation for vision gpt explain images

• Improved Frontend test for Evaluations

Minor fixes​

4th February 2024

v0.10.2

• Addressed issue when invoking LLM app with missing LLM provider key
• Updated LLM providers in Backend enum
• Fixed bug in variant environment deployment
• Fixed the sorting in evaluation tables
• Made use of server timezone instead of UTC

Prompt Versioning​

31st January 2024

v0.10.0

We've introduced the feature to version prompts, allowing you to track changes made by the team and revert to previous versions. To view the change history of the configuration, click on the sign in the playground to access all previous versions.

New JSON Evaluator​

30th January 2024

v0.9.1 We have added a new evaluator to match JSON fields and added the possiblity to use other columns in the test set other than the correct_answer column as the ground truth.

Improved error handling in evaluation​

29th January 2024

v0.9.0

We have improved error handling in evaluation to return more information about the exact source of the error in the evaluation view.

Improvements:

• Added the option in A/B testing human evaluation to mark both variants as correct
• Improved loading state in Human Evaluation

Bring your own API key​

25th January 2024

v0.8.3

Up until know, we required users to use our OpenAI API key when using cloud. Starting now, you can use your own API key for any new application you create.

Improved human evaluation workflow​

24th January 2024

v0.8.2

Faster human evaluation workflow

We have updated the human evaluation table view to add annotation and correct answer columns.

Improvements:

• Simplified the database migration process
• Fixed environment variable injection to enable cloud users to use their own keys
• Disabled import from endpoint in cloud due to security reasons
• Improved query lookup speed for evaluation scenarios
• Improved error handling in playground

Bug fixes:

• Resolved failing Backend tests
• Fixed a bug in rate limit configuration validation
• Fixed issue with all aggregated results
• Resolved issue with live results in A/B testing evaluation not updating

Revamping evaluation​

22nd January 2024

v0.8.0

We've spent the past month re-engineering our evaluation workflow. Here's what's new:

Running Evaluations

1. Simultaneous Evaluations: You can now run multiple evaluations for different app variants and evaluators concurrently.

2. Rate Limit Parameters: Specify these during evaluations and reattempts to ensure reliable results without exceeding open AI rate limits.

3. Reusable Evaluators: Configure evaluators such as similarity match, regex match, or AI critique and use them across multiple evaluations.

Evaluation Reports

1. Dashboard Improvements: We've upgraded our dashboard interface to better display evaluation results. You can now filter and sort results by evaluator, test set, and outcomes.

2. Comparative Analysis: Select multiple evaluation runs and view the results of various LLM applications side-by-side.

Adding Cost and Token Usage to the Playground​

12th January 2024

v0.7.1

We've added a feature that allows you to compare the time taken by an LLM app, its cost, and track token usage, all in one place.

----#

Changes to the SDK​

This necessitated modifications to the SDK. Now, the LLM application API returns a JSON instead of a string. The JSON includes the output message, usage details, and cost:

{
"message": string,
"usage": {
"prompt_tokens": int,
"completion_tokens": int,
"total_tokens": int
},
"cost": float
}

Improving Side-by-side Comparison in the Playground​

19th December 2023

v0.6.6

• Enhanced the side-by-side comparison in the playground for better user experience

Resolved Batch Logic Issue in Evaluation​

18th December 2023

v0.6.5

• Resolved an issue with batch logic in evaluation (users can now run extensive evaluations)

Comprehensive Updates and Bug Fixes​

12th December 2023

v0.6.4

• Incorporated all chat turns to the chat set
• Rectified self-hosting documentation
• Introduced asynchronous support for applications
• Added 'register_default' alias
• Fixed a bug in the side-by-side feature

Integrated File Input and UI Enhancements​

12th December 2023

v0.6.3

• Integrated file input feature in the SDK
• Provided an example that includes images
• Upgraded the human evaluation view to present larger inputs
• Fixed issues related to data overwriting in the cloud
• Implemented UI enhancements to the side bar

Minor Adjustments for Better Performance​

7th December 2023

v0.6.2

• Made minor adjustments

Bug Fix for Application Saving​

7th December 2023

v0.6.1

• Resolved a bug related to saving the application

Introduction of Chat-based Applications​

1st December 2023

v0.6.0

• Introduced chat-based applications
• Fixed a bug in 'export csv' feature in auto evaluation

Multiple UI and CSV Reader Fixes​

1st December 2023

v0.5.8

• Fixed a bug impacting the csv reader
• Addressed an issue of variant overwriting
• Made tabs draggable for better UI navigation
• Implemented support for multiple LLM keys in the UI

Enhanced Self-hosting and Mistral Model Tutorial​

17th November 2023

v0.5.7

• Enhanced and simplified self-hosting feature
• Added a tutorial for the Mistral model
• Resolved a race condition issue in deployment
• Fixed an issue with saving in the playground

Sentry Integration and User Communication Improvements​

12th November 2023

v0.5.6

• Enhanced bug tracking with Sentry integration in the cloud
• Integrated Intercom for better user communication in the cloud
• Upgraded to the latest version of OpenAI
• Cleaned up files post serving in CLI

Cypress Tests and UI Improvements​

2nd November 2023

v0.5.5

• Conducted extensive Cypress tests for improved application stability
• Added a collapsible sidebar for better navigation
• Improved error handling mechanisms
• Added documentation for the evaluation feature

Launch of SDK Version 2 and Cloud-hosted Version​

23rd October 2023

v0.5.0

• Launched SDK version 2
• Launched the cloud-hosted version
• Completed a comprehensive refactoring of the application

Deploying a prompt often needs follow-up work outside Agenta. You may want to sync prompt files into a repository, trigger CI, open a pull request, or notify an internal platform. Before this change, that required custom glue code.

The Solution​

You can now trigger webhooks and GitHub automations directly from Agenta when a deployment event happens. Point Agenta at your own HTTPS endpoint, or send the event straight to GitHub with repository_dispatch or workflow_dispatch.

This gives you a simple way to connect prompt deployments to the rest of your delivery flow. You can keep prompt changes, infrastructure checks, and repository updates in sync without building a separate integration service.

What You Can Do​

• Send deployment events to any HTTPS endpoint
• Verify deliveries with HMAC signatures or use a bearer token
• Trigger GitHub repository_dispatch with a structured event payload
• Trigger GitHub workflow_dispatch for one known workflow on a branch
• Fetch the latest prompt in GitHub Actions and open a pull request automatically

How It Works​

Create an automation in your project settings and subscribe it to deployment events. When a prompt revision is committed to an environment, Agenta sends an HTTP POST to your target.

For generic webhooks, Agenta sends the event payload plus delivery headers such as X-Agenta-Event-Type, X-Agenta-Delivery-Id, and Idempotency-Key. In signature mode, Agenta also signs the raw request body with HMAC-SHA256.

For GitHub automations, Agenta calls the GitHub API directly. repository_dispatch sends a richer JSON payload that includes event metadata and references. workflow_dispatch sends a smaller set of string inputs for a specific workflow file and branch.

Typical Use Cases​

• Sync the latest deployed prompt into a repository
• Open a pull request for prompt changes after each deployment
• Trigger validation or approval workflows in GitHub Actions
• Notify internal tools when a production prompt changes
• Mirror deployment metadata into another system for auditing

Getting Started​

Start with these guides:

• Webhooks
• GitHub
• Fetch Prompts via SDK/API

Building agentic applications that interact with external services means writing integration code: OAuth flows, API wrappers, error handling. You do all of that before you can even test whether your prompt works with the tool.

The Solution​

You can now connect external tools to your prompts and call them from the playground. Browse a catalog of 150+ integrations, authenticate with OAuth or an API key, and attach tool actions to your prompt config. When the LLM generates a tool call, you execute it with one click and send the result back to the chat.

What You Can Do​

• Use Google Sheets or Notion as data sources. Connect a spreadsheet or Notion database and let the LLM query it. Build RAG applications directly from the playground without writing any code.
• Send emails and messages. Attach Gmail or Slack and your prompt can draft and send emails, post to channels, or create threads.
• Automate developer workflows. Connect GitHub to create issues, Jira to update tickets, or any of the 150+ available integrations.
• Manage connections in Settings. A dedicated Tools page shows all your connected integrations, their status, and lets you add or revoke connections.

How It Works​

1. Go to Settings > Tools and connect an integration (e.g., Gmail). You authenticate via OAuth or paste an API key.
2. In the Playground, open the tools panel in your prompt config. Your connected integrations appear there.
3. Select the actions you want available to the LLM (e.g., "Send Email", "List Emails").
4. Run your prompt. When the LLM generates a tool call, click "Execute" to run it and return the result to the conversation.

Getting Started​

Tool integrations are available on Agenta Cloud. Connect your first integration in Settings > Tools.

Prompt engineering is iterative. You write a prompt, test it, notice it's too vague or missing constraints, and rewrite it. Each cycle requires you to figure out how to improve it, not just what to improve. That's the hard part.

The Solution​

You can now refine prompts with AI directly in the playground. Click the wand icon on any prompt section, describe what you want to change in plain English, and get back a refined version with an explanation of what changed and why.

How It Works​

1. Open a prompt in the playground and click the wand icon in the prompt section header.
2. A two-panel modal opens. On the left, type what you want to improve (e.g., "add output format instructions" or "make it more concise"). On the right, see the refined prompt.
3. Each refinement builds on the previous result, so you can iterate. Ask for one change, review it, then ask for another.
4. Toggle the Diff view to see exactly what changed compared to the original.
5. Edit the refined prompt directly if you want to adjust anything before applying.
6. Click Use refined prompt to apply the changes to your playground session.

You can also use the built-in quick action "Optimize the prompt using best practices" for a one-click improvement.

What You Can Do​

• Describe improvements in plain English: "Add a constraint that the output must be valid JSON" or "Make the tone more professional."
• Iterate: Each refinement uses the latest version, so you can make incremental changes across multiple rounds.
• Review diffs: Toggle the diff view to see a side-by-side comparison of the original and refined prompts.
• Edit before applying: The refined prompt is fully editable. Adjust anything before committing.
• Quick optimize: Use the built-in "Optimize the prompt using best practices" shortcut for an instant improvement.

Getting Started​

Open any prompt in the playground. Look for the wand icon in the prompt section header. If you don't see it, the feature may not be enabled for your organization yet.

Agenta has new enterprise features. Multi-organization support, SSO, domain verification, and a US region.

Multi-Organization Support​

You can now create separate organizations for different teams or clients. Each organization has its own billing, projects, and roles. One account can belong to multiple organizations, and you switch between them without signing out.

Inside each organization, you create workspaces and projects. Workspaces group related projects together. Projects scope your prompts, evaluations, and traces.

Roles are per-organization. You can be an owner in one org and a viewer in another. Available roles include owner, workspace admin, editor, viewer, evaluator, and deployment manager.

SSO​

Connect your identity provider to Agenta. We support any OIDC-compliant provider: Okta, Azure AD, Auth0, OneLogin, Google Workspace, and others.

You configure SSO per-organization through the API. Add your provider's issuer URL, client ID, and client secret. Agenta discovers the rest from the OIDC discovery endpoint. You can test the connection before enabling it.

If your security policy requires it, you can enforce SSO-only for an organization and disable password login entirely.

Domain Verification​

Verify your company domain via a DNS TXT record. Once verified, anyone who signs up with a matching email address joins your organization automatically. No need to send invitations one by one.

Domain verification also works with SSO enforcement. Users with a verified domain email are automatically routed to your SSO provider during login.

US Region​

Agenta Cloud now has a US-based region. If your data needs to stay in the United States, you can run your projects there. The US region has the same features as the EU region.

Availability​

SSO, and domain verification are available on Business and Enterprise plans. The US region is available on all plans.

Getting Started​

• Organizations documentation
• SSO documentation

When you're building agents or managing multiple use cases, prompts multiply fast. Finding the right one turns into scrolling through a flat list.

The Solution​

You can now create folders and subfolders to organize your prompts. It works like any file system you've used before.

Create a folder. Drag prompts into it. Create subfolders. Move things around as your structure evolves.

What You Can Do​

• Create folders and nest them as deep as you need
• Move prompts and folders via drag-and-drop or the actions menu
• Search across all folders from the root level
• Share folder URLs with teammates (folder location persists in the URL)

Getting Started​

Go to the Prompts page. Click "Create new" and select "New folder". Name it, and start organizing.

Getting started with a new platform can be overwhelming. You now have an onboarding widget that guides you through Agenta's key features step by step. The widget appears in the sidebar and tracks your progress as you explore the platform.

Each walkthrough highlights the relevant UI elements and explains what they do. You learn by doing, not by reading documentation.

Key Capabilities​

• Progress Tracking: The widget shows which features you've explored and what's left to discover
• Interactive Walkthroughs: Step-by-step tours that highlight UI elements as you go
• Contextual Guidance: Each walkthrough focuses on a specific workflow, from creating prompts to running evaluations
• Dismissible: Once you're comfortable, dismiss the widget and it stays out of your way

Available Walkthroughs​

The onboarding widget includes guided tours for:

• Playground Basics: Create and test prompts, compare variants, and iterate on your configurations
• Running Evaluations: Set up test sets, configure evaluators, and analyze results
• Observability: Trace your LLM calls, add annotations, and debug production issues
• Deployment: Deploy prompts to environments and fetch configurations in your code

How It Works​

The onboarding widget appears automatically for new users. Each walkthrough takes you through the actual UI, highlighting buttons, menus, and features as you go. Complete a walkthrough to mark it done, or skip ahead if you already know the basics.

You can reopen the widget anytime from the sidebar to revisit walkthroughs or check your progress.

Getting Started​

The widget appears automatically when you sign up or create a new project. If you've dismissed it and want it back, look for the help icon in the sidebar.

For detailed documentation on each feature, check out:

• Playground Quick Start
• Evaluation Quick Start
• Observability Quick Start

You can now click directly from any trace to the application, variant, version, or environment that generated it. This makes debugging much faster. Instead of manually searching for which configuration produced a specific output, you can jump there in one click.

This is especially useful when you're investigating production issues. You see a problematic trace, click through to the exact prompt version, and start iterating immediately.

Key Capabilities​

• Clickable Application Links: Jump from a trace to the application that generated it
• Variant Navigation: Go directly to the specific variant and version used
• Environment Context: See and navigate to the environment (production, staging, development) where the trace originated
• Drawer Integration: Links appear in both the trace table and the detailed drawer view

How to Add References to Your Traces​

For the navigation links to appear, you need to store references in your traces. Here's how to do it with the Python SDK and OpenTelemetry.

Using the Python SDK​

import agenta as ag

ag.init()

# Store references to link traces to your configuration
ag.tracing.store_refs({
    "application.slug": "my-chatbot",
    "variant.slug": "gpt-4-optimized",
    "variant.version": "3",
    "environment.slug": "production",
})

# Your LLM calls are now linked to this configuration
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)

Using OpenTelemetry​

If you're using OpenTelemetry for instrumentation, add references as span attributes:

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('my-app');
const span = tracer.startSpan('chat-interaction');

// Add references to link the trace
span.setAttribute('ag.refs.application.slug', 'my-chatbot');
span.setAttribute('ag.refs.variant.slug', 'gpt-4-optimized');
span.setAttribute('ag.refs.variant.version', '3');
span.setAttribute('ag.refs.environment.slug', 'production');

// Your code here
span.end();

Available Reference Keys​

You can use any combination of these reference keys:

Use Cases​

Debug Production Issues​

When a user reports a problem, find the trace and click through to see exactly which prompt version was used. No more guessing or searching through commit history.

Compare Configurations​

Filter traces by variant, then click through to compare how different configurations perform. Jump between variants to understand what changed.

Track Deployments​

See which environment generated each trace. Verify that production is running the expected version by clicking through from any trace.

Getting Started​

Learn more about storing references in our documentation:

• Reference Prompt Versions (Python SDK)
• Semantic Conventions (OpenTelemetry)
• Observability Overview

When you compare evaluation results from last week to today, how do you know the test data didn't change? You don't. Until now.

Test set versioning tracks every change to your test sets. Each edit, upload, or programmatic update creates a new version. Evaluations link to specific versions, so you can trust your comparisons.

We also rebuilt the test set UI from scratch. It handles hundreds of thousands of rows without slowing down. Editing is faster, especially for chat messages and complex JSON data.

Test Set Versioning​

Every change to a test set creates a new version. You can see the version history, compare versions, and revert to previous versions.

What gets versioned:

• Adding, editing, or deleting test cases
• Uploading new data (CSV, JSON)
• Programmatic updates via SDK or API
• Column changes

Evaluation linking: When you run an evaluation, it links to the specific test set version used. This means:

• You can compare evaluations knowing they used the same test data
• If someone updates the test set, your historical evaluations still reference the original version
• You can filter evaluations by test set version

Programmatic versioning: Upload test sets via the SDK or API. The system detects changes and creates new versions automatically.

import agenta as ag

# Upload a test set - creates a new version if content changed
testset = ag.testsets.upload(
    name="my-test-set",
    data=test_cases,  # Your test case data
)

# The testset object includes version information
print(f"Version: {testset.version}")

New Test Set UI​

The test set view is completely rebuilt. It uses virtualized rendering, so it stays fast with large datasets.

What's new:

• Scale: Handle 100,000+ rows without performance issues
• JSON support: View and edit complex JSON directly. Toggle between raw JSON and formatted views
• String or JSON columns: Choose how each column stores data. Use JSON for structured data like chat messages

Chat message editing: Test cases with chat messages (like [{"role": "user", "content": "..."}]) now have a dedicated editor. Add, remove, or reorder messages. Edit content with proper formatting.

Upload options:

• Upload CSV or JSON files
• Create test sets in the UI
• Create programmatically via SDK
• Add spans from observability to test sets

Traceability​

Everything connects. When you view a trace in observability:

• See which test case it came from
• See which test set version
• Filter traces by test case or test set

When you view an evaluation:

• See the exact test set version used
• Compare only evaluations that used the same version
• Navigate to the test set to see the data

Getting Started​

Test set versioning is automatic. Any change creates a new version.

To use versioned test sets in evaluations:

1. Create or upload a test set
2. Make your edits (each save creates a version)
3. Run an evaluation (it links to the current version)
4. Later, compare evaluations knowing they used the same test data

For programmatic access, check the test sets documentation.

Overview​

This release brings three quality-of-life improvements to the Playground that make testing and iterating on prompts faster and more convenient.

What's New​

Provider Cost Display​

You can now see the cost per million tokens directly in the provider selection dropdown. This helps you make informed decisions about which model to use based on both capability and cost. No more switching to external pricing pages to compare costs.

Run Evaluations from the Playground​

You can now trigger evaluations directly from the Playground without navigating to the evaluation menu. When you're testing a prompt and want to run a full evaluation, click the evaluate button to start an evaluation run with your current configuration. This keeps you in flow when iterating on prompts.

Collapsible Test Cases​

Test cases in the Playground can now be collapsed. This is especially useful when working with large test sets or test cases with long inputs and outputs. Collapse completed test cases to focus on what you're working on. You can still see a preview of each test case to maintain context while navigating through your data.

Getting Started​

These features are available now in the Playground. Start using them today:

• Open the Playground and check the provider dropdown to see costs
• Click the evaluate button to run evaluations directly
• Use the collapse controls on test cases to manage your view

Overview​

Chat sessions bring conversation-level observability to Agenta. You can now group related traces from multi-turn conversations together, making it easy to analyze complete user interactions rather than individual requests.

This feature is essential for debugging chatbots, AI assistants, and any application with multi-turn conversations. You get visibility into the entire conversation flow, including costs, latency, and intermediate steps.

Key Capabilities​

• Automatic Grouping: All traces with the same ag.session.id attribute are automatically grouped together
• Session Analytics: Track total cost, latency, and token usage per conversation
• Session Browser: Dedicated UI showing all sessions with first input, last output, and key metrics
• Session Drawer: Detailed view of all traces within a session with parent-child relationships
• Real-time Monitoring: Auto-refresh mode for monitoring active conversations

How to Use Sessions​

Using the Python SDK​

Add session tracking to your application with one line of code:

import agenta as ag

# Initialize Agenta
ag.init()

# Store the session ID for all subsequent traces
ag.tracing.store_session(session_id="conversation_123")

# Your LLM calls are automatically tracked with this session
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)

Using the Chat Run Endpoint​

You can also instrument sessions when calling Agenta-managed prompts via the /chat/run endpoint:

import agenta as ag

# Initialize the Agenta client
agenta = ag.Agenta(api_key="your_api_key")

# Call the chat endpoint with session tracking
response = agenta.run(
    base_id="your_base_id",
    environment="production",
    inputs={
        "chat_history": [
            {"role": "user", "content": "What is the weather like?"}
        ]
    },
    # Add session metadata to group related conversations
    metadata={
        "ag.session.id": "user_456_conv_789"
    }
)

# Follow-up in the same session
follow_up = agenta.run(
    base_id="your_base_id",
    environment="production",
    inputs={
        "chat_history": [
            {"role": "user", "content": "What is the weather like?"},
            {"role": "assistant", "content": response["message"]},
            {"role": "user", "content": "What about tomorrow?"}
        ]
    },
    metadata={
        "ag.session.id": "user_456_conv_789"  # Same session ID
    }
)

Using OpenTelemetry​

If you're using OpenTelemetry for instrumentation:

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('my-app');
const span = tracer.startSpan('chat-interaction');

// Add session ID as a span attribute
span.setAttribute('ag.session.id', 'conversation_123');

// Your code here
span.end();

The UI automatically detects session IDs and groups traces together. You can use any format for session IDs: UUIDs, composite IDs like user_123_session_456, or custom formats.

Use Cases​

Debug Chatbots​

See the complete conversation flow when users report issues. Instead of viewing isolated requests, you can analyze the entire conversation context and understand why a particular response was generated.

Monitor Multi-turn Agents​

Track how your agent handles follow-up questions and maintains context across turns. See which turns are expensive, identify where latency spikes occur, and understand conversation patterns.

Analyze Conversation Costs​

Understand which conversations are expensive and why. Session-level cost tracking helps you identify optimization opportunities and set appropriate pricing for your application.

Optimize Performance​

Identify latency issues across entire conversations, not just single requests. See which conversational patterns lead to performance problems and optimize accordingly.

Getting Started​

Learn more in our documentation:

• Track Chat Sessions (Python SDK)
• Session Tracking (OpenTelemetry)
• Observability Overview

What's Next​

We're continuing to enhance session tracking with upcoming features like session-level annotations, session comparisons, and automated session analysis.

What is JSON Multi-Field Match?​

This evaluator compares specific fields between your model's JSON output and the expected JSON values from your test set. Unlike the old JSON Field Match evaluator (which only checked one field), this evaluator handles any number of fields at once.

For each field you configure, the evaluator produces a separate score (either 1 for a match or 0 for no match). It also calculates an aggregate score showing the percentage of fields that matched correctly.

Key Features​

Multiple Field Comparison​

Configure as many fields as you need to validate. The evaluator checks each field independently and reports results for all of them.

If you're extracting user information, you might configure fields like name, email, phone, and address.city. Each field gets its own score, so you can see exactly which extractions succeeded and which failed.

Three Path Format Options​

The evaluator supports three different ways to specify field paths:

Dot notation (recommended for most cases):

• Simple fields: name, email
• Nested fields: user.address.city
• Array indices: items.0.name

JSON Path (standard JSON Path syntax):

• Simple fields: $.name, $.email
• Nested fields: $.user.address.city
• Array indices: $.items[0].name

JSON Pointer (RFC 6901):

• Simple fields: /name, /email
• Nested fields: /user/address/city
• Array indices: /items/0/name

All three formats work the same way. Use whichever matches your existing tooling or personal preference.

Nested Field and Array Support​

Access deeply nested fields and array elements without restrictions. The evaluator handles any level of nesting.

Per-Field Scoring​

See individual scores for each configured field in the evaluation results. This granular view helps you identify which specific extractions are working well and which need improvement.

Aggregate Score​

The aggregate score shows the percentage of matching fields. If you configure five fields and three match, the aggregate score is 0.6 (or 60%).

Example​

Suppose you're building an entity extraction model that pulls contact information from text. Your ground truth looks like this:

{
  "name": "John Doe",
  "email": "john@example.com",
  "phone": "555-1234",
  "address": {
    "city": "New York",
    "zip": "10001"
  }
}

Your model produces this output:

{
  "name": "John Doe",
  "email": "jane@example.com",
  "phone": "555-1234",
  "address": {
    "city": "New York",
    "zip": "10002"
  }
}

You configure these fields: ["name", "email", "phone", "address.city", "address.zip"]

The evaluator returns:

You can see immediately that the model got the email and zip code wrong but correctly extracted the name, phone, and city.

Auto-Detection in the UI​

When you configure the evaluator in the web interface, Agenta automatically detects available fields from your test set data. Click to add or remove fields using a tag-based interface. This makes setup fast and reduces configuration errors.

Migration from JSON Field Match​

The old JSON Field Match evaluator only supported checking a single field. If you're using it, consider migrating to JSON Multi-Field Match to gain:

• Support for multiple fields in one evaluator
• Per-field scoring for detailed analysis
• Aggregate scoring for overall performance tracking
• Nested field and array support

Existing JSON Field Match configurations continue to work. We recommend migrating to JSON Multi-Field Match for new evaluations.

Next Steps​

Learn more about configuring and using the JSON Multi-Field Match evaluator in the Classification and Entity Extraction Evaluators documentation.

What is PDF Support?​

PDF support lets you attach PDF documents to chat messages when testing prompts in the Playground. The feature works with vision-capable models from OpenAI, Gemini, and Claude. These models can read and understand PDF content to answer questions or perform analysis.

This is useful when you're building applications that need to work with documents. Examples include invoice processing, contract analysis, document Q&A, or content extraction.

Supported Providers​

PDF support works with vision-capable models that handle document inputs.

How to Attach PDFs​

To attach a PDF to a chat message, click "Add attachment" in the message input. You'll see three options:

Upload a File​

Select a PDF from your computer. The file is converted to base64 and sent with your prompt.

Provide a URL​

Paste the URL to a publicly accessible PDF. The model fetches the PDF from the URL.

Use a File ID​

If you've uploaded a file through a provider's API (like the Gemini Files API), you can use the file ID instead. The model retrieves the file from the provider's storage.

Using PDFs in Evaluations​

PDF attachments work in both automatic and human evaluations. You can include PDFs in your test sets and run evaluations across multiple documents.

PDFs in Observability and Tracing​

When you trace requests that include PDFs, you can see the PDF attachment information in the trace data.

Example Use Cases​

Invoice Processing​

Create a prompt that extracts key information from invoices:

Extract the following information from this invoice:
- Invoice number
- Date
- Total amount
- Vendor name
- Line items

Return the information as structured JSON.

Attach sample invoices as PDFs. Test the prompt with different invoice formats to ensure reliable extraction across vendors.

Contract Analysis​

Build a prompt that analyzes legal contracts:

Review the attached contract and identify:
- Key obligations for each party
- Important dates and deadlines
- Termination clauses
- Liability limitations

Provide a summary in plain language.

Attach contract PDFs and verify that the model identifies critical terms consistently.

Document Q&A​

Create an assistant that answers questions about documents:

You are a document assistant. Answer the user's question based on the
attached PDF. Be specific and cite page numbers when possible.

Question: {{question}}

Attach various document types (reports, manuals, research papers) and test question-answering accuracy across different content.

Next Steps​

Learn more about using the Playground to develop and test prompts with PDF attachments.

The MCP server implements the Model Context Protocol, allowing AI assistants to search and retrieve Agenta documentation on demand. Instead of manually searching docs, your AI agent can answer questions about Agenta features, APIs, and code examples.

Read the full setup guide →

What Are Projects?​

Projects provide a way to isolate and organize your AI work within an organization. Each project maintains its own scope for:

• Prompts: All prompt templates and variants stay within the project
• Traces: Observability data is scoped to the project that generated it
• Evaluations: Test sets, evaluators, and evaluation results belong to specific projects

This scoping prevents clutter and makes it easy to focus on one product at a time.

Creating and Managing Projects​

You can create a new project directly from the sidebar in the Agenta interface. Once created, you can switch between projects using the sidebar navigation.

Each team member can work in different projects simultaneously. The interface remembers your last active project, making it easy to pick up where you left off.

When to Use Projects​

Projects work well when you need to:

• Build multiple AI products for different use cases
• Separate development work for different teams or departments
• Keep client work isolated from internal tools

Next Steps​

If you're managing complex AI initiatives across multiple products, projects give you the structure to keep everything organized. You can create your first project from the sidebar and start organizing your prompts and evaluations.

For questions about projects or organizational structure, check the FAQ or reach out through our support channels.

What Are Provider Built-in Tools?​

Provider built-in tools are capabilities that LLM providers offer natively. Unlike custom tools that you define with JSON schemas, these tools are managed by the provider. When the model needs them, the provider handles execution and returns results automatically.

Common built-in tools include:

• Web search: Fetch current information from the internet
• Code execution: Run Python or JavaScript code
• File search: Search through uploaded documents
• Bash scripting: Execute shell commands (Anthropic)

Supported Providers and Tools​

Different providers offer different built-in tools:

OpenAI​

• Web Search: Access current information from the web
• File Search: Search through files you upload to OpenAI

Anthropic​

• Web Search: Retrieve information from the internet
• Bash Scripting: Execute bash commands in a sandboxed environment

Gemini​

• Web Search: Search the web for current information
• Code Execution: Run Python code to perform calculations and data analysis

How to Use Built-in Tools​

Adding Tools in the Playground​

1. Open your prompt in the Playground
2. Click the "Add Tool" button in the configuration panel
3. Choose the tools you want to enable for your prompt
4. Test your prompt; the model will automatically use tools when needed

The tools are saved with your prompt configuration. When you commit changes, the tool configuration is stored with the variant.

Invoking with Tools via LLM Gateway​

When you invoke prompts through Agenta as an LLM gateway, the tools are automatically included in the request. The provider handles tool execution during the call.

Your application receives the final response after all tool calls complete. You don't need to handle tool execution yourself.

Tool Definitions in the Registry​

Tool definitions follow the LiteLLM format. You can view the exact tool schemas in the Prompt Registry. This helps you understand what parameters each tool accepts and how the provider will use it.

Example Use Cases​

Research Assistant with Web Search​

Create a prompt that answers questions using current information:

You are a research assistant. Answer the user's question with accurate,
current information. Use web search when you need recent data.

Question: {{question}}

Enable web search in the tool configuration. When users ask about current events or recent data, the model automatically searches the web for information.

Data Analysis with Code Execution​

Build a data analysis prompt that performs calculations:

Analyze the following data and provide insights:

{{data}}

Calculate statistics and create visualizations as needed.

Enable code execution for Gemini. The model can run Python code to calculate statistics, process data, and generate visualizations.

Document Q&A with File Search​

Create a prompt that answers questions about uploaded documents:

Answer the user's question based on the uploaded documentation.
Be specific and cite relevant sections.

Question: {{question}}

Enable file search for OpenAI. The model searches through your uploaded files to find relevant information.

Next Steps​

Learn more about using the Playground to develop and test prompts with provider built-in tools.

Reasoning effort controls how much computational thinking the model applies before generating a response. This is particularly useful for complex reasoning tasks where you want to balance response quality with latency and cost.

The reasoning effort parameter is part of your prompt template configuration. When you fetch prompts via the SDK or invoke them through Agenta as an LLM gateway, the reasoning effort setting is included in the configuration and applied to your requests automatically.

This gives you fine-grained control over model behavior directly from the playground, making it easier to optimize for your specific use case.

This means you can add sophisticated logic directly into your prompt templates. Use conditional statements, apply filters to variables, and transform data on the fly.

Learn more in our blog post or check the documentation.

Example​

Here's a prompt template that uses Jinja2 to adapt based on user expertise level:

You are {% if expertise_level == "beginner" %}a friendly teacher who explains concepts in simple terms{% else %}a technical expert providing detailed analysis{% endif %}.

Explain {{ topic }} {% if include_examples %}with practical examples{% endif %}.

{% if False %} {{expertise_level}} {{include_examples}} {% endif %}

Note: The {% if False %} block makes variables available to the playground without including them in the final prompt.

Using Jinja2 Prompts​

When you fetch a Jinja2 prompt via the SDK, you get the template format included in the configuration:

{
  "prompt": {
    "messages": [
      {
        "role": "user",
        "content": "You are {% if expertise_level == \"beginner\" %}a friendly teacher...{% endif %}"
      }
    ],
    "llm_config": {
      "model": "gpt-4",
      "temperature": 0.7
    },
    "template_format": "jinja2"
  }
}

The template_format field tells Agenta how to process your variables. This works both when invoking prompts through Agenta as an LLM gateway and when fetching prompts programmatically via the SDK.

What's Open Source​

Every feature you need to build, test, and deploy LLM applications is now open source. This includes the evaluation system, prompt playground and management, observability, and all core workflows.

You can run evaluations using LLM-as-a-Judge, custom code evaluators, or any built-in evaluator. Create and manage test sets. Evaluate end-to-end workflows or specific spans in traces.

Experiment with prompts in the playground. Version and commit changes. Deploy to environments. Fetch configurations programmatically.

Trace your LLM applications with OpenTelemetry support. View detailed execution traces. Monitor costs and performance. Filter and search traces.

Building in Public Again​

We've moved development back to the public repository. You can see what we're building, contribute features, and shape the product direction.

What Remains Under Commercial License​

Only enterprise collaboration features stay under a separate license. This includes role-based access control (RBAC), single sign-on (SSO), and audit logs. These features support teams with specific compliance and security requirements.

Get Started​

Follow the self-hosting quick start guide to deploy Agenta on your infrastructure. View the source code and contribute on GitHub. Read why we made this decision at agenta.ai/blog/commercial-open-source-is-hard-our-journey.

What This Means for You​

You can run Agenta on your infrastructure with full access to evaluation, prompting, and observability features. You can modify the code to fit your needs. You can contribute back to the project.

The MIT license gives you freedom to use, modify, and distribute Agenta. We believe open source creates better products through community collaboration.

Why Programmatic Evaluation?​

Complex AI agents need evaluation that goes beyond UI-based testing. The Evaluation SDK provides code-level control over test data and evaluation logic. You can test agents built with any framework. Run evaluations in your CI/CD pipeline. Debug complex workflows with full trace visibility.

Key Capabilities​

Test Data Management​

Create test sets directly in your code or fetch existing ones from Agenta. Test sets can include ground truth data for reference-based evaluation or work without it for evaluators that only need the output.

Built-in Evaluators​

The SDK includes LLM-as-a-Judge, semantic similarity, and regex matching evaluators. You can also write custom Python evaluators for your specific requirements.

Reusable Configurations​

Save evaluator configurations in Agenta to reuse them across runs. Configure an evaluator once, then reference it in multiple evaluations.

Span-Level Evaluation​

Evaluate your agent end to end or test specific spans in the execution trace. Test individual components like retrieval steps or tool calls separately.

Run on Your Infrastructure​

Evaluations run on your infrastructure. Results appear in the Agenta dashboard with full traces and comparison views.

Getting Started​

Install the SDK:

pip install agenta

Here's a minimal example evaluating a simple agent:

import agenta as ag
from agenta.sdk.evaluations import aevaluate

# Initialize
ag.init()

# Define your application
@ag.application(slug="my_agent")
async def my_agent(question: str):
    # Your agent logic here
    return answer

# Define an evaluator
@ag.evaluator(slug="correctness_check")
async def correctness_check(expected: str, outputs: str):
    return {
        "score": 1.0 if outputs == expected else 0.0,
        "success": outputs == expected,
    }

# Create test data
testset = await ag.testsets.acreate(
    name="Agent Tests",
    data=[
        {"question": "What is 2+2?", "expected": "4"},
        {"question": "What is the capital of France?", "expected": "Paris"},
    ],
)

# Run evaluation
result = await aevaluate(
    name="Agent Correctness Test",
    testsets=[testset.id],
    applications=[my_agent],
    evaluators=[correctness_check],
)

print(f"View results: {result['dashboard_url']}")

Dashboard Integration​

Every evaluation run gets a shareable dashboard link. The dashboard shows full execution traces, comparison views for different versions, aggregated metrics, and individual test case details.

Next Steps​

Check out the Quick Start Guide to build your first evaluation.