Skip to content

Opik Observability

ACE integrates with Opik for tracing, cost tracking, and performance monitoring. All Opik tracing is explicit opt-in — it is never auto-enabled just because the package is installed.

Two independent tracing modes:

  1. Pipeline step (OpikStep) — client-agnostic, logs one Opik trace per sample with ACE context fields.
  2. LiteLLM callback (register_opik_litellm_callback) — LiteLLM-specific, tracks per-LLM-call tokens and costs.

Installation

uv add ace-framework[observability]

Quick Start

from ace import ACELiteLLM

# Easiest: ACELiteLLM enables both tracing modes with one flag
ace = ACELiteLLM.from_model("gpt-4o-mini", opik=True, opik_project="my-experiment")

from ace import (
    ACE, OpikStep,
    Agent, Reflector, SkillManager,
    SimpleEnvironment,
)

# Manual: Add OpikStep via extra_steps
runner = ACE.from_roles(
    agent=Agent("gpt-4o-mini"),
    reflector=Reflector("gpt-4o-mini"),
    skill_manager=SkillManager("gpt-4o-mini"),
    environment=SimpleEnvironment(),
    extra_steps=[OpikStep(project_name="my-experiment")],
)

# LLM-level cost tracking only (no pipeline traces)
from ace import register_opik_litellm_callback

registered = register_opik_litellm_callback(project_name="my-experiment")

Starting the Opik Server

docker run -d -p 5173:5173 --name opik ghcr.io/comet-ml/opik:latest

# View traces at http://localhost:5173

export COMET_API_KEY="your-api-key"
# Traces appear at https://www.comet.com/opik

OpikStep

OpikStep is a terminal side-effect step that logs one Opik trace per sample. It reads context fields but never mutates them — safe to append to any pipeline.

Parameters

Parameter Type Default Description
project_name str "ace-framework" Opik project for organizing traces
tags list[str] None Extra tags attached to every trace

What Gets Logged

Each trace includes:

Field Source
Input Question and context from the sample
Output Answer, reasoning, and skill IDs from AgentOutput
Metadata Epoch, step index, skill count, reflection insights, operation counts
Feedback scores Accuracy extracted from environment feedback (correct / incorrect)

Trace Hierarchy

graph TD
    P["Project: my-experiment"]
    P --> T["Trace: sample_run_001"]
    T --> I["Input: question + context"]
    T --> O["Output: answer + reasoning + skill_ids"]
    T --> M["Metadata: epoch=2, skills=12, ops=3"]
    T --> F["Feedback: accuracy=1.0"]
    T --> L["LLM Calls (automatic)"]
    L --> L1["agent_generate — 450 tokens, $0.0003"]
    L --> L2["reflector_reflect — 620 tokens, $0.0004"]
    L --> L3["skill_manager_update — 380 tokens, $0.0002"]

LLM Cost Tracking

OpikStep does not register the LiteLLM callback — the two tracing modes are independent. To get per-LLM-call cost tracking, call register_opik_litellm_callback() separately:

from ace import register_opik_litellm_callback

success = register_opik_litellm_callback(project_name="cost-tracking")
# Returns True if registered, False if Opik unavailable

Every LLM call is then automatically tracked with:

  • Input / output tokens
  • Model used
  • Cost per call
  • Latency

When using ACELiteLLM with opik=True, both modes are enabled together automatically — no need to call register_opik_litellm_callback() manually.

Environment Variables

Variable Description Default
OPIK_PROJECT_NAME Project name for organizing traces ace-framework
OPIK_DISABLED=true Disable all Opik tracing Not set
OPIK_ENABLED=false Alternative way to disable tracing Not set
OPIK_URL_OVERRIDE Custom Opik server URL http://localhost:5173/api
OPIK_WORKSPACE Opik workspace name default

Error Handling

When using ACELiteLLM with opik=True, errors are raised immediately:

  • ImportError if the opik package is not installed
  • RuntimeError if the Opik client fails to initialize (bad config, disabled via env vars)

This ensures you know immediately if tracing is broken, rather than discovering missing traces later.

When using OpikStep directly via extra_steps, it soft-imports Opik and silently becomes a no-op if the package is absent — useful for pipelines that should work with or without observability.

from ace import OPIK_AVAILABLE

if OPIK_AVAILABLE:
    print("Opik tracing is available")

Troubleshooting: ~/.opik.config

The Opik SDK stores a global config file at ~/.opik.config (created by opik.configure()). This file overrides environment variables and can cause silent failures if it contains stale settings.

If traces aren't appearing, check:

cat ~/.opik.config

A correct config for Comet Cloud looks like:

[opik]
url_override = https://www.comet.com/opik/api/
workspace = your-workspace-name

Common issues:

  • Wrong URL: https://www.comet.com/api/ (missing /opik/) causes 404 errors
  • Wrong workspace: workspace = default instead of your actual workspace name
  • Stale config: Re-run opik.configure() or edit the file directly to fix

Disabling Tracing

# In CI or tests
OPIK_DISABLED=true pytest tests/

# Or via the alternative variable
OPIK_ENABLED=false python my_script.py

Full Example

from ace import ACELiteLLM, Sample, SimpleEnvironment

ace = ACELiteLLM.from_model("gpt-4o-mini", opik=True, opik_project="ace-training")

samples = [
    Sample(question="What is 2+2?", context="", ground_truth="4"),
    Sample(question="Capital of France?", context="", ground_truth="Paris"),
]

results = ace.learn(samples, environment=SimpleEnvironment(), epochs=3)
ace.save("trained.json")

# View traces at http://localhost:5173 → project "ace-training"

from ace import (
    ACE, Agent, Reflector, SkillManager, Skillbook,
    SimpleEnvironment, Sample, OpikStep,
    register_opik_litellm_callback,
)

runner = ACE.from_roles(
    agent=Agent("gpt-4o-mini"),
    reflector=Reflector("gpt-4o-mini"),
    skill_manager=SkillManager("gpt-4o-mini"),
    environment=SimpleEnvironment(),
    extra_steps=[OpikStep(project_name="ace-training")],
)

# Optionally add LLM-level cost tracking
register_opik_litellm_callback(project_name="ace-training")

samples = [
    Sample(question="What is 2+2?", context="", ground_truth="4"),
    Sample(question="Capital of France?", context="", ground_truth="Paris"),
]

results = runner.run(samples, epochs=3)
runner.save("trained.json")