Traces and Live Debugger

Overview

The Inkeep Agent Framework provides powerful traces and live debugging capabilities powered by SigNoz. This gives you:

• Real-time trace visualization - See exactly how your agents execute step-by-step
• Live debugging - Debug agent conversations as they happen
• Full observability - Complete OpenTelemetry instrumentation for monitoring
• Performance insights - Identify bottlenecks and optimize agent performance

Why Set Up SigNoz?

SigNoz powers two critical features:

1. Traces Interface - Visual representation of agent execution flows
2. Live Debugger - Real-time debugging capabilities for agent conversations

Setup Options

You can set up SigNoz in two ways:

1. Cloud Setup: Use SigNoz Cloud
2. Local Setup: Run SigNoz locally using Docker

Option 1: SigNoz Cloud Setup

Step 1: Create a SigNoz Cloud Project

1. Sign up at SigNoz
2. Create a new project or use an existing one

Step 2: Save Your SigNoz Credentials

You'll need to collect three pieces of information from your SigNoz dashboard:

1. API Key:

   • Navigate to Settings → Workspace Settings → API Keys → New Key
   • Choose any role (Admin, Editor, or Viewer) - Viewer is sufficient for observability
   • Set the expiration field to "No Expiry" to prevent the key from expiring
   • Copy the generated API key

2. Ingestion Key:

   • Navigate to Settings → Workspace Settings → Ingestion
   • Set the expiration field to "No Expiry" to prevent the key from expiring
   • Copy the ingestion key

3. SigNoz URL:

   • Copy the URL from your browser's address bar
   • It will look like: https://<your-organization>.signoz.cloud

Step 3: Configure Your Root .env File

# SigNoz
SIGNOZ_URL=https://<your-organization>.signoz.cloud
SIGNOZ_API_KEY=<your-api-key>

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ingest.us.signoz.cloud:443/v1/traces
OTEL_EXPORTER_OTLP_TRACES_HEADERS="signoz-ingestion-key=<your-ingestion-key>"

Step 4: Verify Cloud Setup

1. Restart your development environment:

   pnpm dev

2. Generate some traces by interacting with your agents

3. Open your SigNoz cloud dashboard and navigate to "Traces" to see your agent traces

Option 2: Local SigNoz Setup

Prerequisites

• Docker installed on your machine

Step 1: Clone the Optional Services Repository

Clone the Inkeep optional local development services repository:

git clone https://github.com/inkeep/agents-optional-local-dev
cd agents-optional-local-dev

Step 2: Configure SigNoz Credentials (Optional)

In the agents-optional-local-dev repository, you can optionally set credentials by adding these to your .env file:

SIGNOZ_ADMIN_EMAIL=admin@example.com
SIGNOZ_ADMIN_PASSWORD="adminADMIN#1"

Step 3: Start SigNoz Services

Run the following command to start SigNoz and related services:

docker-compose --profile signoz up -d

This will start:

• SigNoz frontend (accessible at http://localhost:3080)
• SigNoz query service
• SigNoz OTEL collector
• ClickHouse database

When you visit http://localhost:3080, you can log in using the credentials you configured above.

Step 4: Configure Environment Variables

In your root project directory (of my-agent-directory), update your .env file:

# SigNoz Configuration
SIGNOZ_URL=http://localhost:3080
SIGNOZ_API_KEY=your-signoz-api-key

# IMPORTANT: Comment out the OTEL Configuration
# OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ingest.us.signoz.cloud:443/v1/traces
# OTEL_EXPORTER_OTLP_TRACES_HEADERS="signoz-ingestion-key=<your-ingestion-key>"

To get your SigNoz API key:

1. Open SigNoz at http://localhost:3080
2. Navigate to Settings → Account Settings → API Keys → New Key
3. Create a new API key or copy an existing one.
   • Choose any role (Admin, Editor, or Viewer) - Viewer is sufficient for observability
   • Set the expiration field to "No Expiry" to prevent the key from expiring

Step 5: Verify Setup

1. Restart your Inkeep agents:

   pnpm dev

2. Make some requests to your agents to generate traces

3. Open SigNoz at http://localhost:3080 and navigate to the "Traces" section to see your agent traces

Viewing Traces and Using the Live Debugger

Once SigNoz is set up, you can access traces and live debugging in two ways:

1. Visual Builder Traces Interface

If you're using the Visual Builder:

1. Open your agent project in the Visual Builder
2. Navigate to the Traces section
3. You'll see real-time traces of your agent executions
4. Click on any trace to see detailed execution flow and timing

The traces overview shows conversation metrics and recent activity:

Click on any conversation to see detailed execution flow:

2. SigNoz Dashboard

For detailed analysis and further debugging:

1. Open your SigNoz dashboard (cloud or local)
2. Navigate to Traces to see all agent executions
3. Use filters to find specific conversations or agents
4. Click on traces to see:
   • Step-by-step execution details
   • Performance metrics
   • Error information
   • Agent-to-agent communication flows

For more detailed information on using traces, see the SigNoz Usage guide.

Additional Observability: Langfuse Integration

👉 If you want to connect to Langfuse for specialized LLM observability alongside SigNoz see the Langfuse Usage guide for complete setup instructions.