Monitoring

Gain deep insights into your prompts' performance, costs, and behavior in production environments with Basalt's monitoring system. On this page, we'll explore how to use the monitoring API to track and analyze your AI application's performance.

The monitoring model

The monitoring system is built around traces and logs, allowing you to track the execution of your AI features and analyze their performance. Each trace represents a single execution of a feature, and logs represent individual steps or operations within that trace.

Properties

Name

trace

Type

object

Description

The top-level container for a monitoring session.

{
  id: string;
  chainSlug: string;
  input?: string;
  output?: string;
  metadata?: Record<string, any>;
  organization?: {
    id: string;
    name: string;
  };
  user?: {
    id: string;
    name: string;
  };
  startTime?: string;
  endTime?: string;
}

Name

log

Type

object

Description

An individual operation or step within a trace.

{
  id: string;
  traceId: string;
  type: 'span' | 'generation' | 'function' | 'tool' | 'retrieval' | 'event';
  name: string;
  startTime?: string;
  endTime?: string;
  metadata?: Record<string, any>;
  parentId?: string;
  input?: string;
  output?: string;
  prompt?: {
    slug: string;
    version?: string;
    tag?: string;
  };
  variables?: Array<{
    label: string;
    value: string;
  }>;
}

Log Types

The monitoring system supports various log types to represent different operations:

span: A way to group sub logs together
generation: Represents a prompt execution with an LLM
function: Represents a function call
tool: Represents a tool call such as an MPC call
retrieval: Represents a retrieval operation
event: For any other type of log

POST/monitor/trace

Create a trace

This endpoint allows you to create a new trace to track the execution of a feature. You can optionally include logs to record individual steps within the trace.

Required parameters

Name
chainSlug
Type
string
Description
The unique identifier for the feature being monitored.

Optional parameters

Name
input
Type
string
Description
The input provided to the feature.
Name
output
Type
string
Description
The output produced by the feature.
Name
metadata
Type
object
Description
Additional metadata about the trace.
Name
organization
Type
object
Description
Information about the organization associated with this trace.
Name
user
Type
object
Description
Information about the user associated with this trace.
Name
startTime
Type
string
Description
ISO timestamp for when the trace started.
Name
endTime
Type
string
Description
ISO timestamp for when the trace ended.
Name
logs
Type
array
Description
Array of log objects to be created with the trace.

Workspace context

All requests must include workspace authentication. The endpoint will only create traces for features that belong to the authenticated workspace's scope.

Request

POST

/monitor/trace

curl -X POST https://api.getbasalt.ai/monitor/trace \
  -H "Authorization: Bearer {apiKey}" \
  -H "Content-Type: application/json" \
  -d '{
    "chainSlug": "customer-support",
    "input": "How do I reset my password?",
    "output": "You can reset your password by clicking the Forgot Password link on the login page.",
    "metadata": {
      "sessionId": "sess_12345",
      "channel": "web"
    },
    "organization": {
      "id": "org_12345",
      "name": "Acme Inc."
    },
    "user": {
      "id": "user_12345",
      "name": "John Doe"
    },
    "logs": [
      {
        "id": "client-log-1",
        "type": "generation",
        "name": "Password Reset Instructions",
        "input": "How do I reset my password?",
        "output": "You can reset your password by clicking the Forgot Password link on the login page.",
      }
    ]
  }'

Response 200

{
  "trace": {
    "id": "trace_abc123def456"
  }
}

Response 404 (Feature not found)

{
  "error": "No feature with slug \"customer-support\" exists."
}

Response 500 (Server error)

{
  "error": "Failed to create trace"
}

POST/monitor/log

Create a prompt generation log

This endpoint allows you to create a new log to track the execution of a prompt generation. The Basalt prompt object is retrieved from the Basalt platform with an existing slug. If the version is not specified we fallback on the tag. If the tag or the version are not specified we fallback on the production version.

Required parameters

Name
name
Type
string
Description
A descriptive name for the log.
Name
slug
Type
string
Description
The slug of the prompt in the basalt platform.

Optional parameters

Name
version
Type
string
Description
The version of the prompt to use.
Name
tag
Type
string
Description
The tag of the prompt to use.
Name
startTime
Type
string
Description
ISO timestamp for when the log operation started.
Name
startTime
Type
string
Description
ISO timestamp for when the log operation started.
Name
endTime
Type
string
Description
ISO timestamp for when the log operation ended.
Name
metadata
Type
object
Description
Additional metadata about the log.
Name
input
Type
string
Description
The input provided to the operation.
Name
output
Type
string
Description
The output produced by the operation.
Name
organization
Type
object
Description
Information about the organization associated with this trace.
Name
user
Type
object
Description
Information about the user associated with this trace.

Request

POST

/monitor/log

curl -X POST https://api.getbasalt.ai/monitor/log \
  -H "Authorization: Bearer {apiKey}" \
  -H "Content-Type: application/json" \
  -d '{
    "slug": "subscription-help",
    "version": "2.1",
    "tag": "production"
    "name": "Fetch User Data",
    "startTime": "2023-06-15T10:30:00Z",
    "endTime": "2023-06-15T10:30:01Z",
    "metadata": {
      "functionName": "getUserData",
      "status": "success"
    },
    "input": "{\"userId\": \"user_12345\"}",
    "output": "{\"name\": \"John Doe\", \"email\": \"john@example.com\"}"
  }'

Response 200

{
  "log": {
    "id": "log_xyz789"
  }
}

Response 404 (Prompt not found)

{
  "error": "No prompt with slug \"subscription-help\" exists."
}

Response 500 (Server error)

{
  "error": "Failed to create log"
}

Using prompt logs

When logging prompt generations, you can include additional information about the prompt used. This allows you to track which prompts are being used and how they're performing. If you provide a prompt object, we will automatically retrieve the prompt from the Basalt platform and use it in the log. If you want to provide custom prompt information that is not stored in Basalt you need to omit the prompt object.

Prompt log structure

For logs of type "generation", you can include a prompt object with the following properties:

slug: The unique identifier for the prompt
version (optional): The specific version of the prompt used
tag (optional): The tag of the prompt version used (e.g., "production", "staging")

You can also include variables to record the values used to fill prompt templates.

Prompt log example

POST

/monitor/trace

{
  "chainSlug": "customer-support",
  "logs": [
    {
      "id": "client-log-1",
      "type": "generation",
      "name": "Customer Support Response",
      "input": "How do I cancel my subscription?",
      "output": "You can cancel your subscription by going to Settings > Subscription and clicking the Cancel button.",
      "prompt": {
        "slug": "subscription-help",
        "version": "2.1",
        "tag": "production"
      },
      "variables": [
        {
          "label": "inquiry",
          "value": "How do I cancel my subscription?"
        },
        {
          "label": "tone",
          "value": "helpful"
        }
      ]
    }
  ]
}

Nested logs and tracing

The monitoring system supports nested logs to represent complex operations. By specifying a parentId, you can create a hierarchy of logs that shows the relationship between different operations.

Creating nested logs

There are two ways to create nested logs:

Include all logs in the initial trace creation request, with client-side IDs and parentId references
Create logs separately and reference the parent log's ID in the parentId field

The system will automatically build the hierarchy based on these relationships.

Nested logs example

POST

/monitor/trace

{
  "chainSlug": "customer-support",
  "logs": [
    {
      "id": "span-1",
      "type": "span",
      "name": "Customer Support Response",
      "input": "How do I cancel my subscription?",
      "output": "You can cancel your subscription by going to Settings > Subscription and clicking the Cancel button."
    },
    {
      "id": "span-2",
      "type": "span",
      "parentId": "span-1",
      "name": "Classify question",
      "input": "How do I cancel my subscription?"
    },
    {
      "id": "log-1",
      "parentId": "span-2",
      "type": "generation",
      "name": "Classifier LLM",
      "input": "How do I cancel my subscription?",
      "output": "subscription-related"
    },
    {
      "id": "span-3",
      "type": "span",
      "parentId": "span-1",
      "name": "Translation",
      "input": "How do I cancel my subscription?"
    },
    {
      "id": "log-2",
      "parentId": "span-3",
      "type": "generation",
      "name": "Translation LLM",
      "input": "How do I cancel my subscription?",
      "output": "Comment puis-je annuler mon abonnement?"
    }
  ]
}

Stay Updated

We're continuously enhancing our monitoring capabilities. For the latest updates:

For early access to our monitoring features or to discuss specific monitoring needs, please contact our team.