Observability OTEL Endpoints Documentation¶

Note: To use the Observability APIs, set the environment variable USE_AIR_API_V2_BASE_URL=True in your SDK environment. Queries to the observability endpoints will then use https://api-prod-k8s.airefinery.accenture.com/. This feature is available starting from SDK version 1.25.0. This is a temporary setup, and we will transition to the regular URL soon.

This documentation provides an overview of the OpenTelemetry (OTel)-based observability endpoints within AI Refinery. These endpoints enable you to query logs, metrics, and distributed traces from your AI applications. You can access telemetry data through direct API calls to monitor application performance, debug issues, and gain insights into your AI workloads.

Overview¶

We provide access to three types of telemetry data, Logs, Metrics, and Traces , collected via OpenTelemetry. Thus, we have the following endpoints each for the corresponding telemetry data:

/logs - Query Loki for AIRefinery logs
- Logs, stored in Loki, capture time-stamped records of discrete events for debugging and auditing.
/metrics - Query Prometheus for AIRefinery metrics
- Metrics, stored in Prometheus, aggregate numerical measurements over time for monitoring performance trends.
/traces - Query Tempo for AIRefinery traces
- Traces, stored in Tempo, track request flows across AIRefinery services for identifying agent workflows and dependencies.

All endpoints support two-scope filtering:

Organization-level: Filter by organization_id (returns data for all projects)
Project-level: Filter by project_name (returns data for specific project)

Authentication¶

All endpoints require authentication, just like other AIRefinery services. A bearer access token in the request header is required for sdk version higher than 1.13.0.

-H "Authorization: Bearer <api-key>"

Additionally, the organization_id from the request is enforced. Tenants from each organization can only access observability data within their organization.

POST /observability/logs¶

Query Loki for AIRefinery logs. Users can view application logs with timestamps, filterable by labels and time range. These logs capture request handling, authentication flows, system interactions, and external dependency behavior, helping diagnose runtime issues and system health.

Parameters:¶

Parameter	Type	Required	Description
`organization_id`	string	No	Organization ID to filter logs (indexed as Loki label)
`project_name`	string	No	Project name to filter logs (indexed as Loki label)
`time_window`	string	No	Time range for logs (e.g., '5m', '1h', '24h'). Default: '24h'
`limit`	integer	No	Maximum number of log entries to return. Default: 500

Example Usage¶

Get logs for a specific organization within 1 hr:

curl -X POST https://api-prod-k8s.airefinery.accenture.com/observability/logs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
    -d '{"organization_id": "org-123", "time_window": "1h"}'

Get 100 logs for a specific project within 30min:

curl -X POST https://api-prod-k8s.airefinery.accenture.com/observability/logs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
  -d '{
    "organization_id": "org-123",
    "project_name": "project-x",
    "time_window": "30m",
    "limit": 100
  }'

POST /observability/metrics¶

Query Prometheus for application metrics. This endpoint provides access to a series of metrics covering inference performance, agent operations, and session analytics. For a complete list of available metrics and their descriptions, see the configuration of observability data retrieval.

Parameters:¶

Parameter	Type	Required	Description
`metric`	string	Yes	Metric name from the configuration of observability data retrieval. (e.g., 'token_consumption', 'agent_task_total')
`organization_id`	string	No	Organization ID to filter metrics
`project_name`	string	No	Project name to filter metrics
`agent_name`	string	No	Agent name to filter metrics (for agent metrics)
`model_key`	string	No	Model identifier for inference metrics
`time_window`	string	No	Time range for rate/increase queries (e.g., '5m', '1h', '24h'). Default: '1h'

Example Usage¶

Token consumption metrics (organization-level):

curl -X POST https://api-prod-k8s.airefinery.accenture.com/observability/metrics \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
  -d '{
    "metric": "token_consumption",
    "organization_id": "org-123",
    "time_window": "1h"
  }'

Agent task metrics (project-level):

curl -X POST https://api-prod-k8s.airefinery.accenture.com/observability/metrics \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
  -d '{
    "metric": "agent_task_total",
    "organization_id": "org-123",
    "project_name": "project-x"
  }'

Agent metrics with agent filter:

curl -X POST https://api-prod-k8s.airefinery.accenture.com/observability/metrics \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
  -d '{
    "metric": "agent_task_total",
    "organization_id": "org-123",
    "agent_name": "orchestrator",
    "time_window": "30m"
  }'

Inference metrics with model filter:

curl -X POST https://api-prod-k8s.airefinery.accenture.com/observability/metrics \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
  -d '{
    "metric": "inference_requests_total",
    "organization_id": "org-123",
    "model_key": "gpt-4",
    "time_window": "5m"
  }'

POST /observability/traces¶

Query Tempo for distributed traces using trace definitions from the configuration of observability data retrieval. This endpoint provides access to request traces across AIRefinery services, enabling you to inspect agent workflows, identify performance bottlenecks, and debug cross-service interactions.

Parameters:¶

Parameter	Type	Required	Description
`trace`	string	Yes	Trace name from the configuration of observability data retrieval. (e.g., 'inference_traces', 'distiller_traces')
`organization_id`	string	Yes	Organization ID to filter traces
`project_name`	string	No	Project name to filter traces
`trace_id`	string	No	Specific trace ID to retrieve
`time_window`	string	No	Time range for query (e.g., '5m', '1h', '24h')
`detail`	boolean	No	Whether to include detailed trace information. Default: true
`limit`	integer	No	Maximum number of traces to return. Default: 100

Example Usage¶

Organization-level inference traces:

curl -X POST "https://api-prod-k8s.airefinery.accenture.com/observability/traces" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
  -d '{
    "trace": "inference_traces",
    "organization_id": "org-123",
    "time_window": "1h"
  }'

Project-level distiller traces:

curl -X POST https://api-prod-k8s.airefinery.accenture.com/observability/traces \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
  -d '{
    "trace": "distiller_traces",
    "organization_id": "org-123",
    "project_name": "project-x",
    "time_window": "30m"
  }'

Get specific trace by ID:

curl -X POST https://api-prod-k8s.airefinery.accenture.com/observability/traces \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
  -d '{
    "trace": "inference_traces",
    "organization_id": "org-123",
    "trace_id": "abc123def456"
  }'

Search without detailed trace data:

curl -X POST https://api-prod-k8s.airefinery.accenture.com/observability/traces \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -H "sdk_version: <sdk-version>" \
  -d '{
    "trace": "inference_traces",
    "organization_id": "org-123",
    "detail": false,
    "limit": 50
  }'

Notes¶

The organization_id from authentication is required in request payload to restrict access to observability data within the organization.

Time windows support units: 'm' (minutes), 'h' (hours), 'd' (days)