diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bf4075c..0f326fa2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,40 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## Unreleased + +### Dashboard & Telemetry — dev console readiness + +The dashboard engine — Active Agent's local dev console — now works out of +the box (production observability is the hosted platform product): + +- **Engine load paths fixed**: `Engine.find_root` now points at the + dashboard directory, so `ActiveAgent::TelemetryTrace`, + `ProcessTelemetryTracesJob`, the API controller, views and engine routes + are auto-discovered in host apps (previously they required manual + `require`s). The engine is also required eagerly with Rails, since + engines defined lazily miss initializer collection. +- **Routes now match shipped controllers**: the engine exposes traces, + metrics and the ingest API (`/active_agent/api/traces`, matching + `Telemetry::Configuration::LOCAL_ENDPOINT_PATH`); routes to + never-shipped controllers (agents, sandboxes, templates, recordings, + api/v1) were removed. Engine root renders the traces index. +- **`local_storage` telemetry mode fixed**: tracer payloads are + symbol-keyed and were silently dropped by the string-keyed ingestion + normalizer; the reporter now stringifies and honors + `ActiveAgent::Dashboard.trace_model` overrides. +- **Token totals no longer double-count**: instrumentation mirrors LLM + token usage onto the root span; `TelemetryTrace.create_from_payload` + now counts child spans as the source of truth. +- **Span waterfall renders real offsets** (was pinned to 0ms), turbo-rails + is now optional (previously 500s without it), layout route helpers fixed, + `Agent.for_owner` scope added, synchronous ingest capped at 100 + traces/request. +- **New docs** (`docs/framework/dashboard.md`, README section) covering + install, authentication (none by default — see docs), remote ingestion + and multi-tenant mode; dashboard engine test suite added + (`test/dashboard/`). + ## [1.0.0] - 2025-11-21 Major refactor with breaking changes. Complete provider rewrite. New modular architecture. diff --git a/README.md b/README.md index 78bafbc9..1af2f782 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,32 @@ development: service: "RubyLLM" ``` +## Dev Console & Observability + +Active Agent includes a local dev console — traces with span waterfalls, +token usage, and per-agent metrics — as a mountable Rails engine, so you +can watch your agents while you build: + +```bash +rails generate active_agent:dashboard:install +rails db:migrate +``` + +```yaml +# config/active_agent.yml +telemetry: + enabled: true + local_storage: true +``` + +Open `/active_agent` and every generation appears as a trace. See +[docs/framework/dashboard.md](docs/framework/dashboard.md) for +authentication, remote ingestion, and multi-tenant mode. For production +monitoring — evaluations, cost estimates, retention, team workspaces — +point telemetry at the hosted platform at +[activeagents.ai](https://activeagents.ai); every workspace starts with a +free low-volume trial. + ## Features - **Agent-Oriented Programming**: Build AI applications using familiar Rails patterns diff --git a/docs/framework/dashboard.md b/docs/framework/dashboard.md new file mode 100644 index 00000000..95aaff06 --- /dev/null +++ b/docs/framework/dashboard.md @@ -0,0 +1,144 @@ +# Dev Console (Dashboard Engine) + +Active Agent ships a local development console as a Rails engine: every +agent generation recorded as a trace with a span waterfall, plus a metrics +overview — running inside your app against your own database while you +build. It uses the same telemetry pipeline as the hosted +[activeagents.ai](https://activeagents.ai) platform, which is the +production observability product: what you see locally in development is +what the platform shows (plus evaluations, cost estimates, retention, and +team workspaces) once you point telemetry at it. Every platform workspace +starts with a free low-volume trial. + +![Dashboard: traces list with expandable span timelines] + +## Quick start + +```bash +rails generate active_agent:dashboard:install +rails db:migrate +``` + +The generator: + +- copies the `active_agent_telemetry_traces` migration (plus agent, + run, template, sandbox and recording tables for the full install), +- mounts the engine at `/active_agent`, +- writes `config/initializers/active_agent_dashboard.rb`. + +Then enable telemetry with local storage in `config/active_agent.yml`: + +```yaml +telemetry: + enabled: true + local_storage: true +``` + +That's it. Run any agent and open `/active_agent` — each generation +appears as a trace with prompt/LLM/tool spans, timing, token usage +(input / output / thinking), provider and model. + +## What you get + +| Page | Path | Contents | +|------|------|----------| +| Traces | `/active_agent/traces` | Every generation: agent + action, status, duration, tokens; expandable span timeline; All/Errors filter; 30s auto-refresh | +| Trace detail | `/active_agent/traces/:id` | Span waterfall with relative offsets, token breakdown, error details, raw payload | +| Metrics | `/active_agent/traces/metrics` | Last-24h totals: traces, tokens, avg duration, error rate, active agents; per-agent statistics | +| Ingest API | `POST /active_agent/api/traces` | JSON trace ingestion (used by `local_storage` mode and remote SDKs) | + +Time-series charts on the metrics page use the optional +[groupdate](https://github.com/ankane/groupdate) gem when present and +degrade gracefully without it. + +## Authentication + +**The dashboard has no authentication by default.** Anyone who can reach +the route can read your traces. Before deploying anywhere non-local, set +an authentication method in the initializer: + +```ruby +ActiveAgent::Dashboard.configure do |config| + # Any proc that authenticates the request — Devise, Rails 8 sessions, basic auth… + config.authentication_method = ->(controller) do + controller.authenticate_admin! + end +end +``` + +Or constrain the mount in `config/routes.rb`: + +```ruby +authenticate :user, ->(u) { u.admin? } do + mount ActiveAgent::Dashboard::Engine => "/active_agent" +end +``` + +The local ingest endpoint is unauthenticated in local mode by design (it +receives traces from your own app process). In multi-tenant mode it +requires a Bearer token (see below). + +## Sending traces to a remote endpoint instead + +Point telemetry at any compatible receiver — including the hosted +platform — instead of (or in addition to) local storage: + +```yaml +telemetry: + enabled: true + endpoint: https://api.activeagents.ai/v1/traces + api_key: <%= ENV["ACTIVEAGENTS_API_KEY"] %> +``` + +The wire format is documented in [telemetry.md](./telemetry.md) under +"self-hosting endpoint requirements" — anything that speaks it can feed +or receive these traces. + +## Multi-tenant mode (running your own platform) + +The engine also supports account-scoped deployments — this is exactly how +the hosted platform runs it: + +```ruby +ActiveAgent::Dashboard.configure do |config| + config.multi_tenant = true + config.account_class = "Account" # must have a telemetry_api_key column + config.trace_model_class = "TelemetryTrace" # optional model override +end +``` + +In multi-tenant mode the ingest API authenticates with +`Authorization: Bearer ` and processes traces +asynchronously through `ActiveAgent::ProcessTelemetryTracesJob` +(idempotent per trace_id, capped at 100 traces per request). Add an +`increment_telemetry_usage!` method to your account model to hook usage +tracking or rate limiting. + +## Relationship to the hosted platform + +| | Dev console (this engine) | activeagents.ai (production) | +|---|---|---| +| Intended use | Development | Production monitoring | +| Traces + span waterfall | ✓ | ✓ | +| Metrics + per-agent stats | ✓ | ✓ | +| Trace ingest API | ✓ (single tenant, local) | ✓ (multi-tenant, quotas) | +| Conversation persistence | via [solid_agent](https://github.com/activeagents/solid_agent) | ✓ built in | +| Evaluations, cost estimates, retention, team accounts | — | ✓ | + +One pipeline, two contexts: the console shows your traces while you +develop; the platform runs the same engine multi-tenant with managed +infrastructure for production. Free workspaces get a low-volume trial +tier; paid plans lift the quotas. + +## Conversation persistence + +Pair the dashboard with the `solid_agent` gem to persist full +conversations (contexts, messages, generations) alongside traces; its +generation records carry the same `trace_id` for correlation: + +```ruby +class ApplicationAgent < ActiveAgent::Base + include SolidAgent::HasContext + has_context contextual: :user +end +``` diff --git a/lib/active_agent.rb b/lib/active_agent.rb index adc56b5d..7aeb9a31 100644 --- a/lib/active_agent.rb +++ b/lib/active_agent.rb @@ -16,6 +16,10 @@ # Module Level Extensions require "active_agent/configuration" require "active_agent/railtie" if defined?(Rails) +# The dashboard is a Rails engine; engines must be defined when the gem is +# required (before the host app collects railtie initializers), so it +# cannot be left to the Dashboard autoload above. +require "active_agent/dashboard" if defined?(Rails) # ActiveAgent is a framework for building AI agents with Rails-like conventions. # diff --git a/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/api/traces_controller.rb b/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/api/traces_controller.rb index f6889140..d58325ad 100644 --- a/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/api/traces_controller.rb +++ b/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/api/traces_controller.rb @@ -34,9 +34,13 @@ module Api class TracesController < ActionController::API before_action :authenticate_api_key!, if: -> { ActiveAgent::Dashboard.multi_tenant? } + # Maximum traces accepted per request (mirrors + # ProcessTelemetryTracesJob::MAX_TRACES_PER_JOB). + MAX_TRACES_PER_REQUEST = 100 + # POST /active_agent/api/traces def create - traces = params[:traces] || [] + traces = Array(params[:traces]).take(MAX_TRACES_PER_REQUEST) sdk_info = params[:sdk] || {} return head :accepted if traces.empty? diff --git a/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/application_controller.rb b/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/application_controller.rb index 9c7c19be..89b34be4 100644 --- a/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/application_controller.rb +++ b/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/application_controller.rb @@ -21,7 +21,17 @@ class ApplicationController < ActionController::Base private def authenticate_dashboard! - return if ActiveAgent::Dashboard.authentication_method.nil? + if ActiveAgent::Dashboard.authentication_method.nil? + # Traces contain prompts, outputs, and error backtraces. Refuse to + # serve them unauthenticated in production rather than exposing + # them to the internet by default. + if Rails.env.production? + render plain: "ActiveAgent Dashboard: set ActiveAgent::Dashboard.authentication_method " \ + "(see docs/framework/dashboard.md) to enable access in production.", + status: :forbidden + end + return + end result = ActiveAgent::Dashboard.authentication_method.call(self) head :unauthorized unless result diff --git a/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/dashboard_controller.rb b/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/dashboard_controller.rb index 1e794107..9286b9bd 100644 --- a/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/dashboard_controller.rb +++ b/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/dashboard_controller.rb @@ -6,23 +6,26 @@ module Dashboard # class DashboardController < ApplicationController def index + unless ActiveAgent::Dashboard.use_inertia && defined?(InertiaRails) + # No ERB overview ships yet — the traces list is the dashboard. + # Loading agent/run data here would also require the full + # dashboard tables, which telemetry-only installs don't have. + return redirect_to(traces_path) + end + @agents = fetch_agents.limit(10) @recent_runs = fetch_recent_runs.limit(10) @recent_traces = fetch_recent_traces.limit(10) @metrics = calculate_metrics - if ActiveAgent::Dashboard.use_inertia && defined?(InertiaRails) - render inertia: "Dashboard", props: { - agents: serialize_agents(@agents), - recentRuns: serialize_runs(@recent_runs), - recentTraces: serialize_traces(@recent_traces), - metrics: @metrics, - user: current_user_props, - account: current_account_props - } - else - render :index - end + render inertia: "Dashboard", props: { + agents: serialize_agents(@agents), + recentRuns: serialize_runs(@recent_runs), + recentTraces: serialize_traces(@recent_traces), + metrics: @metrics, + user: current_user_props, + account: current_account_props + } end private diff --git a/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/traces_controller.rb b/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/traces_controller.rb index f72cc9be..26d0b19a 100644 --- a/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/traces_controller.rb +++ b/lib/active_agent/dashboard/app/controllers/active_agent/dashboard/traces_controller.rb @@ -16,12 +16,14 @@ def index respond_to do |format| format.html - format.turbo_stream + # turbo-rails is optional; responding to an unregistered MIME + # type raises in host apps without it. + format.turbo_stream if turbo_stream_available? end end def show - @trace = ActiveAgent::TelemetryTrace.find(params[:id]) + @trace = ActiveAgent::Dashboard.trace_model.find(params[:id]) end def metrics @@ -31,12 +33,16 @@ def metrics respond_to do |format| format.html - format.turbo_stream + format.turbo_stream if turbo_stream_available? end end private + def turbo_stream_available? + Mime::Type.lookup_by_extension(:turbo_stream).present? + end + def fetch_traces traces = ActiveAgent::TelemetryTrace.recent diff --git a/lib/active_agent/dashboard/app/jobs/active_agent/process_telemetry_traces_job.rb b/lib/active_agent/dashboard/app/jobs/active_agent/process_telemetry_traces_job.rb index 0456e8dc..9c202f94 100644 --- a/lib/active_agent/dashboard/app/jobs/active_agent/process_telemetry_traces_job.rb +++ b/lib/active_agent/dashboard/app/jobs/active_agent/process_telemetry_traces_job.rb @@ -39,7 +39,16 @@ def perform(account_id: nil, traces:, sdk_info:, received_at:) return end + # Process in bounded slices; anything beyond the cap is re-enqueued + # rather than silently dropped (the client already received 202). + remainder = traces.drop(MAX_TRACES_PER_JOB) traces = traces.take(MAX_TRACES_PER_JOB) + if remainder.any? + self.class.perform_later( + account_id: account_id, traces: remainder, + sdk_info: sdk_info, received_at: received_at + ) + end traces.each do |trace| process_trace(trace, sdk_info, account) diff --git a/lib/active_agent/dashboard/app/models/active_agent/telemetry_trace.rb b/lib/active_agent/dashboard/app/models/active_agent/telemetry_trace.rb index b178e724..5fcd9438 100644 --- a/lib/active_agent/dashboard/app/models/active_agent/telemetry_trace.rb +++ b/lib/active_agent/dashboard/app/models/active_agent/telemetry_trace.rb @@ -54,13 +54,20 @@ def self.create_from_payload(trace, sdk_info = {}, account: nil) spans = trace["spans"] || [] root_span = spans.find { |s| s["parent_span_id"].nil? } || spans.first || {} - # Calculate totals from all spans total_duration = root_span["duration_ms"] + + # Instrumentation mirrors LLM token usage onto the root span for + # display, so summing every span double-counts. When child spans + # carry token data, they are the source of truth; the root span only + # counts for single-span traces. + counted_spans = spans.reject { |s| s["parent_span_id"].nil? } + counted_spans = spans if counted_spans.none? { |s| span_token_sum(s).positive? } + total_input = 0 total_output = 0 total_thinking = 0 - spans.each do |span| + counted_spans.each do |span| tokens = span["tokens"] || {} total_input += (tokens["input"] || 0) total_output += (tokens["output"] || 0) @@ -100,6 +107,15 @@ def self.create_from_payload(trace, sdk_info = {}, account: nil) create!(attrs) end + # Sums a span's token counts (used to decide which spans carry the + # authoritative token data during ingestion). + # + # @api private + def self.span_token_sum(span) + tokens = span["tokens"] || {} + tokens.fetch("input", 0).to_i + tokens.fetch("output", 0).to_i + tokens.fetch("thinking", 0).to_i + end + # Returns the root span of this trace. # # @return [Hash, nil] The root span or nil diff --git a/lib/active_agent/dashboard/app/views/active_agent/dashboard/traces/_trace_detail.html.erb b/lib/active_agent/dashboard/app/views/active_agent/dashboard/traces/_trace_detail.html.erb index 68787a2b..bb84a211 100644 --- a/lib/active_agent/dashboard/app/views/active_agent/dashboard/traces/_trace_detail.html.erb +++ b/lib/active_agent/dashboard/app/views/active_agent/dashboard/traces/_trace_detail.html.erb @@ -59,11 +59,23 @@

Span Timeline

- <% total_duration = trace.total_duration_ms || 1 %> + <% total_duration = (trace.total_duration_ms || 1).to_f %> + <% total_duration = 1.0 if total_duration.zero? %> + <% + # Offsets relative to the root span's start. Span payloads carry + # absolute ISO8601 start_time values, not relative offsets. + root = trace.root_span || trace.spans.first || {} + root_start = Time.parse(root["start_time"]) rescue nil + %> <% trace.spans.each_with_index do |span, index| %> <% - start_pct = ((span["start_time_relative_ms"] || 0) / total_duration * 100).clamp(0, 100) - width_pct = ((span["duration_ms"] || 0) / total_duration * 100).clamp(0.5, 100 - start_pct) + relative_ms = span["start_time_relative_ms"] + if relative_ms.nil? && root_start && span["start_time"].present? + span_start = Time.parse(span["start_time"]) rescue nil + relative_ms = ((span_start - root_start) * 1000).round(2) if span_start + end + start_pct = ((relative_ms || 0).to_f / total_duration * 100).clamp(0, 100) + width_pct = ((span["duration_ms"] || 0).to_f / total_duration * 100).clamp(0.5, [ 100 - start_pct, 0.5 ].max) span_type = span["type"] || "unknown" colors = { "root" => "bg-gray-400", diff --git a/lib/active_agent/dashboard/app/views/layouts/active_agent/dashboard/application.html.erb b/lib/active_agent/dashboard/app/views/layouts/active_agent/dashboard/application.html.erb index 5c15d9e1..1388fddc 100644 --- a/lib/active_agent/dashboard/app/views/layouts/active_agent/dashboard/application.html.erb +++ b/lib/active_agent/dashboard/app/views/layouts/active_agent/dashboard/application.html.erb @@ -69,11 +69,11 @@
<%= link_to "Traces", - active_agent_dashboard.traces_path, - class: "px-3 py-2 rounded-md text-sm font-medium #{request.path == active_agent_dashboard.traces_path ? 'bg-indigo-100 text-indigo-700 dark:bg-indigo-900 dark:text-indigo-200' : 'text-gray-600 hover:bg-gray-100 dark:text-gray-300 dark:hover:bg-gray-700'}" %> + traces_path, + class: "px-3 py-2 rounded-md text-sm font-medium #{request.path == traces_path ? 'bg-indigo-100 text-indigo-700 dark:bg-indigo-900 dark:text-indigo-200' : 'text-gray-600 hover:bg-gray-100 dark:text-gray-300 dark:hover:bg-gray-700'}" %> <%= link_to "Metrics", - active_agent_dashboard.metrics_traces_path, - class: "px-3 py-2 rounded-md text-sm font-medium #{request.path == active_agent_dashboard.metrics_traces_path ? 'bg-indigo-100 text-indigo-700 dark:bg-indigo-900 dark:text-indigo-200' : 'text-gray-600 hover:bg-gray-100 dark:text-gray-300 dark:hover:bg-gray-700'}" %> + metrics_traces_path, + class: "px-3 py-2 rounded-md text-sm font-medium #{request.path == metrics_traces_path ? 'bg-indigo-100 text-indigo-700 dark:bg-indigo-900 dark:text-indigo-200' : 'text-gray-600 hover:bg-gray-100 dark:text-gray-300 dark:hover:bg-gray-700'}" %>
diff --git a/lib/active_agent/dashboard/config/routes.rb b/lib/active_agent/dashboard/config/routes.rb index 0820ff69..0f894df6 100644 --- a/lib/active_agent/dashboard/config/routes.rb +++ b/lib/active_agent/dashboard/config/routes.rb @@ -1,78 +1,20 @@ # frozen_string_literal: true ActiveAgent::Dashboard::Engine.routes.draw do - # Dashboard root - root to: "dashboard#index" - - # Dashboard + # The functional dashboard surface: traces + metrics. + root to: "traces#index" get "dashboard", to: "dashboard#index" - # Traces resources :traces, only: [ :index, :show ] do collection do get :metrics end end - # Agents - resources :agents do - member do - post :execute - post :test - get :versions - post :restore_version - end - resources :runs, controller: "agent_runs", only: [ :index, :show ] do - member do - post :cancel - end - end - end - - # Agent Templates - resources :templates, only: [ :index, :show ] do - member do - post :create_agent - end - end - - # Sandbox Sessions - resources :sandboxes, controller: "sandbox_sessions" do - member do - post :provision - post :execute - post :expire - end - resources :runs, controller: "sandbox_runs", only: [ :index, :show ] - end - - # Session Recordings - resources :recordings, controller: "session_recordings", only: [ :index, :show ] do - member do - get :timeline - get :playback - end - end - - # API namespace + # Telemetry ingestion — matches + # ActiveAgent::Telemetry::Configuration::LOCAL_ENDPOINT_PATH + # (/active_agent/api/traces when mounted at the default /active_agent). namespace :api do - namespace :v1 do - # Telemetry ingestion - resources :traces, only: [ :create ] - - # Agent execution API - resources :agents, only: [ :index, :show ] do - member do - post :execute - end - end - - # Sandbox API - resources :sandboxes, only: [ :create, :show ] do - member do - post :execute - end - end - end + resources :traces, only: [ :create ] end end diff --git a/lib/active_agent/dashboard/engine.rb b/lib/active_agent/dashboard/engine.rb index 7fac7feb..c313e846 100644 --- a/lib/active_agent/dashboard/engine.rb +++ b/lib/active_agent/dashboard/engine.rb @@ -4,36 +4,40 @@ module ActiveAgent module Dashboard # Rails Engine for the ActiveAgent Dashboard. # - # Provides a full-featured dashboard for managing agents, viewing telemetry, - # running sandboxes, and recording sessions. + # Provides the free self-hosted dashboard: telemetry traces, metrics, + # and the local trace ingestion API. # # Mount in your routes: # mount ActiveAgent::Dashboard::Engine => "/active_agent" # class Engine < ::Rails::Engine - isolate_namespace ActiveAgent::Dashboard - - # Use a unique engine name to avoid conflicts - engine_name "active_agent" + # The engine lives at lib/active_agent/dashboard rather than the gem + # root. Rails derives an engine's load paths (app/models, + # app/controllers, app/jobs, app/views, config/routes.rb) from + # find_root when +config+ is first materialized — which + # isolate_namespace below triggers via +routes+ — so this override + # must be defined before anything touches config. Overriding only + # +root+ leaves the engine's classes off host applications' autoload + # paths entirely. + def self.find_root(_from) + root + end - # Override engine root to point to the dashboard directory - # This must be set before paths are computed def self.root @root ||= Pathname.new(File.expand_path("..", __FILE__)) end + isolate_namespace ActiveAgent::Dashboard + + # Use a unique engine name to avoid conflicts + engine_name "active_agent" + # Alias for consistency with existing code def self.dashboard_root root end config.active_agent_dashboard = ActiveSupport::OrderedOptions.new - - initializer "active_agent.dashboard.append_view_paths" do - ActiveSupport.on_load(:action_controller) do - prepend_view_path ActiveAgent::Dashboard::Engine.root.join("app", "views").to_s - end - end end end end diff --git a/lib/active_agent/railtie.rb b/lib/active_agent/railtie.rb index baba8f9e..e8ae4a2c 100644 --- a/lib/active_agent/railtie.rb +++ b/lib/active_agent/railtie.rb @@ -67,6 +67,11 @@ class Railtie < Rails::Railtie # :nodoc: include ActiveAgent::Telemetry::Instrumentation instrument_telemetry! end + + # Flush remaining traces when the process exits — without this, + # short-lived processes (rails runner, jobs, deploys rolling a + # server) drop whatever was buffered since the last interval flush. + at_exit { ActiveAgent::Telemetry.shutdown } end # endregion telemetry_configuration diff --git a/lib/active_agent/telemetry/configuration.rb b/lib/active_agent/telemetry/configuration.rb index 89e7b032..6ba61474 100644 --- a/lib/active_agent/telemetry/configuration.rb +++ b/lib/active_agent/telemetry/configuration.rb @@ -47,9 +47,11 @@ class Configuration attr_accessor :timeout # @return [Boolean] Whether to capture request/response bodies (default: false) + # @note Reserved: not yet consumed by the tracer/instrumentation. attr_accessor :capture_bodies # @return [Array] Attributes to redact from traces + # @note Reserved: not yet consumed by the tracer/instrumentation. attr_accessor :redact_attributes # @return [String] Service name for trace attribution diff --git a/lib/active_agent/telemetry/instrumentation.rb b/lib/active_agent/telemetry/instrumentation.rb index f9ddb53a..c1643102 100644 --- a/lib/active_agent/telemetry/instrumentation.rb +++ b/lib/active_agent/telemetry/instrumentation.rb @@ -46,7 +46,18 @@ module GenerationInstrumentation def process_prompt return super unless Telemetry.enabled? - Telemetry.trace("#{self.class.name}.#{action_name}", span_type: :root) do |span| + # Reuse (or mint) the generation's trace id so the telemetry trace + # shares one id with everything else that reads + # prompt_options[:trace_id] — the generation request parameters and + # e.g. solid_agent's persisted generation records. Without this the + # tracer generates its own id and traces can't be correlated with + # the records they describe. + trace_id = nil + if respond_to?(:prompt_options) && prompt_options.is_a?(Hash) + trace_id = (prompt_options[:trace_id] ||= SecureRandom.uuid) + end + + Telemetry.trace("#{self.class.name}.#{action_name}", span_type: :root, **{ trace_id: trace_id }.compact) do |span| span.set_attribute("agent.class", self.class.name) span.set_attribute("agent.action", action_name.to_s) span.set_attribute("agent.provider", provider_name) if respond_to?(:provider_name) diff --git a/lib/active_agent/telemetry/reporter.rb b/lib/active_agent/telemetry/reporter.rb index 11679f84..5ee31bae 100644 --- a/lib/active_agent/telemetry/reporter.rb +++ b/lib/active_agent/telemetry/reporter.rb @@ -27,6 +27,7 @@ def initialize(configuration) @mutex = Mutex.new @running = false @thread = nil + @send_threads = [] @shutdown = false start_flush_thread if configuration.enabled? @@ -49,22 +50,27 @@ def report(trace) end end - # Flushes all buffered traces immediately. + # Flushes all buffered traces immediately and waits for the send to + # complete, so callers (tests, rails runner, job shutdown) can rely on + # the traces having been delivered/stored when this returns. # # @return [void] def flush - @mutex.synchronize do - flush_buffer - end + thread = @mutex.synchronize { flush_buffer } + thread&.join(configuration.timeout) end - # Shuts down the reporter, flushing remaining traces. + # Shuts down the reporter, flushing remaining traces and waiting for + # any in-flight sends — without this, traces flushed near process exit + # were silently dropped. # # @return [void] def shutdown @shutdown = true + @running = false flush @thread&.join(5) # Wait up to 5 seconds for thread to finish + @mutex.synchronize { @send_threads.dup }.each { |t| t.join(configuration.timeout) } end private @@ -85,16 +91,20 @@ def start_flush_thread end end - # Flushes the buffer by sending traces to the endpoint. + # Flushes the buffer by sending traces to the endpoint on a background + # thread. Must be called within @mutex synchronization. # - # Must be called within @mutex synchronization. + # @return [Thread, nil] the send thread, so callers can join it def flush_buffer return if @buffer.empty? traces = @buffer.dup @buffer.clear - Thread.new { send_traces(traces) } + @send_threads.select!(&:alive?) + thread = Thread.new { send_traces(traces) } + @send_threads << thread + thread end # Sends traces to the configured endpoint. @@ -147,17 +157,28 @@ def send_traces(traces) # @param traces [Array] Traces to store def store_traces_locally(traces) sdk_info = { - name: "activeagent", - version: ActiveAgent::VERSION, - language: "ruby", - runtime_version: RUBY_VERSION + "name" => "activeagent", + "version" => ActiveAgent::VERSION, + "language" => "ruby", + "runtime_version" => RUBY_VERSION } + model = local_trace_model + unless model + log_error("local_storage is enabled but no trace model is available — " \ + "run `rails generate active_agent:dashboard:install` first") + return + end + traces.each do |trace| + # Tracer payloads are symbol-keyed; create_from_payload reads + # string keys (as it does for JSON ingested over HTTP). + trace = trace.deep_stringify_keys if trace.respond_to?(:deep_stringify_keys) + # Skip if trace already exists (idempotency) - next if ActiveAgent::TelemetryTrace.exists?(trace_id: trace["trace_id"]) + next if model.exists?(trace_id: trace["trace_id"]) - ActiveAgent::TelemetryTrace.create_from_payload(trace, sdk_info) + model.create_from_payload(trace, sdk_info) rescue StandardError => e log_error("Failed to store trace locally: #{e.class} - #{e.message}") end @@ -165,6 +186,18 @@ def store_traces_locally(traces) log_error("Error storing traces locally: #{e.class} - #{e.message}") end + # Resolves the configured trace model (honors + # ActiveAgent::Dashboard.trace_model_class overrides). + def local_trace_model + if defined?(ActiveAgent::Dashboard) && ActiveAgent::Dashboard.respond_to?(:trace_model) + ActiveAgent::Dashboard.trace_model + elsif defined?(ActiveAgent::TelemetryTrace) + ActiveAgent::TelemetryTrace + end + rescue NameError + nil + end + # Logs an error message. # # @param message [String] Error message diff --git a/lib/active_agent/telemetry/tracer.rb b/lib/active_agent/telemetry/tracer.rb index a9665ca1..0edfbad1 100644 --- a/lib/active_agent/telemetry/tracer.rb +++ b/lib/active_agent/telemetry/tracer.rb @@ -47,7 +47,10 @@ def initialize(configuration) def trace(name, **attributes, &block) return yield(Telemetry::NullSpan.new) unless should_trace? - trace_id = generate_trace_id + # Callers may supply the trace id (instrumentation passes the + # generation's prompt_options[:trace_id]) so external records can + # correlate with this trace; otherwise generate one. + trace_id = attributes.delete(:trace_id) || generate_trace_id root_span = Span.new( name, trace_id: trace_id, diff --git a/lib/generators/active_agent/dashboard/install/install_generator.rb b/lib/generators/active_agent/dashboard/install/install_generator.rb index ede09e99..8b3ea015 100644 --- a/lib/generators/active_agent/dashboard/install/install_generator.rb +++ b/lib/generators/active_agent/dashboard/install/install_generator.rb @@ -75,7 +75,6 @@ def show_post_install say "" say "Next steps:" say " 1. Run migrations: rails db:migrate" - say " 2. Seed templates: rails active_agent:dashboard:seed" say " 3. Configure authentication in config/initializers/active_agent_dashboard.rb" say " 4. Visit /active_agent to access the dashboard" say "" diff --git a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_runs.rb b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_runs.rb index ae754c27..523a98ef 100644 --- a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_runs.rb +++ b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_runs.rb @@ -7,11 +7,11 @@ def change # Input t.text :input_prompt - t.jsonb :input_params, default: {} + t.json :input_params, default: {} # Output t.text :output - t.jsonb :output_metadata, default: {} + t.json :output_metadata, default: {} # Execution details t.integer :status, default: 0, null: false @@ -30,7 +30,7 @@ def change # Trace for debugging t.string :trace_id - t.jsonb :logs, default: [] + t.json :logs, default: [] t.timestamps end diff --git a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_templates.rb b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_templates.rb index 71b5e121..a969e924 100644 --- a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_templates.rb +++ b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_templates.rb @@ -13,11 +13,11 @@ def change t.string :model, default: "gpt-4o-mini" t.text :instructions t.string :preset_type - t.jsonb :appearance, default: {} - t.jsonb :instruction_sets, default: [] - t.jsonb :tools, default: [] - t.jsonb :mcp_servers, default: {} - t.jsonb :model_config, default: {} + t.json :appearance, default: {} + t.json :instruction_sets, default: [] + t.json :tools, default: [] + t.json :mcp_servers, default: {} + t.json :model_config, default: {} # Metadata t.string :icon diff --git a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_versions.rb b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_versions.rb index 6f1dcb63..a89ecccf 100644 --- a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_versions.rb +++ b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agent_versions.rb @@ -9,7 +9,7 @@ def change t.string :change_summary # Snapshot of agent configuration at this version - t.jsonb :configuration_snapshot, null: false, default: {} + t.json :configuration_snapshot, null: false, default: {} # Who made the change t.string :created_by diff --git a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agents.rb b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agents.rb index 83c79055..37e5e32f 100644 --- a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agents.rb +++ b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_agents.rb @@ -17,18 +17,18 @@ def change # Avatar/appearance configuration t.string :preset_type - t.jsonb :appearance, default: {} + t.json :appearance, default: {} # Capabilities - t.jsonb :instruction_sets, default: [] - t.jsonb :tools, default: [] - t.jsonb :mcp_servers, default: [] + t.json :instruction_sets, default: [] + t.json :tools, default: [] + t.json :mcp_servers, default: [] # Model configuration - t.jsonb :model_config, default: {} + t.json :model_config, default: {} # Response format - t.jsonb :response_format, default: {} + t.json :response_format, default: {} # Status t.integer :status, default: 0, null: false diff --git a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_sandbox_runs.rb b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_sandbox_runs.rb index f03ea9f1..69ed12ba 100644 --- a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_sandbox_runs.rb +++ b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_sandbox_runs.rb @@ -17,7 +17,7 @@ def change t.datetime :completed_at # Screenshots - t.jsonb :screenshots, default: [] + t.json :screenshots, default: [] t.timestamps end diff --git a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_sandbox_sessions.rb b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_sandbox_sessions.rb index 23888727..fe148db1 100644 --- a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_sandbox_sessions.rb +++ b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_sandbox_sessions.rb @@ -28,7 +28,7 @@ def change t.integer :total_duration_ms, default: 0 # Results - t.jsonb :runs, default: [] + t.json :runs, default: [] t.text :error_message t.timestamps diff --git a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_session_recordings.rb b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_session_recordings.rb index 170cfd3e..747e7a79 100644 --- a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_session_recordings.rb +++ b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_session_recordings.rb @@ -9,7 +9,7 @@ def change t.integer :status, default: 0, null: false t.integer :duration_ms t.integer :action_count, default: 0 - t.jsonb :metadata, default: {} + t.json :metadata, default: {} t.timestamps end @@ -22,7 +22,7 @@ def change t.text :value t.string :screenshot_key t.string :dom_snapshot_key - t.jsonb :metadata, default: {} + t.json :metadata, default: {} t.timestamps end diff --git a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_telemetry_traces.rb b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_telemetry_traces.rb index a06d96e3..c974efe3 100644 --- a/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_telemetry_traces.rb +++ b/lib/generators/active_agent/dashboard/install/templates/migrations/create_active_agent_telemetry_traces.rb @@ -16,13 +16,13 @@ def change t.datetime :timestamp, null: false # Span data (JSON array of spans) - t.jsonb :spans, default: [] + t.json :spans, default: [] # Resource attributes - t.jsonb :resource_attributes, default: {} + t.json :resource_attributes, default: {} # SDK info - t.jsonb :sdk_info, default: {} + t.json :sdk_info, default: {} # Aggregated metrics (for quick queries) t.integer :total_duration_ms diff --git a/lib/generators/active_agent/dashboard/templates/create_active_agent_telemetry_traces.rb.erb b/lib/generators/active_agent/dashboard/templates/create_active_agent_telemetry_traces.rb.erb index 803e4df7..a706db23 100644 --- a/lib/generators/active_agent/dashboard/templates/create_active_agent_telemetry_traces.rb.erb +++ b/lib/generators/active_agent/dashboard/templates/create_active_agent_telemetry_traces.rb.erb @@ -7,9 +7,9 @@ class CreateActiveAgentTelemetryTraces < ActiveRecord::Migration<%= migration_ve t.string :service_name t.string :environment t.datetime :timestamp, index: true - t.jsonb :spans, default: [] - t.jsonb :resource_attributes, default: {} - t.jsonb :sdk_info, default: {} + t.json :spans, default: [] + t.json :resource_attributes, default: {} + t.json :sdk_info, default: {} t.decimal :total_duration_ms, precision: 12, scale: 3 t.integer :total_input_tokens, default: 0 t.integer :total_output_tokens, default: 0 diff --git a/test/dashboard/engine_integration_test.rb b/test/dashboard/engine_integration_test.rb new file mode 100644 index 00000000..5c720b93 --- /dev/null +++ b/test/dashboard/engine_integration_test.rb @@ -0,0 +1,131 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "telemetry_trace_test" + +class DashboardEngineIntegrationTest < ActionDispatch::IntegrationTest + def setup + ActiveAgent::TelemetryTrace.delete_all + end + + test "engine paths resolve to the dashboard directory" do + engine = ActiveAgent::Dashboard::Engine.instance + root = ActiveAgent::Dashboard::Engine.root + + assert_equal root.to_s, engine.root.to_s + assert_includes engine.paths["app/models"].existent, root.join("app", "models").to_s + assert_includes engine.paths["config/routes.rb"].existent, root.join("config", "routes.rb").to_s + end + + test "dashboard classes are autoloadable without manual requires" do + assert_equal "active_agent_telemetry_traces", ActiveAgent::TelemetryTrace.table_name + assert ActiveAgent::ProcessTelemetryTracesJob < ActiveJob::Base + assert ActiveAgent::Dashboard::Api::TracesController < ActionController::API + end + + test "every engine route maps to a shipped controller action" do + ActiveAgent::Dashboard::Engine.routes.routes.each do |route| + controller = route.defaults[:controller] + action = route.defaults[:action] + next unless controller + + controller_class = "active_agent/#{controller.delete_prefix('active_agent/')}" + .camelize.concat("Controller").constantize + assert controller_class.action_methods.include?(action.to_s) || controller_class.instance_methods.include?(action.to_sym), + "route #{route.path.spec} points at missing #{controller_class}##{action}" + end + end + + test "traces index renders" do + ActiveAgent::TelemetryTrace.create_from_payload(sample_payload) + + get "/active_agent/traces" + + assert_response :success + assert_includes response.body, "SupportAgent" + end + + test "engine root renders the traces index" do + get "/active_agent/" + + assert_response :success + end + + test "dashboard overview redirects to traces in ERB mode" do + get "/active_agent/dashboard" + + assert_response :redirect + assert_includes response.location, "/active_agent/traces" + end + + test "dashboard refuses unauthenticated access in production when no auth is configured" do + Rails.env.stub(:production?, true) do + get "/active_agent/traces" + end + + assert_response :forbidden + assert_includes response.body, "authentication_method" + end + + test "local ingest endpoint matches LOCAL_ENDPOINT_PATH and persists traces" do + endpoint = ActiveAgent::Telemetry::Configuration::LOCAL_ENDPOINT_PATH + assert_equal "/active_agent/api/traces", endpoint + + payload = sample_payload + post endpoint, params: { traces: [ payload ], sdk: { name: "activeagent" } }, as: :json + + assert_response :accepted + trace = ActiveAgent::TelemetryTrace.find_by(trace_id: payload["trace_id"]) + assert trace + assert_equal "SupportAgent", trace.agent_class + assert_equal 100, trace.total_input_tokens + end + + test "reporter local storage persists symbol-keyed tracer payloads" do + config = ActiveAgent::Telemetry::Configuration.new + config.enabled = true + config.local_storage = true + reporter = ActiveAgent::Telemetry::Reporter.new(config) + + symbol_payload = { + trace_id: SecureRandom.hex(16), + service_name: "dummy", + environment: "test", + timestamp: Time.current.iso8601(6), + resource_attributes: {}, + spans: [ + { span_id: "r1", parent_span_id: nil, name: "SupportAgent.respond", type: "root", + duration_ms: 10.0, status: "OK", + attributes: { "agent.class" => "SupportAgent", "agent.action" => "respond" }, + tokens: { input: 9, output: 4, thinking: 0 } } + ] + } + + reporter.send(:store_traces_locally, [ symbol_payload ]) + + trace = ActiveAgent::TelemetryTrace.find_by(trace_id: symbol_payload[:trace_id]) + assert trace, "symbol-keyed payload was not persisted" + assert_equal 9, trace.total_input_tokens + ensure + reporter&.shutdown + end + + private + + def sample_payload + { + "trace_id" => SecureRandom.hex(16), + "service_name" => "dummy", + "environment" => "test", + "timestamp" => Time.current.iso8601(6), + "resource_attributes" => {}, + "spans" => [ + { "span_id" => "r1", "parent_span_id" => nil, "name" => "SupportAgent.respond", + "type" => "root", "start_time" => 1.second.ago.iso8601(6), + "end_time" => Time.current.iso8601(6), "duration_ms" => 1000.0, "status" => "OK", + "attributes" => { "agent.class" => "SupportAgent", "agent.action" => "respond" }, + "tokens" => { "input" => 100, "output" => 40, "thinking" => 0 } } + ] + } + end +end diff --git a/test/dashboard/telemetry_correlation_test.rb b/test/dashboard/telemetry_correlation_test.rb new file mode 100644 index 00000000..3d3623f8 --- /dev/null +++ b/test/dashboard/telemetry_correlation_test.rb @@ -0,0 +1,125 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "telemetry_trace_test" + +# Trace-id correlation and flush determinism: the pieces that let an app's +# own records (e.g. solid_agent generations reading +# prompt_options[:trace_id]) point at the telemetry trace that produced +# them, and that make Telemetry.flush reliable in short-lived processes. +class TelemetryCorrelationTest < ActiveSupport::TestCase + TelemetryTraceTest.ensure_table! + + def setup + ActiveAgent::TelemetryTrace.delete_all + + @configuration = ActiveAgent::Telemetry::Configuration.new + @configuration.enabled = true + @configuration.local_storage = true + @configuration.service_name = "dummy" + end + + test "tracer honors a caller-supplied trace_id" do + tracer = ActiveAgent::Telemetry::Tracer.new(@configuration) + + supplied = SecureRandom.uuid + tracer.trace("SupportAgent.respond", trace_id: supplied) do |span| + span.set_tokens(input: 10, output: 5) + end + tracer.flush + + trace = ActiveAgent::TelemetryTrace.find_by(trace_id: supplied) + assert trace.present?, "trace should be stored under the supplied trace_id" + end + + test "tracer generates a trace_id when none is supplied" do + tracer = ActiveAgent::Telemetry::Tracer.new(@configuration) + + tracer.trace("SupportAgent.respond") { |span| span.set_tokens(input: 1, output: 1) } + tracer.flush + + assert_equal 1, ActiveAgent::TelemetryTrace.count + assert ActiveAgent::TelemetryTrace.first.trace_id.present? + end + + test "flush waits for the send to complete" do + reporter = ActiveAgent::Telemetry::Reporter.new(@configuration) + + reporter.report(stored_payload) + reporter.flush + + # No sleep: flush must not return before the trace is persisted. + assert_equal 1, ActiveAgent::TelemetryTrace.count + ensure + reporter&.shutdown + end + + test "shutdown flushes buffered traces" do + reporter = ActiveAgent::Telemetry::Reporter.new(@configuration) + + reporter.report(stored_payload) + reporter.shutdown + + assert_equal 1, ActiveAgent::TelemetryTrace.count + end + + test "instrumented generations share the trace_id exposed in prompt_options" do + original_tracer = swap_global_tracer(ActiveAgent::Telemetry::Tracer.new(@configuration)) + + agent_class = Class.new(ApplicationAgent) do + def self.name = "CorrelationProbeAgent" + + def ping + prompt(message: "hello", instructions: "Reply briefly.") + end + end + agent_class.include(ActiveAgent::Telemetry::Instrumentation) + agent_class.instrument_telemetry! + + agent = agent_class.with({}).ping + agent.generate_now rescue nil # provider errors don't matter; the trace does + + ActiveAgent::Telemetry.flush + + trace = ActiveAgent::TelemetryTrace.order(:created_at).last + assert trace.present?, "instrumented generation should store a trace" + # Instrumentation mints the id via SecureRandom.uuid into + # prompt_options[:trace_id] (dashed UUID); the tracer's own fallback is + # 32-char hex. A dashed UUID here proves the id flowed from + # prompt_options — i.e. external records reading prompt_options can + # correlate with this trace. + assert_match(/\A\h{8}-\h{4}-\h{4}-\h{4}-\h{12}\z/, trace.trace_id, + "trace_id should be the prompt_options UUID, not a tracer-generated hex id") + ensure + swap_global_tracer(original_tracer) + end + + private + + def stored_payload + { + "trace_id" => SecureRandom.hex(16), + "service_name" => "dummy", + "environment" => "test", + "timestamp" => Time.current.iso8601(6), + "spans" => [ + { + "span_id" => "root1", "parent_span_id" => nil, "name" => "SupportAgent.respond", + "type" => "root", "duration_ms" => 10.0, "status" => "OK", + "attributes" => { "agent.class" => "SupportAgent", "agent.action" => "respond" }, + "tokens" => { "input" => 5, "output" => 2 } + } + ] + } + end + + def swap_global_tracer(tracer) + previous = ActiveAgent::Telemetry.instance_variable_get(:@tracer) + ActiveAgent::Telemetry.instance_variable_set(:@tracer, tracer) + previous_enabled = ActiveAgent::Telemetry.configuration.enabled + ActiveAgent::Telemetry.configuration.enabled = true + ActiveAgent::Telemetry.configuration.local_storage = true + @restore_enabled = previous_enabled + previous + end +end diff --git a/test/dashboard/telemetry_trace_test.rb b/test/dashboard/telemetry_trace_test.rb new file mode 100644 index 00000000..b6846ff3 --- /dev/null +++ b/test/dashboard/telemetry_trace_test.rb @@ -0,0 +1,112 @@ +# frozen_string_literal: true + +require "test_helper" + +class TelemetryTraceTest < ActiveSupport::TestCase + def setup + ActiveAgent::TelemetryTrace.delete_all + end + + # Must run OUTSIDE test transactions (transactional tests roll DDL back + # in SQLite), so callers invoke it at file-load time. + def self.ensure_table! + connection = ActiveRecord::Base.connection + unless connection.table_exists?(:active_agent_telemetry_traces) + connection.create_table :active_agent_telemetry_traces do |t| + t.string :trace_id, null: false + t.string :service_name + t.string :environment + t.datetime :timestamp, null: false + t.json :spans, default: [] + t.json :resource_attributes, default: {} + t.json :sdk_info, default: {} + t.integer :total_duration_ms + t.integer :total_input_tokens, default: 0 + t.integer :total_output_tokens, default: 0 + t.integer :total_thinking_tokens, default: 0 + t.string :status, default: "UNSET" + t.string :agent_class + t.string :agent_action + t.text :error_message + t.timestamps + end + end + end + + def payload(spans:) + { + "trace_id" => SecureRandom.hex(16), + "service_name" => "dummy", + "environment" => "test", + "timestamp" => Time.current.iso8601(6), + "resource_attributes" => {}, + "spans" => spans + } + end + + def root_span(tokens: {}, status: "OK", attributes: {}) + { + "span_id" => "root1", "parent_span_id" => nil, "name" => "SupportAgent.respond", + "type" => "root", "duration_ms" => 1200.0, "status" => status, + "attributes" => { "agent.class" => "SupportAgent", "agent.action" => "respond" }.merge(attributes), + "tokens" => tokens + } + end + + def llm_span(tokens:, status: "OK") + { + "span_id" => "llm1", "parent_span_id" => "root1", "name" => "llm.generate", + "type" => "llm", "duration_ms" => 1100.0, "status" => status, + "attributes" => { "llm.provider" => "mock", "llm.model" => "mock-model" }, + "tokens" => tokens + } + end + + test "does not double-count tokens mirrored onto the root span" do + tokens = { "input" => 500, "output" => 220, "thinking" => 10 } + trace = ActiveAgent::TelemetryTrace.create_from_payload( + payload(spans: [ root_span(tokens: tokens), llm_span(tokens: tokens) ]) + ) + + assert_equal 500, trace.total_input_tokens + assert_equal 220, trace.total_output_tokens + assert_equal 10, trace.total_thinking_tokens + end + + test "counts root span tokens for single-span traces" do + trace = ActiveAgent::TelemetryTrace.create_from_payload( + payload(spans: [ root_span(tokens: { "input" => 42, "output" => 7 }) ]) + ) + + assert_equal 42, trace.total_input_tokens + assert_equal 7, trace.total_output_tokens + end + + test "counts root span tokens when child spans carry none" do + trace = ActiveAgent::TelemetryTrace.create_from_payload( + payload(spans: [ root_span(tokens: { "input" => 30, "output" => 12 }), llm_span(tokens: {}) ]) + ) + + assert_equal 30, trace.total_input_tokens + assert_equal 12, trace.total_output_tokens + end + + test "extracts agent info, status and error message" do + error_payload = payload( + spans: [ + root_span(status: "ERROR", attributes: { "error.message" => "Rate limit exceeded" }), + llm_span(tokens: { "input" => 5 }, status: "ERROR") + ] + ) + trace = ActiveAgent::TelemetryTrace.create_from_payload(error_payload) + + assert_equal "SupportAgent", trace.agent_class + assert_equal "respond", trace.agent_action + assert trace.error? + assert_equal "Rate limit exceeded", trace.error_message + assert_equal "mock", trace.provider + assert_equal "mock-model", trace.model + end +end + +TelemetryTraceTest.ensure_table! diff --git a/test/dummy/config/routes.rb b/test/dummy/config/routes.rb index 2cba1443..f41192b4 100644 --- a/test/dummy/config/routes.rb +++ b/test/dummy/config/routes.rb @@ -1,6 +1,9 @@ Rails.application.routes.draw do # Define your application routes per the DSL in https://guides.rubyonrails.org/routing.html + # Free self-hosted dashboard (exercised by test/dashboard tests) + mount ActiveAgent::Dashboard::Engine => "/active_agent" + # Reveal health status on /up that returns 200 if the app boots with no exceptions, otherwise 500. # Can be used by load balancers and uptime monitors to verify that the app is live. get "up" => "rails/health#show", :as => :rails_health_check