Telemetry

Use telemetry to connect a slow request to the operations it called and to measure recurring behavior across requests. A span describes one operation; related spans form a trace. Metrics aggregate measurements over time. Context-aware logs can carry the same trace identity.

telemetry.Telemetry is the shared handle for these signals. Its driver decides where they go; choose and configure one on Drivers. Telemetry can be sampled, aggregated, or dropped. It is not an audit ledger or proof that a business action completed; use auditing with events when individual actions need a durable record.

Shared Handle

With your selected driver, give the handle to the application. Import application from arkoren.dev/core/application and telemetry from arkoren.dev/core/service/telemetry:

app := application.New(func(app *application.Builder) {
	app.Telemetry(telemetry.New(driver))
})

Construction performs no I/O. After composition succeeds, the application starts telemetry before user startup callbacks and stops it after cleanup and the final app stopped log. It supplies the service name, version, and environment as the telemetry resource. Without configured telemetry or diagnostic capture, app.Telemetry() remains a nonnil, unstarted no-op handle.

Pass that handle explicitly when constructing an instrumented service. With an existing database driver backend and database imported from arkoren.dev/core/service/database:

db := database.New(backend, func(connection *database.Builder) {
	connection.Telemetry(app.Telemetry())
})

The component catalog identifies each builder and its observations. An omitted or nil handle follows telemetry.Default() at operation time. To publish the application’s logger and telemetry together:

app.SetDefaults()

This changes process-wide state; prefer explicit handles when applications or tests share a process. telemetry.SetDefault(nil) restores the unstarted no-op default. Publishing a handle does not start it.

Focused fragments keep error results visible and omit routine propagation. Complete application code must handle those errors before using results.

Recording Spans

Suppose your report renderer accepts a caller’s ctx and a validated format restricted to pdf or csv. Name the tracer’s instrumentation scope after your application’s package, and keep the operation name fixed:

tracer := app.Telemetry().Tracer("example.com/reports")
ctx, span := tracer.Start(ctx, "report render",
	telemetry.Attrs(telemetry.String("format", format)),
)
defer span.End()

Pass the returned context to the renderer’s dependencies. Their spans can then be children of this span. The default kind is Internal; use Client for a remote call, Producer for sending a message, and Server or Consumer for receiving work. Without a started trace driver or a trace-capable observer, Start returns the original context and a non-recording span without new IDs.

Record a failure with a fixed description rather than raw diagnostics. Here render is your existing func(context.Context, string) error:

err := render(ctx, format)
if err != nil {
	span.SetStatus(telemetry.Error, "render failed")
	span.SetAttributes(telemetry.String("error.type", "render_failed"))
}

RecordError(err) also records err.Error() as an exception event. Use it only when that text is approved for capture and export. Built-in components classify failures without recording their text; see Telemetry Privacy. Unset is the initial status, not a failure. OK is final; setting Unset, an empty status, or an unknown status does not clear an existing verdict.

Call End on every span, normally with defer. Its first call closes admission to updates and waits for already-admitted recorder calls; subsequent calls and late updates do nothing. Recorder callbacks must return and must not synchronously end their own span, including from error formatting.

Attributes use String, Int, Int64, Float64, Bool, Strings, or Duration (float seconds). Empty keys are dropped. Attribute containers are copied at publication and backend boundaries, but copying does not sanitize their contents. Check span.IsRecording() before expensive attribute work.

Recording Metrics

Create instruments once, for example in your service constructor. For the same report renderer, count completions by the bounded format value:

meter := app.Telemetry().Meter("example.com/reports")
renders := meter.Counter("reports.renders",
	telemetry.Unit("{render}"), telemetry.Keys("format"),
)

At the operation boundary, record one observation with its context:

renders.Add(ctx, 1, telemetry.String("format", format))

Use a histogram for distributions, an up-down counter for in-flight work, and a gauge for a current value. To record rendering duration, import standard-library time; startedAt is captured before rendering:

duration := meter.Histogram("reports.render.duration",
	telemetry.Unit("s"), telemetry.Keys("format"),
)
duration.Record(ctx, time.Since(startedAt).Seconds(), telemetry.String("format", format))

Keep the histogram declaration in the constructor too. Instruments can be declared before startup and bind on first use afterward. Matching scope/name pairs share a backend instrument; a later declaration with different kind or keys is disabled with a warning through Setup.Logger. Driver rejection also disables that instrument, not the observed application operation.

Keys filters recordings to the declared schema. Missing values become empty strings; undeclared keys disappear and non-string values become text. This does not bound the number or size of label values: never use request IDs, arbitrary paths, or user input. Negative counter additions and NaN histogram values are ignored. Bucket hints are sorted, deduplicated, and stripped of non-finite bounds. Published key and bucket containers are detached from option drafts, and backends receive independent copies; retaining or normalizing a draft cannot change the facade’s label schema.

For reusable package instrumentation that follows changing handles, telemetry.NewInstruments lazily builds and caches instruments for the most recent handle; For(handle) rebuilds when the handle changes. Do not keep one global instrument bound to an unrelated application’s handle. Enabled() is a startup-state check, not proof of a configured signal: a started New(nil) also returns true so observers can participate.

Trace Propagation

A process boundary needs serialized trace context, not a transported Go context. HTTP, queues, broadcast, and workflows provide their own integration. For an application-specific transport, initialize a carrier and inject the sending context:

metadata := telemetry.MapCarrier{}
app.Telemetry().Inject(ctx, metadata)

Send this carrier with the message. Only after authenticating or otherwise trusting its source should the receiver extract it into its local operation context:

ctx = app.Telemetry().Extract(ctx, metadata)

HeaderCarrier adapts http.Header; MapCarrier adapts string metadata. The driver owns the wire format. Missing or malformed context leaves the context unchanged. Raw extraction does not authenticate the sender or scrub baggage. Request Tracing explains the separate HTTP defaults: reject incoming headers, but allow outgoing injection to internal destinations.

Correlated Logs

The application wraps its logger. Use context-aware methods while the context contains a valid span:

app.Logger().InfoContext(ctx, "report rendered", "format", format)

The normal handler receives trace_id and span_id; plain Info has no context and adds neither. A log-capable driver can also export the complete record. Correlation is not redaction: review log messages and attributes before enabling log export.

telemetry.SpanFromContext(ctx) never returns nil. Its Context() exposes TraceID and SpanID, whose String() methods return hex. A valid identity does not guarantee that a backend retained the trace.

Passive Observation

Use Observe when another in-process component needs facade observations without replacing the configured backend. With an existing started observer implementing telemetry.Observer:

unsubscribe := app.Telemetry().Observe(observer)
defer unsubscribe()

Observation neither starts nor stops the observer, and the facade must also be started. For application-owned bounded history and protected access, use Capture Setup, not an ad hoc public debug endpoint.

Callbacks are synchronous on producing goroutines and can run concurrently. Keep them fast, nonblocking, panic-free, and nonrecursive for the same signal. Finished spans report the backend-selected context and parent; attributes, events, status, and timestamps reflect facade calls, not the whole native SDK stream. Native OpenTelemetry parent selection takes precedence when valid.

Span snapshots are shared read-only. Copy containers before editing, clone log records before retaining them, and copy measurement label values before the callback returns. Only spans started while subscribed are captured; they can finish after detachment if notification is admitted while telemetry remains enabled. Stop suppresses new notifications but neither drains nor cancels already-admitted callbacks. Coordinate your own observer completion barrier.

An observer implementing TraceDriver or MetricDriver can supply a missing signal; the first eligible observer wins and a configured driver is never replaced. That observer records through its driver role, without duplicate passive notification for the same signal. Detaching restores the next eligible observer or the no-op; existing spans retain their original recorder.

Lifecycle Rules

Before Start and after Stop, the facade records nothing. A second Start wraps ErrStarted; a start after stop or failed startup wraps ErrStopped. Construct a new handle to retry a failed startup. Stop is safe before startup and on repeated calls. Lifecycle calls serialize across backend work; their callbacks must return and must not synchronously reenter the same lifecycle. Cancellation cannot forcibly interrupt backend work or waiting for an earlier lifecycle call. Startup receives its context without retaining it; stop forwards the supplied context, including cancellation, after disabling new recordings and notifications.

The application manages this lifecycle for a configured handle. It gives telemetry the remaining shutdown budget, with at least ten seconds. Drain application work and end spans before telemetry stops. Background exporters can lose the final batch if the process exits without stopping them.

Outside an application, start the handle yourself. Here handle is your configured *telemetry.Telemetry:

err := handle.Start(ctx, telemetry.Setup{
	Resource: telemetry.Resource{Service: "reports-cli"},
})

After successful startup and completed work, use a fresh bounded shutdown context instead of reusing a canceled operation context. Import context, errors, and time; err is the operation error you want to preserve:

shutdown, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
err = errors.Join(err, handle.Stop(shutdown))

Preserve shutdown errors. OpenTelemetry flushes with this budget but completes owned native teardown on an uncanceled context, so Stop can outlast the deadline; see export lifecycle.

For assertions on reported work, use the telemetry fake. For all declarations, see the telemetry API reference.