Instrumented Components

Use this catalog when you need the exact instrument behind a slow operation or an alert. It describes observations made through Arkoren facades, not every native SDK call. Configure a telemetry handle first; following a trace across processes also requires propagation and telemetry in each process.

Shared Rules

Core service scopes are their package paths, such as arkoren.dev/core/service/cache. Unless an entry says otherwise, spans are Client spans and duration histograms use seconds. Each listed error label is error.type; classifications vary by component between known sentinels, fixed fallbacks, and Go type names. Automatic spans and metrics omit raw error text and do not call RecordError.

Set the component’s Builder.Telemetry(handle) to select an explicit handle. Nil or omission follows the process default at operation time; messaging, workflow, and LLM use their role-specific builders below. Native driver calls bypass facade instrumentation. An inactive handle skips observation work; an enabled handle does not imply every signal has a backend.

The exclusions here are not a redaction filter for logs, custom attributes, or diagnostic tools. Review Telemetry Privacy separately. Counts describe observed calls or attempts, not an exactly-once business feed; events and auditing have different responsibilities.

Database

database.Builder.Telemetry selects the handle. Statements through Exec, Select, Find, and Cursor record spans named by SQL verb and known table, such as SELECT reports; without a known table they use the verb, and without a known verb they use the database system. Cursor timing covers traversal, not a separate span per row. Lock and TryLock observe acquisition as LOCK and TRY_LOCK.

db.client.operation.duration uses db.system.name, db.namespace, db.operation.name, db.collection.name, and error.type. Recognized leading verbs are SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, DROP, and TRUNCATE; this classification does not parse comments or CTEs. Missing rows are not telemetry failures.

QueryText defaults to false. Enabling it exports compiled SQL verbatim, including literals and comments; bound arguments stay separate but embedded values are not redacted. Query text never becomes a metric label. See database monitoring before opting in.

Cache

cache.Builder.Telemetry selects spans named cache {operation} and these metrics:

Operations are get, put, compare_and_put, delete, add, has, pull, increment, decrement, increment_with_ttl, ttl, lock, try_lock, and remember. Store is Builder.Name, or default. Lock spans cover acquisition, including the blocking wait, not lease extension or release.

Known reads add a boolean arkoren.cache.hit span attribute and a true or false counter label. A missing key is a miss, not an error; finding bytes that fail decoding is still a hit. Unknown lookup outcomes and non-read operations have no span hit attribute and an empty counter hit label. compare_and_put guard mismatch is a successful comparison, not a miss or failure. Keys, guards, and values are omitted. For a correctly filtered hit ratio and nested remember observations, see cache monitoring.

Feature Flags

feature.Builder.Telemetry observes each evaluation with an internal feature evaluate span.

The provider name is Builder.Name or default. Reason comes from the driver, such as static or targeting_match, and becomes error on failure. A supplied feature_flag.result.variant is span-only. Subject keys, subject attributes, and flag values are omitted; flag keys and variants still require your privacy and cardinality review. See feature evaluation.

Search

search.Builder.Telemetry observes sync, check, put, delete, drop, and search as search {operation} spans. arkoren.search.operations is the counter; arkoren.search.operation.duration is the histogram. Both use arkoren.search.operation, arkoren.search.name, arkoren.search.index, and error.type.

Name is Builder.Name or default; index is empty for engine-wide sync and check. Successful search spans add arkoren.search.hits for the returned page only. Queries, documents, and raw diagnostics are omitted. Index names can still reveal tenants; keep them bounded. See Search.

Storage

storage.Builder.Telemetry records storage {operation} spans for open, put, stat, delete, list, and sign. The counter arkoren.storage.operations and histogram arkoren.storage.operation.duration use arkoren.storage.operation, arkoren.storage.store, and error.type. Store is Builder.Name or default; successful writes add the span-only arkoren.storage.object.size in bytes. Histogram bounds span 1 ms to 1 minute.

open ends when the driver returns the body, not after the caller reads or closes it. put measures reader submission; upload helpers reuse that observation. list measures one page, and sign measures URL creation rather than the later request using it. Missing objects are not failures. Object keys, prefixes, cursors, and signed bearer URLs are omitted. See storage observations.

Mail

mail.Builder.Telemetry records mail send for Mailer.Send with a nonnil context. arkoren.mail.operation is send; span-only counts are arkoren.mail.recipients (to, cc, and bcc combined) and arkoren.mail.attachments. The arkoren.mail.operations counter and arkoren.mail.operation.duration histogram use operation and error.type. Histogram bounds span 10 ms to 1 minute.

Validation failures are observed before submission. Error precedence is context.Canceled, context.DeadlineExceeded, mail.ErrInvalidMessage, then the innermost wrapped Go error type. Addresses, subjects, bodies, provider diagnostics, and receipts are omitted.

These metrics count attempts, not confirmed delivery. A driver panic closes the mailer’s span and records attempt count/duration without ending the parent span, retaining panic data, or classifying the panic itself; the panic still propagates and emits no Sent or Failed notification. Nil context produces neither telemetry nor events. Notification failure classes are bounded and separate from telemetry’s native-type fallback; see mail observations.

Push Notifications

push.Builder.Telemetry observes calls after portable preflight validation as push send. The counter arkoren.push.operations and histogram arkoren.push.operation.duration use arkoren.push.operation and error.type. Operation is send.

push.ErrOutcomeUnknown wins over joined or wrapped cancellation, deadline, and provider failures in telemetry. Otherwise cancellation precedes deadline, known push sentinels, and the innermost wrapped Go type. This changes observation classification only: returned errors preserve their causes and notifications retain their separate bounded classes.

No recipient, endpoint, token, encryption material, content, application data, or receipt ID enters these fields. Acceptance does not prove device delivery or display. See Push Notifications.

Payments

payment.Builder.Telemetry observes provider operations after portable validation as payment {operation}. The counter arkoren.payment.operations and histogram arkoren.payment.operation.duration use arkoren.payment.operation and error.type.

Keys, IDs, amounts, approval URLs, customer details, and payloads are omitted. API acceptance is not settlement evidence. For the distinction between these observations and business reconciliation, see Payments.

Secrets

secret.Builder.Telemetry observes provider get, put, update, and delete as secret {operation}. The arkoren.secret.operations counter and arkoren.secret.operation.duration histogram use arkoren.secret.operation and error.type. Failure classes are only context.Canceled, context.DeadlineExceeded, secret.ErrNotFound, or other, in that order. Unlike a cache miss, a missing secret is a failure observation.

Raw observes one provider read; String and typed Get reuse it. A later JSON decode failure is not a provider failure. Direct Driver() or native calls bypass these observations. Names, versions, values, sizes, and native diagnostics are omitted. See Managing Secrets.

Hashing And Encryption

hash.Builder.Telemetry and crypto.Builder.Telemetry record metrics only: their contextless APIs create no synthetic root spans.

Hash operations are hash, check, and needs_rehash; outcomes are ok, match, mismatch, current, rehash_needed, or error. Mismatch is not an error. Failure classes are hash.ErrInvalidHash or error. Crypto operations are encrypt and decrypt; outcomes are ok, previous_key, or error. Its classes are crypto.ErrEncrypterClosed, crypto.ErrInvalidCiphertext, or error.

Metrics cover normally completed facade calls, including JSON work; panics do not create completed observations. Decryption reports one final outcome across all keys, not a failure per attempted key. No input, output, key, or native error text/type is retained. Rehash-needed and previous-key outcomes do not prove a persisted upgrade; applications own persistence as described in Hashing and Encryption.

Queues And Broadcast

Set telemetry on every participating queue.DispatcherBuilder, queue.ConsumerBuilder, broadcast.PublisherBuilder, and broadcast.ConsumerBuilder. Producers record send {destination}; consumers record process {destination}. With propagation enabled on both sides and preserved by the transport, consumer work can continue the producer trace. Queue retries are separate consumer attempts.

Counters are messaging.client.sent.messages and messaging.client.consumed.messages; histograms are messaging.client.operation.duration for producers and messaging.process.duration for consumers. All four use messaging.system, messaging.operation.name, messaging.destination.name, and error.type. Send counts include attempted sends, not only confirmed acceptance.

Payloads and metadata values are not span attributes or labels, but propagation does write trace context into metadata readable by broker operators. HTTP’s incoming trust policy does not authenticate broker messages. See queue telemetry and broadcast metadata.

Scheduling

schedule.Builder.Telemetry records an internal schedule run span for a recurring entry or deferred task. Its returned execution context lets the task’s dependencies become child spans.

The counter arkoren.schedule.run.count and histogram arkoren.schedule.run.duration use arkoren.schedule.entry, arkoren.schedule.kind, arkoren.schedule.outcome, and error.type. Histogram bounds span 1 ms to 1 hour. Kind is entry or task; outcome is ok, skipped, or failed. schedule.ErrSkipped is not a failed span; recovered panics use schedule.panic as their error class. Task payloads and error text are omitted, but developer-selected names are recorded. See scheduled failures.

HTTP

Use the handle’s Middleware() and Transport() rather than service-builder options. Their scope is arkoren.dev/core/service/telemetry.

Unknown methods use HTTP in span names and _OTHER in method attributes; unmatched server requests omit the route from the name. Server duration uses method, route, status, scheme, and error.type; active requests use method and scheme. Client duration uses method, destination address/port, status, and error.type. Full keys and outcome rules are in Request Tracing.

Server completion includes router error rendering, not proof of delivery to the client. Client completion includes consuming or closing the response body. The retrying HTTP client already installs telemetry per attempt and marks resends with http.request.resend_count; wrapping it again duplicates observations.

MCP

mcpserver.Builder.Telemetry selects managed-call observations in scope arkoren.dev/driver/mcp. An internal mcp call_tool span, the counter arkoren.mcp.calls, and histogram arkoren.mcp.call.duration use only arkoren.mcp.tool (registered name) and arkoren.mcp.outcome (success or error).

Arguments, results, raw errors, unknown tool names, and model-inference fields are omitted. Native SDK registrations bypass this instrumentation. This does not make the tool result itself private; diagnostic tools can return raw logs and attributes. See managed MCP observation and protected diagnostic registration.

Language Models

llm.ModelBuilder.Telemetry and llm.EmbedderBuilder.Telemetry select these observations in arkoren.dev/core/service/llm:

gen_ai.client.operation.duration uses gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, and error.type. gen_ai.client.token.usage adds gen_ai.token.type (input or output) when usage is reported. Other span-only fields include response ID, finish reasons, agent/tool names, token details, agent step count, and embedding input count. Model, agent, and tool names must remain bounded and nonsensitive.

Prompts, replies, tool arguments/results, and error text are excluded. Stream traversal and cooperative iterator cleanup affect completion; usage is provider-reported rather than an invoice. See Observing Calls for the consumption and notification boundaries.

Application Lifecycle

The application creates no lifecycle spans. It supplies the resource, owns telemetry startup/shutdown, and wraps its logger for correlation. Built-in log messages include app started, app stopping, and app stopped; there is no app starting log. application.Starting runs after telemetry starts and before user startup callbacks, so listeners can log through active telemetry. See application lifecycle.

Workflow Attempts

Configure workflow.ClientBuilder.Telemetry in producers and workflow.RegistryBuilder.Telemetry through workflow.NewRegistry in workers. Each process follows its own dynamic default when omitted. Client spans are workflow start, workflow signal, workflow cancel, workflow status, and workflow result; actual registered handler invocations get internal workflow step_attempt spans.

The counter arkoren.workflow.operation.count and histogram arkoren.workflow.operation.duration use arkoren.workflow.operation, arkoren.workflow.outcome, and error.type. Outcomes are success, error, or already_started; duplicate-start handling preserves the API result while not classifying it as a telemetry failure. Names, run IDs, payloads, signals, and metadata are excluded from these automatic fields.

Retries create separate attempts. Replayed orchestration and recorded results create none; do not add ordinary observation side effects to replayed workflow functions. result measures the caller’s wait, not workflow runtime, and canceling that wait does not cancel the run. Local events and polling are not an authoritative terminal feed. Export from each process to a shared backend for cross-replica inspection; local capture cannot aggregate pods.

Serialized StartOptions.Trace is separate from application metadata and is extracted at supported native attempt boundaries without transporting the caller’s Go context. Signals do not replace the start parent. Upgrade Temporal workers before traced producers: older workers cannot decode traced envelopes; untraced histories retain their shape. See the available workflow drivers and Events And Telemetry.

For custom observations, follow Recording Spans and Recording Metrics. Full declarations live in the telemetry API, with adapter-specific MCP settings in the mcpserver API.