# Arkoren guides

> Bulk documentation archive — not the recommended starting point for coding tasks.

This exhaustive export is intended for offline indexing, bulk processing, or deliberate large-context use. Loading it all can exhaust an agent's context window and increase processing cost.

Start with [the documentation index](https://arkoren.dev/llms.txt) and fetch only the pages your task needs.

Documentation content: 209 pages; 1,949,638 UTF-8 bytes; approximately 487,238 tokens (rough estimate: characters / 4, not a model tokenizer; actual counts vary by model).

---

Source: https://arkoren.dev/docs/guides/about/introduction.md
[HTML documentation](https://arkoren.dev/docs/guides/about/introduction)

# Introduction

Arkoren is a modular Go toolkit for building services. You can add a cache to an
existing application, use its HTTP packages for a new service, or adopt its
application lifecycle to coordinate startup and shutdown. Each choice is separate:
using one capability does not require adopting the rest.

## Modular Adoption

Keep the parts of your application that already work. Arkoren services do not
require its router, and neither the router nor the services require
`application.Application`. You construct dependencies in Go and pass them to the
code that uses them. The optional [application lifecycle](https://arkoren.dev/docs/guides/application/overview.md)
coordinates the startup hooks, workers, and cleanup you explicitly register; it
does not discover dependencies or manage resources merely because you created them.

The module boundaries let you make the same choice about dependencies. Core,
`arkoren.dev/core`, provides HTTP packages, application lifecycle, and service
APIs. It depends on standard-library-only foundation modules, not vendor SDKs.
Foundation libraries, such as validation and SQL statement construction, are also
available independently. Vendor integrations live in separate driver modules,
which bring their own dependencies when you add them.

If you are deciding whether these conventions suit your application, read
[Choosing Arkoren](https://arkoren.dev/docs/guides/about/choosing-arkoren.md) before installing anything.

## Shared Service APIs

A cache used by one process has different storage needs from a cache shared by
several servers. Arkoren separates those backend choices from the code that uses
the cache. A **facade** is the application-facing API; a **driver** implements
the service's **contract**, the Go interface that specifies its backend behavior.

For example, you can use the cache facade with its bundled memory driver for
process-local data, then configure a Redis driver when servers need shared state.
Calls through the facade retain the same API. That does not make the backends
equivalent: memory-cache contents disappear with the process, while Redis requires
a running service and its own configuration and operating policy.

The [facades and drivers guide](https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers.md) explains
this separation. Driver guides cover setup and backend limits; conformance suites
test shared contract behavior rather than proving identical durability or
performance.

## Where to Begin

Choose the path that matches what you have already built:

- For an existing Go application, follow [Installation](https://arkoren.dev/docs/guides/first-steps/installation.md#existing-applications)
  to add one capability without replacing your entry point or HTTP stack.
- For a new service, use [Creating Projects](https://arkoren.dev/docs/guides/first-steps/creating-a-project.md)
  for bootstrap files, then follow the [Quickstart](https://arkoren.dev/docs/guides/first-steps/quickstart.md)
  to run it and verify an HTTP response.
- To install the optional command-line tool, follow
  [Installing Arkoren](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren). Library use does not
  require the CLI.

After that first task, read the guide for the capability you need rather than
working through every category. Each service guide explains ordinary usage and
links to driver-specific setup where backend choices matter.

## Reading the Guides

Focused code snippets assume the collaborators named in the surrounding prose.
They keep error results visible but omit routine error propagation. Checks remain
when they explain recovery, authorization, resource ownership, or another
consequential decision. Complete programs handle errors before using results;
your application must do the same.

For exact declarations, use `go doc` in your application's module directory or
the package links to pkg.go.dev. Local `go doc` reflects the dependencies selected
by your module graph and active workspace, so check that selection when a version
matters. The guides teach usage rather than maintaining a second API inventory.

These same guides ship as an agent skill. [Working with Agents](https://arkoren.dev/docs/guides/agent-skills/working-with-agents.md)
explains how to give a coding agent access to them.

---

Source: https://arkoren.dev/docs/guides/about/choosing-arkoren.md
[HTML documentation](https://arkoren.dev/docs/guides/about/choosing-arkoren)

# Choosing Arkoren

Choose Arkoren for a capability whose shared API and integration conventions help
your application. You do not need to replace a working HTTP stack or standardize
every dependency at once. The useful comparison is between the integration work
you would otherwise maintain and the conventions and limits you accept by using
Arkoren.

## Evaluate the Fit

If several services need common application-facing APIs and test substitutes,
Arkoren's [facades and drivers](https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers.md) provide
that separation. Your application calls a facade while a driver supplies backend
behavior. For a cache, this lets a unit test use process-local storage while a
deployed service uses a shared backend without changing the facade calls.

That boundary is useful only if the contract covers your task. If most of your
code needs a vendor's specialized operations, a portable facade might add little
value. Use the vendor package directly where appropriate; do not assume every
vendor feature has an Arkoren equivalent.

The same reasoning applies to application structure. Explicit constructors and
optional lifecycle hooks fit applications where you want dependency wiring in
Go. If a small `net/http` service already meets your needs, keeping it is a valid
choice. Arkoren does not provide automatic dependency discovery or a container
that wires the application for you.

## Understand the Boundaries

Shared APIs do not remove backend differences. Review the relevant
[driver guide](https://arkoren.dev/docs/guides/drivers/choosing-a-driver.md) before choosing an implementation,
especially if your task depends on persistence, coordination between servers,
or an optional operation. Some contracts permit unsupported operations to return
`ErrUnsupported`; the presence of a facade method alone is not proof of support
by every driver.

Memory drivers serve local and test scenarios, but their state is process-local
and does not survive a restart. A passing [conformance suite](https://arkoren.dev/docs/guides/testing/conformance-suites.md)
is evidence about the behavior that suite tests, not about distributed operation,
production capacity, or identical durability across backends. Vendor integration
tests add backend-specific evidence; your deployment still needs its own tests.

You also retain responsibility for credentials, resource startup and cleanup,
schema migrations, and authorization policy. Installing a module does not
configure those responsibilities. There are no built-in [arks](https://arkoren.dev/docs/guides/core-concepts/arks.md),
the optional domain features intended to combine services with product-specific
rules; do not assume signup or another complete business flow is included.

Core's Go dependencies are Arkoren's standard-library-only foundation modules,
not third-party libraries. Vendor modules do bring external dependencies.
Evaluate the module graph you actually install rather than treating the whole
application as dependency-free.

## Keep Existing Components

For a first adoption, choose a service or foundation library that does not affect
HTTP routing. Construct it where your application already constructs dependencies
and pass it to existing handlers. Neither the Arkoren router nor its application
lifecycle is required.

If you later adopt Arkoren HTTP handlers, its router and error-returning handler
type both implement `http.Handler`. An existing router can serve them wherever it
accepts that interface. In the other direction, `router.HTTP` adapts a standard
handler, and `Mount` serves a handler tree beneath a prefix. `Mount` strips that
prefix before forwarding the request, so test path handling and middleware order
at the boundary. [Routing](https://arkoren.dev/docs/guides/http-fundamentals/routing.md#mount-native-handlers)
shows the integration.

## Try One Capability

Arkoren requires Go 1.27 or later and is pre-1.0. Modules version independently,
and v0 minor releases can break APIs. Read the
[compatibility and support policy](https://arkoren.dev/docs/guides/first-steps/installation.md#compatibility-and-support)
before adopting a module: security fixes target its latest published version,
with no guaranteed backports or fixed long-term support window. Conformance and
integration tests do not establish an API freeze, performance guarantee, or
production track record. If you require those assurances, establish them
separately before committing.

Start with the [existing-application installation example](https://arkoren.dev/docs/guides/first-steps/installation.md#existing-applications)
and add a test for the behavior you need. Before expanding adoption, test the
chosen production driver against its backend, including the failure and
unsupported-operation cases relevant to your application. Keep the dependency
versions recorded in your application's module files and rerun those tests when
upgrading.

---

Source: https://arkoren.dev/docs/guides/about/contributing.md
[HTML documentation](https://arkoren.dev/docs/guides/about/contributing)

# Contributing

This page is for changing Arkoren itself, not for adding it to an application.
Start with a checkout of `github.com/arkoren/arkoren` and read its `AGENTS.md`.
Identify the module that owns your change before editing: a service contract
can affect several drivers even when the interface change is small.

Run the commands below from the repository root. Go work needs Go 1.27 or later;
linting also needs `golangci-lint`, and race tests need a C compiler. Additional
requirements for integration and documentation checks appear beside those tasks.

## Locate the Change

The repository has no root Go module. Its product modules have separate
manifests and dependency versions:

| Location | Responsibility |
| --- | --- |
| `core/` | Application lifecycle, HTTP packages, service contracts and facades, bundled local implementations, and conformance suites. |
| `foundation/<name>/` | Independent libraries that import only the standard library. |
| `driver/<vendor>/` | Vendor implementations and provider-specific adapters, one module per vendor. |
| `ark/<name>/` | Reserved for opt-in domain features; none are built in yet. |
| `cmd/arkoren/` | Consumer CLI, project templates, and embedded documentation skill. |

Keep vendor dependencies outside core. Core must not import a driver, ark, or
CLI; driver modules must not require other driver modules, and arks must not
require other arks. A provider-specific HTTP adapter can belong in its vendor
module without introducing a portable service contract. Read the contributor
`arkoren-codebase` skill before a change that crosses package boundaries or adds
a package or driver, and `arkoren-go` before writing Go.

The committed `go.work` supports development across the modules it lists.
Module manifests declare real dependency versions without local `replace`
directives. Workspace builds can use edited local dependencies, but the shared
Make recipes set `GOWORK=off` to verify each module independently.

Do not run `go work sync` or a repository-wide `go mod tidy`: they can rewrite
unrelated modules and change dependency versions. If dependency maintenance
requires `go mod tidy`, restrict it to the affected module and review the diff.

## Coordinate Contract Changes

A service's `Driver` interface specifies backend behavior. If you change that
contract, find every implementation and caller rather than relying on the
interface package's tests alone. Update these together:

- The `<contract>test` conformance suite and the bundled memory implementation,
  when one exists.
- Every vendor implementation of the contract and its affected tests.
- Public Go doc comments, relevant consumer guides, and contributor instructions
  affected by the behavior change.

Use the conformance suite to test the documented behavior. Add coverage for a
new requirement rather than leaving it as an untested promise. The
[driver-writing guide](https://arkoren.dev/docs/guides/drivers/writing-a-driver.md) explains how implementations
use these suites.

Preserve application ownership when changing persistence: drivers do not run
DDL themselves. A driver that needs tables supplies `Migrations()` for the
application to execute, and existing migration groups must remain stable.

## Run Module Checks

While developing, select the owning module and narrow tests to the affected
package. For an HTTP change in core, for example:

```sh
make test MODULE=core PKG=./http/... RUN=TestRequest
```

`MODULE` names the module directory; `PKG` is relative to that module, and `RUN`
is a Go test regular expression. This runs short tests with normal built-in vet
and normal test-result caching. Choose a pattern that actually matches your tests.

Before submitting a Go change, check each affected module. For core:

```sh
make check MODULE=core
```

`check` runs lint, including configured govet, and short race tests. Race tests
disable test-time vet to avoid repeating it; ordinary test caching remains
enabled. `make vet MODULE=core` runs standalone toolchain vet when you need that
additional audit, rather than replacing the routine lint check.

## Run Integration Tests

For a module with `compose.yaml`, install Docker Compose and make sure its fixed
host ports are available. Integration teardown removes containers and volumes.
By default, the target creates a unique disposable Compose project. If you set
`COMPOSE_PROJECT_NAME`, use a disposable project: the target removes that named
project's containers and volumes too.

To check the PostgreSQL driver against its local service:

```sh
make integration MODULE=driver/postgres
```

The target waits for healthy services, runs uncached full race tests without
`-short`, and attempts cleanup on success, failure, or handled interruption.
Failure diagnostics include service logs. A cleanup failure fails an otherwise
successful run; if tests already failed, their failure status is preserved.
Live-provider tests can need credentials beyond this local Compose setup.

For manual investigation, start services with disposable test data:

```sh
make services MODULE=driver/postgres
```

Unlike integration runs, this command leaves services running in the persistent
default project `arkoren-manual-driver-postgres`. When your investigation is
finished, remove the services and their volumes:

```sh
make cleanup MODULE=driver/postgres
```

Cleanup deletes the project's test data. If you override `COMPOSE_PROJECT_NAME`,
pass the same value to both commands.

## Update Documentation

Author consumer guides in `cmd/arkoren/skills/arkoren/references/guides/`. The CLI
embeds the skill tree directly and the website renders the same Markdown. Keep
API declarations in Go doc comments rather than copying signature inventories
into guides.

Before changing a guide, read the contributor `developer-documentation` skill
under `.agents/skills/`. For a material rewrite, establish the reader's task and
page scope, check claims and examples against source and tests, and review the
finished page separately for comprehension and technical fidelity. Driver-page
changes also require rechecking their compatibility matrices against every
affected implementation.

Install the website's Node.js dependencies and Playwright Chromium using
`website/README.md`, then run:

```sh
make generate
make docs
```

Generation updates the consumer guide index; there is no documentation archive
to generate. The docs target runs website checks, a build, and tests. The build
checks index freshness without repairing it and validates metadata; rendered
tests check links and navigation. Review relative Markdown links and editorial
conventions yourself. Passing rendered tests does not verify API claims or
example correctness.

## Prepare the Review

Describe the behavior changed, the modules and guides affected, and the evidence
you collected. Distinguish tests actually run from source inspection, and report
checks you could not run. For documentation examples, distinguish compiled
programs from API-checked fragments.

Routine CI discovers canonical modules with `scripts/ci-modules.sh` and checks
them independently. Compose modules run lint in the check jobs and full race
suites in integration. Require both the `CI gate` for Go and tooling and the
separate `website` check; the gate alone does not cover rendered documentation.
The scheduled and manually dispatched audit separately runs standalone vet,
vulnerability scans, and the CLI generated-project end-to-end test.

If you edit workflows, run `make workflow-check` with its required tools,
including `actionlint`, rerun the affected shared recipes, and inspect the actual
GitHub CI results. Local fixtures do not emulate GitHub Actions.

## License

Arkoren uses the MIT license in the repository's `LICENSE` file. Contributions
are made under those same terms.

---

Source: https://arkoren.dev/docs/guides/first-steps/installation.md
[HTML documentation](https://arkoren.dev/docs/guides/first-steps/installation)

# Installation

Add the Arkoren capability your application needs, keeping your existing packages
and entry point. You do not need the CLI, a generated project, an Arkoren router,
or `application.Application` to use a library or service facade.

For a new HTTP service with bootstrap files, use [Creating Projects](https://arkoren.dev/docs/guides/first-steps/creating-a-project.md)
after checking the requirements below. Installation itself does not generate an application.

## Go Requirements

Use Go 1.27 or later. Check the toolchain before adding dependencies:

```sh
go version
```

Run installation commands from the directory containing your application's
`go.mod`. In a repository with several Go modules, choose the module that will
import the package. If you have no module yet, create one in your application's
directory with `go mod init example.com/your-app`, using your intended module path.

Core has no external-service prerequisite. Vendor integrations can require a
running backend, credentials, or native dependencies; follow the relevant
[driver guide](https://arkoren.dev/docs/guides/drivers/choosing-a-driver.md) before connecting one.

## Compatibility and Support

Arkoren is pre-1.0. Each module versions independently: core, a foundation
library, and a driver do not need matching version numbers. Go resolves the
dependency versions declared by the modules your application uses.

During v0, breaking API changes use a minor release, such as `v0.1.x` to
`v0.2.0`; patch releases are intended to remain compatible within that minor
line. A minor release can also add features without breaking existing callers.
Review the [release notes](https://arkoren.dev/changelog) before upgrading, commit
your resolved `go.mod` and `go.sum`, and run your application's tests against the
backends you deploy. Pre-1.0 APIs are not frozen.

Security fixes target the latest published version of each affected module.
Older versions have no guaranteed security backports, so receiving a fix can
require an upgrade to a newer minor release. There is no fixed long-term support
window or guaranteed response or resolution time. Use the
[security policy](https://github.com/arkoren/arkoren/security/policy) to report
suspected vulnerabilities privately.

GitHub's prerelease label does not change Go version selection: a plain `v0.X.Y`
tag is not a Go prerelease version and can be selected by `@latest`.

## Choosing Modules

`arkoren.dev` is an import namespace, not a module you install. Choose the module
that contains the package you want:

| Need | Module | Example Import |
| --- | --- | --- |
| HTTP, application lifecycle, or service facades | `arkoren.dev/core` | `arkoren.dev/core/service/cache` |
| An independent library | `arkoren.dev/foundation/<name>` | `arkoren.dev/foundation/validate` |
| A vendor integration | `arkoren.dev/driver/<vendor>` | `arkoren.dev/driver/redis/rediscache` |

To use a core package, add core:

```sh
go get arkoren.dev/core
```

This includes the core service contracts and their bundled local implementations,
including the memory cache. It does not start services or require that you use
other core packages. Go compiles only packages reachable through your imports.

Foundation libraries are independent modules. If you need validation without
core, install it directly:

```sh
go get arkoren.dev/foundation/validate
```

Each foundation module uses only the standard library. When core already requires
a foundation library, Go resolves that dependency automatically.

Add a vendor module when your application needs its implementation. For example,
a shared Redis cache uses:

```sh
go get arkoren.dev/driver/redis
```

The module contains vendor-specific packages such as `rediscache`, rather than
replacing the core `cache` package. Installing it does not configure credentials,
connect to Redis, or register cleanup. [Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md) owns
that setup. Integrations marked pending in the driver guides are unpublished;
do not infer an installation command from their proposed package names.

## Existing Applications

Keep dependency construction wherever your application already does it. A cache
has a **facade**, which application code calls, and a **driver**, which stores
the values. With these imports, construct a process-local cache:

```go
import (
	"arkoren.dev/core/service/cache"
	"arkoren.dev/core/service/cache/memorycache"
)
```

The construction needs no network service or credentials:

```go
driver := memorycache.New()
store := cache.New(driver)
```

Pass `store` to the code that needs it through your existing constructors or
struct fields. A library can accept a caller-supplied cache instead of creating
one. Neither approach requires changes to your HTTP stack.

To cache a rebuildable display label, define its value type and write it using
an existing operation context `ctx`. Import `time` for the expiry:

```go
label := cache.NewKey[string]("catalog.label.featured", func(entry *cache.KeyBuilder) {
	entry.TTL(time.Minute)
})
err := store.Put(ctx, label, "Featured Items")
```

The key supplies the exact storage name, `string` value type, and one-minute write
TTL. Check `err` in application code. The guides' focused fragments keep routine
error results visible without repeating propagation; complete runnable examples
handle their errors.

The memory driver's constructor starts no goroutines. Expired entries are not
returned even without its optional expiry janitor. Values disappear on process
exit and are not shared between replicas, so this is not durable storage. If you
start the janitor or adopt a network driver, the code that owns the driver must
also stop it; see [Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md).

Continue with [Cache Overview](https://arkoren.dev/docs/guides/cache/overview.md) for reads and misses, or
[Key Definitions](https://arkoren.dev/docs/guides/cache/keys.md) for reusable names and expiry policies.

## Optional Tooling

The CLI is separate from runtime dependencies. Install it if you want project
generation, bundled documentation, or diagnostics commands:

```sh
go install arkoren.dev/cmd/arkoren@latest
```

Alternatively, pin it as a tool in your application's module:

```sh
go get -tool arkoren.dev/cmd/arkoren@latest
```

Use `go tool arkoren` for the module-tool installation. For executable lookup
and `PATH`, see [CLI Installation](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren).
Then [create a project](https://arkoren.dev/docs/guides/first-steps/creating-a-project.md) or keep integrating individual
packages into the application you already have.

---

Source: https://arkoren.dev/docs/guides/first-steps/creating-a-project.md
[HTML documentation](https://arkoren.dev/docs/guides/first-steps/creating-a-project)

# Creating Projects

Use `arkoren new` when you want a new HTTP service with composition, probes, and
shutdown already connected. The result is ordinary Go code you own, not a project
that needs regeneration. For an existing application, [install individual
modules](https://arkoren.dev/docs/guides/first-steps/installation.md#existing-applications) instead.

## Installing Tools

With [Go 1.27 or later](https://arkoren.dev/docs/guides/first-steps/installation.md#go-requirements), install the CLI:

```sh
go install arkoren.dev/cmd/arkoren@latest
```

Go places it in `GOBIN`, if set, otherwise under the first `GOPATH` entry's
`bin` directory, usually `$HOME/go/bin`. Put that directory on `PATH`, then
check the available generation options:

```sh
arkoren new --help
```

Help and dry-run work offline. Creation needs no running application or MCP
endpoint configuration, but resolves Go dependencies unless you use `--skip-tidy`.
[CLI Installation](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren) also
offers module-tool installation; use `go tool arkoren` in place of `arkoren`
throughout this page if you choose it.

## Previewing Files

Choose a destination and the Go module path your application will keep. From
the destination's parent directory, preview a service named `hello`:

```sh
arkoren new hello --module example.com/hello --dry-run --verbose
```

The JSON report lists the absolute destination, module, planned files, and
conflicts. Without `--verbose`, it reports counts instead of full inventories.
Dry runs do not prompt, write files, or download dependencies. A conflicting plan
still returns a nonzero status.

The destination is required; use `.` explicitly for the current directory.
Without `--module`, the module path is the destination's basename. Invalid
names are rejected, not repaired, so set `--module` explicitly for shared projects.

## Writing Files

Use an unused or empty directory for the ordinary path. After inspecting the
preview, write the project:

```sh
arkoren new hello --module example.com/hello
```

Success produces one JSON report containing `created: true`, the absolute
destination, module, and file count. After writing, the CLI runs `go mod tidy`
inside the project with `GOWORK=off` to download dependencies and populate
`go.sum`. Generation initializes neither Git nor application infrastructure and
creates no application tests.

A nonempty destination, even one containing only `.git`, requires confirmation.
The default is no; only a complete `y` or `yes` line approves it. Without an
interactive terminal, explicitly approve that directory after reviewing its plan:

```sh
arkoren new hello --module example.com/hello --yes
```

This is an alternative invocation, not a second generation step for an already
created project. `--yes` never permits overwriting. Existing files or symlinks at generated paths,
unsafe parent paths, case-insensitive filename collisions, and planned
file/directory overlaps block generation before writes. Choose another
destination or resolve the conflict yourself; there is no automatic merge.

An I/O failure during writing can leave a partial project. Inspect the paths
reported by the error before retrying: files already created remain and become
conflicts. Do not delete an existing directory merely to clear a generation error.

## Resolving Dependencies

Enter the generated project:

```sh
cd hello
```

Dependencies are already resolved after successful creation. Check the code:

```sh
go vet ./...
```

For offline file creation, pass `--skip-tidy` to `arkoren new`, then run
`go mod tidy` from the project directory when downloads are available. If automatic
dependency setup fails, the CLI exits with an error and preserves the generated
files. Retry `go mod tidy` there instead of rerunning `arkoren new`.

The starter also declares
godotenv for local environment loading, Tint for terminal logs, and go-isatty for
terminal detection. They are application dependencies, not dependencies of core.
Commit the resolved `go.mod` and `go.sum`.

The generated Makefile is optional application tooling. `make` lists its targets;
`make check` verifies formatting, runs vet, and runs tests without modifying
dependencies. `make fmt` changes formatting. Race tests require a C compiler.
Continue to [Quickstart](https://arkoren.dev/docs/guides/first-steps/quickstart.md#first-request) to run the service and add
your first endpoint.

## Starter Defaults

`cmd/server/main.go` creates the application and calls
`app.Run(ctx, bootstrap.Compose)`. Files in `internal/bootstrap` construct the
router, optional diagnostics endpoint, and server. Typed factories return
constructed values; the application starts components only after composition
succeeds. The generated signal context requests shutdown on Ctrl+C or `SIGTERM`.

The router serves a welcome page at `/` and probes at `/livez`, `/readyz`, and
`/startupz`. It applies security headers, request telemetry, panic recovery, and
same-origin CSRF protection before registering routes. It grants no extra trusted
origins or CORS permissions. These defaults are not authentication, host validation,
proxy trust, or rate limiting. Keep GET, HEAD, and OPTIONS free of state changes.
HSTS requires an explicit HTTPS deployment policy.

The executable loads `.env` from its working directory without replacing existing
environment values; loading errors are ignored. The generated file binds
`HTTP_ADDR=127.0.0.1:8080` and sets `APP_DEBUG=true`. Empty or unset `HTTP_ADDR`
falls back to `:8080`, listening on all interfaces.

> [!WARNING]
> Debug mounts unauthenticated diagnostics at `/mcp` on the main listener.
> Anyone who can reach it can inspect logs and telemetry attributes. Keep local
> listeners on loopback and set `APP_DEBUG=false` in production.

Only the exact value `true` enables debug. Debug capture retains bounded recent
history in memory and loses it on exit; it needs no external telemetry backend.
See [Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md) for customization. If you
change the listener address, update `ARKOREN_MCP_URL` before using CLI diagnostics.

The generated `.env` is excluded from Git and created with restricted access
(POSIX mode `0600`, or a protected current-user DACL on Windows). Protect its
parent directory too. `.env.example` disables debug and can be committed. No
application secret or signing key is generated; provision credentials only when
the feature needs them.

The starter's logger uses Tint on terminals and JSON when stderr is redirected.
`NO_COLOR` disables terminal colors. Customize `Logger()` in
`internal/bootstrap/logging.go`, rather than assuming application environment
alone selects the starter's log format.

## Agent Files

By default, generation includes `AGENTS.md` and the bundled documentation skill
under `.agents/skills/arkoren/`, including its installation manifest. Commit them
if you use them. Agent discovery varies; configure your agent to read the
generated instructions when needed.

If your agent needs a different skills parent, choose it during generation
instead of the earlier invocation:

```sh
arkoren new hello --module example.com/hello --skills-dir .claude/skills
```

The CLI appends `arkoren` and updates the generated pointers. The parent must be
a project-relative slash-separated path using letters, digits, dots, underscores,
or hyphens, without `..` components. File/directory conflicts still block writes.

Alternatively, omit both instructions and the skill with `--no-agent-files`.
Do not combine that flag with a custom `--skills-dir`. Existing agent-file
collisions are not merged or overwritten.

Application files and `AGENTS.md` remain yours.
[`arkoren skills update`](https://arkoren.dev/docs/guides/agent-skills/working-with-agents.md#updating-skills)
manages only the skill directory and rejects modified managed files. There is
no application regeneration or upgrade step. Grow the code using
[Project Layout](https://arkoren.dev/docs/guides/first-steps/project-layout.md), and use [Deployment](https://arkoren.dev/docs/guides/first-steps/deploying.md) when
you are ready to build an image or binary.

---

Source: https://arkoren.dev/docs/guides/first-steps/quickstart.md
[HTML documentation](https://arkoren.dev/docs/guides/first-steps/quickstart)

# Quickstart

Build a greeting endpoint, count greetings in memory, and reject an oversized
name with an HTTP error. No database, broker, or credentials are needed.

Start with the `hello` service from [Creating Projects](https://arkoren.dev/docs/guides/first-steps/creating-a-project.md),
including its dependency-resolution step. Run every command below from that
project's root. Keep the generated bootstrap files: the edits extend them rather
than replacing their lifecycle, middleware, or probes.

## First Request

Start the service on loopback:

```sh
HTTP_ADDR=127.0.0.1:8080 go run ./cmd/server
```

The shell examples use POSIX environment assignments. In PowerShell, set
`$env:HTTP_ADDR = "127.0.0.1:8080"` before running `go run ./cmd/server`.
If port 8080 is occupied, choose another port and use it in the requests below.

The development `.env` enables unauthenticated diagnostics on the same listener.
Keep the loopback address; do not expose this development process publicly.
[Starter Defaults](https://arkoren.dev/docs/guides/first-steps/creating-a-project.md#starter-defaults) explains the settings.

In another terminal, request the welcome page and readiness probe:

```sh
curl -i http://127.0.0.1:8080/
curl -i http://127.0.0.1:8080/readyz
```

Both return `200 OK`; `/` contains an HTML welcome page and `/readyz` contains
`ok`. The server's startup log reports its bound address. If binding fails, the
process exits unsuccessfully rather than reporting a successful HTTP start.

Stop with Ctrl+C before editing. Repeat the start command after each edit below;
the starter does not watch or reload source files.

## Greeting Handler

An Arkoren HTTP handler takes the familiar response writer and request, then
returns an error. The router renders returned errors centrally, so handlers can
return failures without writing a second response.

Create `internal/bootstrap/greeting.go`. Start it with the package, imports,
and a typed response:

```go
package bootstrap

import (
	"net/http"

	"arkoren.dev/core/http/response"
)

// Greeting is the JSON returned for an accepted greeting.
type Greeting struct {
	Message string `json:"message"`
}
```

Append this handler to the same file. It reads a route wildcard named `name` and
returns JSON:

```go
// greet responds to a greeting request using the route's name.
func greet(w http.ResponseWriter, r *http.Request) error {
	name := r.PathValue("name")

	return response.JSON(w, http.StatusOK, Greeting{
		Message: "Welcome, " + name + "!",
	})
}
```

`response.JSON` sets the content type and encodes the response. Returning its error
keeps encoding or write failures visible to the router.

## Route Registration

In `internal/bootstrap/router.go`, inside the callback returned by `Router()`,
add this line before `return routes`:

```go
routes.Post("/greetings/{name}", greet)
```

Keep the existing middleware, welcome route, and probe routes. Registration maps
the method and path to the handler; it does not open a listener. Use POST because
the next step records a greeting and changes state.

Restart the service, then send a greeting:

```sh
curl -i -X POST http://127.0.0.1:8080/greetings/Mira
```

The response is `200 OK` with this JSON content:

```json
{
  "message": "Welcome, Mira!"
}
```

The starter's CSRF protection allows this command-line request without Origin
or Sec-Fetch-Site headers. That is not authentication or permission to expose
state-changing routes without an application security policy.

## Counting Greetings

Keep counters behind a cache facade so the handler does not depend on how values
are stored. In `internal/bootstrap/bootstrap.go`, add these imports alongside
the existing `application` import:

```go
"arkoren.dev/core/service/cache"
"arkoren.dev/core/service/cache/memorycache"
```

Inside `Compose`, replace only `routes := compose.Use(Router())` with:

```go
driver := memorycache.New()
compose.Cleanup(driver.Stop)
compose.Startup(driver.Start)
visits := cache.New(driver)

routes := compose.Use(Router(visits))
```

Keep the following `compose.UseDebug(MCP(routes))` and
`compose.Use(Server(routes))` calls. Cleanup is registered immediately at the
driver's owner and runs in reverse order, so the server stops before the cache.
Composition finishes before startup starts the cache's expiry janitor and binds
the HTTP listener. Failures during composition or startup also release acquired
resources.

In `internal/bootstrap/router.go`, import `arkoren.dev/core/service/cache` and
change the factory declaration to accept the facade:

```go
func Router(visits *cache.Cache) application.Factory[*router.Router] {
```

Keep the existing factory body and replace the greeting registration with this
closure, which passes the shared facade to the handler:

```go
routes.Post("/greetings/{name}", func(w http.ResponseWriter, r *http.Request) error {
	return greet(w, r, visits)
})
```

Add `net/http` to that file's imports. In `greeting.go`, add the `cache` import
and a `Count int64` field with the tag `json:"count"` to `Greeting`. Replace
the entire `greet` function with:

```go
// greet counts an accepted greeting and returns its current process-local total.
func greet(w http.ResponseWriter, r *http.Request, visits *cache.Cache) error {
	name := r.PathValue("name")
	key := cache.NewKey[int64]("greetings." + name)
	count, err := visits.Increment(r.Context(), key, 1)
	if err != nil {
		return err
	}

	return response.JSON(w, http.StatusOK, Greeting{
		Message: "Welcome, " + name + "!",
		Count:   count,
	})
}
```

The typed key names Mira's counter exactly `greetings.Mira`. `Increment` creates
a missing counter and updates it atomically; it does not use the key's write TTL.
These counters have no expiry, belong only to this process, and reset on restart.
They are demonstration state, not durable business records or an exactly-once
record of delivered responses. A write can succeed even if sending its response
later fails.

Restart, then issue the POST request twice. The second response contains:

```json
{
  "message": "Welcome, Mira!",
  "count": 2
}
```

For shared counters across replicas, follow [Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md)
for backend construction and ownership. The handler continues to use the facade.
Use [Key Definitions](https://arkoren.dev/docs/guides/cache/keys.md) and [Counters](https://arkoren.dev/docs/guides/cache/counters.md) when
you need reusable naming or bounded counter windows.

## Expected Errors

Reject an oversized name before changing its counter. Add
`arkoren.dev/foundation/problem` to `greeting.go`'s imports. Immediately after
`name := r.PathValue("name")`, insert:

```go
if len(name) > 32 {
	return problem.UnprocessableContent().Derive(func(details *problem.Builder) {
		details.Detail("The name must be at most 32 bytes.")
	})
}
```

`len` counts bytes, not Unicode characters. Restart and send 33 ASCII characters:

```sh
curl -i -X POST http://127.0.0.1:8080/greetings/abcdefghijklmnopqrstuvwxyzabcdefg
```

The response has status `422 Unprocessable Content` and a Problem Details body:

```json
{
  "type": "about:blank",
  "title": "Unprocessable Content",
  "status": 422,
  "detail": "The name must be at most 32 bytes.",
  "instance": "/greetings/abcdefghijklmnopqrstuvwxyzabcdefg"
}
```

No counter is incremented for this rejected request. The router can render an
error only before the response is committed; it cannot replace headers or body
bytes already sent. See [Error Handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md)
for private internal failures and custom problem responses.

## Stopping Safely

Press Ctrl+C in the service terminal. The generated executable's signal context
also handles `SIGTERM`. The application stops serving, runs registered cleanup,
and returns before the executable exits; clean shutdown exits with status 0,
while lifecycle failures exit with status 1.

The HTTP runner allows in-flight requests up to 15 seconds to drain by default.
After exit, another request fails to connect and the memory counters are gone.
This walkthrough creates no persistent backend or container to remove. Keep or
delete only the `hello` project you created; do not remove a shared parent directory.

You now have a working route, a shared dependency inside one process, and a
checked error path. Continue with [Configuration](https://arkoren.dev/docs/guides/first-steps/configuration.md) to translate
application settings, [Project Layout](https://arkoren.dev/docs/guides/first-steps/project-layout.md) to separate features,
or [Deployment](https://arkoren.dev/docs/guides/first-steps/deploying.md#shutdown-budgets) before running outside development.

---

Source: https://arkoren.dev/docs/guides/first-steps/configuration.md
[HTML documentation](https://arkoren.dev/docs/guides/first-steps/configuration)

# Configuration

Your application decides where settings come from; Arkoren components receive
the values they need when you construct them. Required dependencies and settings
are constructor arguments. Optional settings use a **builder callback**, a
function that edits a temporary construction draft.

Keep environment and file formats in your application rather than decoding them
into package-internal settings. This lets your deployment configuration stay
stable as you change components or drivers.

## Component Construction

With an existing HTTP handler named `handler`, construct a server using its
defaults. Import `arkoren.dev/core/http/server`:

```go
listener := server.New(handler)
```

The handler is a required dependency. Construction does not bind a socket;
`Start` opens the listener. The generated project registers that startup and
cleanup in `internal/bootstrap/server.go`.

To select a loopback address, change only that setting during construction:

```go
listener := server.New(handler, func(listener *server.Builder) {
	listener.Addr("127.0.0.1:9090")
})
```

These are alternative constructions, not consecutive declarations. The callback
changes the address without replacing the server's other defaults. Its parameter
is a `*server.Builder`, not a server you can reconfigure later. After construction,
changing a retained draft does not change the published component.

Required arguments can also be settings with no safe default, such as a validated
signing key or a database connection string. Their validation and provisioning
belong to the relevant [key](https://arkoren.dev/docs/guides/keys/overview.md) or
[driver guide](https://arkoren.dev/docs/guides/database/drivers.md), not an optional callback.

## Preserving Defaults

Omit settings you do not need to change. For example, the server's default header
read timeout is 10 seconds; changing its address leaves that timeout in place.
Its default address is `:8080`, which listens on all interfaces, not only loopback.
Defaults are not a substitute for application security policy.

Zero and empty values are package-specific. The server restores its default for
an empty address or zero timeout. Its negative write timeout disables the write
deadline; that exception is useful for streaming but should not be applied to
ordinary endpoints without a reason. Other builders can treat an empty value as
an explicit instruction to disable or clear a setting.

Do not copy every documented default into your callback or assume every
collection setting appends. Check the particular method's contract when replacing
lists, clearing values, or disabling protection.

## Shared Policies

When several listeners need the same header deadline, give that construction
policy a typed name. Import `time` for the duration:

```go
headers := server.BuilderFunc(func(listener *server.Builder) {
	listener.ReadHeaderTimeout(5 * time.Second)
})
```

Apply the policy, then add the settings specific to this listener:

```go
listener := server.New(handler, headers, func(listener *server.Builder) {
	listener.Addr("127.0.0.1:9090")
})
```

Callbacks run once, synchronously, in argument order on one fresh draft. A later
scalar setting replaces an earlier one. `server.BuilderFunc` is an alias for the
ordinary callback type, so it also works in slices passed with `policies...`.
Use the alias belonging to the constructor's role; a server policy cannot configure
a cache or queue consumer.

To omit optional policies, omit arguments or expand an empty slice. Supplying a
nil callback panics; it does not mean “use defaults.” Do not mutate the draft
concurrently with construction.

## Environment Settings

Read values beside the code that constructs their consumer. In the generated
`internal/bootstrap/server.go`, the existing factory callback imports `os` and
constructs its listener with:

```go
listener := server.New(handler, func(listener *server.Builder) {
	listener.Addr(os.Getenv("HTTP_ADDR"))
})
```

The factory reads `HTTP_ADDR` when composition invokes its callback, not when
`Server(handler)` returns the factory. Empty or unset values preserve `:8080`.
Later environment changes do not reconfigure that listener.

Core packages do not load your application environment or files. The starter's
executable separately imports `godotenv/autoload`, which reads `.env` during
initialization without overriding process environment values. Its loading errors
are ignored, so validate required settings explicitly rather than treating a
present `.env` as proof of valid configuration.

For example, if you add a required service credential, reject a missing value in
your application's loader before constructing the client. In a function returning
an error, with `errors` and `os` imported:

```go
token := os.Getenv("REPORTS_TOKEN")
if token == "" {
	return errors.New("REPORTS_TOKEN is required")
}
```

Do not log the token or include it in the error. This validation establishes
presence, not authorization or remote reachability. A driver's documented startup
or first operation establishes those later conditions.

## Typed Settings

Extract a loader when several components share values or parsing becomes
substantial. Keep its result application-owned. For example, a server loader
could return this type from your own `internal/settings` package:

```go
type HTTP struct {
	Addr            string
	ShutdownTimeout time.Duration
}
```

Import `time` in that package. Begin with a zero `ShutdownTimeout`, preserving the
server's default, and parse an override only when it is present. Inside a loader
returning `(HTTP, error)`, use `errors`, `os`, and `time`:

```go
cfg := HTTP{Addr: os.Getenv("HTTP_ADDR")}
if value := os.Getenv("HTTP_SHUTDOWN_TIMEOUT"); value != "" {
	timeout, err := time.ParseDuration(value)
	if err != nil || timeout <= 0 {
		return HTTP{}, errors.New("HTTP_SHUTDOWN_TIMEOUT must be a positive duration")
	}
	cfg.ShutdownTimeout = timeout
}
```

This application policy permits positive overrides such as `20s`; it rejects
malformed, zero, or negative input rather than silently changing shutdown behavior.
Return `cfg, nil` after any other application checks. The fragments show the
loader's type and parsing branch, not a complete loader implementation.

After checking the loader's error, translate its result into component settings
where you construct the server:

```go
listener := server.New(handler, func(listener *server.Builder) {
	listener.Addr(cfg.Addr)
	listener.ShutdownTimeout(cfg.ShutdownTimeout)
})
```

An absent override leaves the server's 15-second drain default intact. Keep your
JSON/YAML schema or environment variable names on `HTTP` and its loader, not on
an Arkoren reporting snapshot. Components that expose `Config()` do not thereby
accept that record as constructor input.

## Validation Boundaries

Check each constructor's contract rather than assuming every mistake becomes a
returned error at construction. Some constructors return configuration errors;
some middleware constructors panic for invalid code-authored policy; reusable
definitions can retain domain errors until registration or use. Network and TLS
failures can appear during startup. A successful constructor does not establish
that a backend is reachable.

Publication separates owned settings from the temporary draft, not every object
reachable from it. Clients, service pointers, functions, captures, and native TLS
configuration can remain shared. Finish their setup before concurrent use.
Before wrapping a native client with `NewFrom`, read the driver's ownership rules:
wrapping does not necessarily mean borrowing, and `Stop` can close that client.

For startup and cleanup ownership, continue with
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md). The
[`server` API reference](https://pkg.go.dev/arkoren.dev/core/http/server)
documents the defaults and builder methods used here.

---

Source: https://arkoren.dev/docs/guides/first-steps/project-layout.md
[HTML documentation](https://arkoren.dev/docs/guides/first-steps/project-layout)

# Project Layout

When one feature changes, its routes, rules, and persistence code should be easy
to find together. Group application behavior by feature and keep the code that
connects features in a visible composition root. Arkoren does not discover files
or require a particular directory layout.

A service with only the [Quickstart](https://arkoren.dev/docs/guides/first-steps/quickstart.md) endpoint can keep its handler
in `internal/bootstrap`. Extract a package when that behavior has enough rules,
tests, or dependencies to deserve its own boundary—not to fill empty directories.

## Growing Structure

For a greeting service that later adds a catalog, an application-owned layout
could be:

```text
hello/
├── cmd/server/main.go
├── internal/
│   ├── bootstrap/
│   │   ├── app.go
│   │   ├── bootstrap.go
│   │   ├── logging.go
│   │   ├── mcp.go
│   │   ├── router.go
│   │   └── server.go
│   ├── greeting/
│   │   ├── greeting.go
│   │   ├── http.go
│   │   └── http_test.go
│   └── catalog/
│       ├── catalog.go
│       ├── store.go
│       └── store_test.go
└── go.mod
```

`cmd/server` owns process entry, signals, and exit status. `bootstrap` constructs
dependencies, mounts features, and registers lifecycle work. `greeting` and
`catalog` own application behavior. `internal` keeps these implementation packages
behind Go's internal import boundary.

The generator creates only the executable and bootstrap, not these feature
packages or tests. Add files as the application needs them. Environment loading
can remain beside construction; introduce an application settings package only
when the parsing or sharing described in [Configuration](https://arkoren.dev/docs/guides/first-steps/configuration.md)
justifies one.

## Feature Ownership

A feature package owns related behavior from its incoming request to its data
operations. Put its HTTP adapters, business rules, queries, and tests together
rather than spreading them across global `handlers`, `models`, and
`repositories` folders.

That does not mean putting HTTP or SQL inside every rule. Keep calculations and
decisions in plain Go functions when they do not need I/O. Let adapters handle
request decoding and responses, and let a feature-owned store handle database
access. Dependencies should be explicit fields or arguments, not package globals.

If a feature adds jobs, events, or schema changes, keep those declarations with
the feature too. Application startup and deployment decide when to run them;
the presence of a file does not register a worker or apply a migration. Follow
[Migration Deployment](https://arkoren.dev/docs/guides/migrations/deployment.md) for schema ownership and
rollout ordering.

## Mounting Routes

To extract the Quickstart's endpoint, move its `Greeting` type and `greet` helper
into `internal/greeting`, changing their package declaration to `package greeting`.
In that package's `http.go`, import `net/http`,
`arkoren.dev/core/http/router`, and `arkoren.dev/core/service/cache`, then expose
one route-registration function:

```go
// Routes mounts greeting endpoints using the caller's cache.
func Routes(routes *router.Router, visits *cache.Cache) {
	routes.Post("/greetings/{name}", func(w http.ResponseWriter, r *http.Request) error {
		return greet(w, r, visits)
	})
}
```

This function registers handlers; it neither constructs a listener nor starts
or stops the cache. The handler keeps the request context for cache calls.
The feature borrows a facade whose lifetime the caller owns.

Inside the existing `Router(visits)` factory callback in
`internal/bootstrap/router.go`, replace the old greeting registration with:

```go
greeting.Routes(routes, visits)
```

Import your application's package, `example.com/hello/internal/greeting`, and
remove `net/http` from `router.go` if the old closure was its only use there.
Keep middleware registration before feature routes. Remove the original helper
file from `bootstrap` only after moving its implementation, so there is one
owner and one registration of the endpoint.

## Composition Root

The composition root is where concrete dependencies meet. In the starter,
`main` passes `bootstrap.Compose` to `app.Run`; the root constructs shared
dependencies and passes them into typed factories. The Quickstart's cache wiring
stays there even after the handler moves:

```go
driver := memorycache.New()
compose.Cleanup(driver.Stop)
compose.Startup(driver.Start)
visits := cache.New(driver)

routes := compose.Use(Router(visits))
```

This fragment uses the Quickstart's `application.Composer` callback and imports
for `cache` and `memorycache`. Keep diagnostics and listener selection after it.
The acquiring owner registers cleanup once, immediately after acquisition.
Do not add another cleanup hook in every feature that receives the facade.

Factories capture dependencies when obtained; their callbacks construct objects
and mount routes when selected through `compose.Use`. Returning a value does not
memoize it, transfer cleanup ownership, or start it. Construct a shared dependency
once and pass that same value to its consumers. Split growing bootstrap code into
typed factories rather than a global service lookup. See
[Application Composition](https://arkoren.dev/docs/guides/application/composition.md) for shared outputs and
conditional features.

## Feature Boundaries

When one feature needs an immediate answer from another, depend on the narrow
behavior it needs. For example, an application-owned checkout feature could
define this interface, with `context` imported:

```go
// Availability supplies the stock observation needed by checkout.
type Availability interface {
	Available(ctx context.Context, sku string) (bool, error)
}
```

Bootstrap can pass a catalog implementation that satisfies it without making
checkout import the whole catalog package. This observation alone does not
reserve stock; a checkout requiring an atomic reservation needs a contract that
expresses that operation instead.

If no immediate answer is needed, an [application event](https://arkoren.dev/docs/guides/events/overview.md)
can notify another feature after a change. The in-process event bus is not a
durable cross-process handoff; use a queue when work must survive that boundary.
If two packages continually require each other's details, reconsider whether
they represent one feature rather than adding more wiring to preserve the split.

## Testing Features

Test pure rules directly. Test HTTP behavior through the feature's registered
router using request/response recorders; a listening socket is unnecessary.
Supply local drivers or fakes for the contracts the feature consumes, then use
integration tests for the backend-specific behavior those substitutes cannot
establish. [Testing Overview](https://arkoren.dev/docs/guides/testing/overview.md) shows these boundaries.

Local infrastructure is optional application tooling, not a layout requirement.
If you add a Compose file, own its credentials, health checks, and teardown in
your project. Follow the chosen driver's setup rather than assuming every test
needs a container.

The result is still ordinary Go packages: move or combine them as the application
changes. Before deploying the assembled executable, review
[Deployment](https://arkoren.dev/docs/guides/first-steps/deploying.md) for runtime configuration, probes, and shutdown.

---

Source: https://arkoren.dev/docs/guides/first-steps/deploying.md
[HTML documentation](https://arkoren.dev/docs/guides/first-steps/deploying)

# Deployment

Deploy the application you have already tested, not a different production
bootstrap. The [generated service](https://arkoren.dev/docs/guides/first-steps/creating-a-project.md) connects server startup,
serving, cleanup, and signal handling; your platform supplies runtime settings,
credentials, traffic routing, and a termination grace period.

This page assumes that starter or equivalent `application.Application` wiring.
Keep driver-specific setup in its owning guide and decide
[migration rollout](https://arkoren.dev/docs/guides/migrations/deployment.md) before deploying schema changes.

## Binary Builds

From your application's module root, build the server executable:

```sh
CGO_ENABLED=0 go build -trimpath -o bin/server ./cmd/server
```

This targets the build machine's OS and architecture. Select `GOOS` and `GOARCH`
for a different target, or use the container build below. Core does not require
cgo, but confirm that every dependency you select supports this build mode.
CGO-dependent libraries need a suitable builder and runtime image instead.

Commit `go.mod` and `go.sum`, use a supported patched Go toolchain, and rebuild
when the Go runtime, standard library, or compiled dependencies need security
updates. Patching the host or replacing a runtime image does not patch code
already inside the binary. Test the resulting artifact on the target platform.

## Production Settings

Before starting the process, supply required application settings through your
platform's environment or your own validated settings loader. Core does not
automatically read environment variables. The starter explicitly reads
`HTTP_ADDR` and `APP_DEBUG`; vendor SDK credential chains can have additional
documented sources.

> [!WARNING]
> The starter's debug `/mcp` endpoint has no authentication and can reveal logs
> and telemetry attributes. Set `APP_DEBUG=false` explicitly in production.
> Selecting the production environment alone does not disable debug.

For a local check of the built binary, bind to loopback and disable debug:

```sh
APP_DEBUG=false HTTP_ADDR=127.0.0.1:8080 ./bin/server
```

The executable still autoloads `.env` from its working directory, without
overriding existing environment values. Do not depend on a development file for
production configuration. Explicit process values above override that file.
The platform must select an appropriate listener address: a container usually
needs `:8080` inside its network namespace, while a host-local reverse proxy can
use loopback. Empty `HTTP_ADDR` selects `:8080` on all interfaces.

Set the application environment in code if you want production metadata. In
`cmd/server/main.go`'s existing `run` helper, replace its construction statement
with the following, preserving signal setup and the call to `app.Run`:

```go
app := application.New(bootstrap.Application(), func(app *application.Builder) {
	app.Environment(application.Production)
})
```

The later callback overrides environment while retaining the starter's name and
logger policy. `APP_ENV` is not a built-in loader. The starter's custom logger
uses JSON when stderr is not a terminal and Tint otherwise; changing environment
does not replace that policy. Without a custom log handler, the application's
default selects source-enabled text for development and JSON otherwise.

Inject passwords, API tokens, and keys outside the image and source tree. The
application owner provisions them and grants the process least-privilege access,
whether through environment values, mounted files, SDK workload identity, or a
[secret store](https://arkoren.dev/docs/guides/secrets/overview.md). Validate required values before serving,
never log them, and do not generate a replacement key to repair a missing startup
secret. Rotation can require coordinated rollout or fallback keys; follow the
feature's credential rules.

## Container Builds

The generated Dockerfile already builds a stripped, cgo-disabled Linux binary
using BuildKit source mounts and module/compiler caches. Its runtime is
`gcr.io/distroless/static-debian13:nonroot`, with CA certificates and timezone
data, but no shell or Go toolchain. The generated `.dockerignore` excludes local
environment files from the build context.

With Docker available, build the image from the project root:

```sh
docker build -t hello:dev .
```

The build needs access to resolve modules unless its caches already contain them.
For a different Linux architecture, use Docker Buildx with the target `--platform`;
the generated build uses its target OS and architecture arguments. Pin base
images by digest when your release process requires reproducible image inputs.

Run a disposable local container, publishing only on host loopback:

```sh
docker run --rm --name hello-demo -p 127.0.0.1:8080:8080 \
  -e HTTP_ADDR=:8080 -e APP_DEBUG=false hello:dev
```

Check `http://127.0.0.1:8080/readyz` from another terminal. Stop the test container
with `docker stop hello-demo`; `--rm` removes it after exit. These commands
create no database volume. The equivalent starter shortcuts are
`make docker-build IMAGE=hello:dev` and `make docker-run IMAGE=hello:dev`.
Runtime credentials belong in the platform's secret mechanism, not Docker build
arguments, copied `.env` files, or committed image layers.

Configure HTTPS at your ingress or explicitly on the [HTTP server](https://arkoren.dev/docs/guides/http-networking/server.md).
The starter does not enable HSTS before you establish that HTTPS policy. It
also does not establish authentication, trusted proxy headers, host validation,
or cross-origin permissions merely by running in a container.

## Health Probes

The starter already mounts the application's three probe handlers. Configure
your platform to use their paths rather than adding duplicate registrations:

| Path | Meaning | Platform Decision |
| --- | --- | --- |
| `/startupz` | Startup has completed successfully. | Allow the startup phase to finish before applying ordinary health thresholds. |
| `/livez` | The application is running and no supervised runner has failed. | Restart an unhealthy process. |
| `/readyz` | The application is live and its latest background health pass is acceptable. | Send or withhold traffic without restarting the process. |

Plain probes return `200` with `ok` or `503` with `unavailable`. Startup stays
successful after completed startup, including while stopping; it is not a traffic
readiness signal. Liveness and readiness become unavailable during shutdown,
although an HTTP response is possible only while the listener can still serve it.

Only readiness depends on registered dependency checks. Inside an existing
composition callback, a driver with `Ping(context.Context) error` can supply one:

```go
compose.Check("catalog", driver.Ping)
```

The application runs checks in the background, every five seconds by default;
probe requests read the latest result instead of calling the backend. With checks
registered, readiness waits for the first successful pass and rejects results
older than three intervals. Without checks, readiness equals liveness. Callbacks
must honor their context deadline; a timeout cannot forcibly interrupt them.
See [Health and Readiness](https://arkoren.dev/docs/guides/application/health-and-readiness.md) for tuning.

Keep operator endpoints off public routes unless you add access control.
`InfoHandler`, verbose readiness (`?verbose`), and diagnostics can reveal internal
metadata. A second port or loopback bind is not authentication; enforce the
actual network and authorization boundary appropriate to your platform.

## Shutdown Budgets

Ensure the platform sends `SIGTERM` to the executable rather than only to a shell
wrapper. The starter calls `application.NewSignalContext`, defers its stop
function, and passes the context to `app.Run`. Keep that signal cleanup even
when startup fails. Lifecycle errors are reported before telemetry finalization;
the executable uses the returned error for status 1 rather than logging it again.

On cancellation, the server runner stops accepting connections and drains
in-flight requests. After runners return, cleanup runs in reverse registration
order. Register backend cleanup before listener cleanup so the backend remains
available while requests finish.

The HTTP drain defaults to 15 seconds. To change it, add a setting inside the
existing constructor callback in `internal/bootstrap/server.go`, with `time`
imported:

```go
listener.ShutdownTimeout(20 * time.Second)
```

The application's runner-wait and shutdown budget defaults to 30 seconds. Change
it in the callback returned by `Application()` in `internal/bootstrap/app.go`:

```go
app.ShutdownTimeout(45 * time.Second)
```

These lines edit existing construction callbacks; they are not runtime setters.
The HTTP drain must fit within the application budget. Cleanup receives at least
a fresh ten-second context when the preceding budget is exhausted. Configured
telemetry finalization runs last with at least another ten seconds. Allow room
for these phases and platform scheduling overhead, not just the application
timeout, when setting Kubernetes `terminationGracePeriodSeconds` or an equivalent
platform grace period. Cleanup callbacks must cooperate with their deadlines.

Long-lived handlers should observe `server.ShuttingDown(r.Context())` and return
when draining begins. Drain timeout closes tracked HTTP connections; it does not
join hijacked connections or forcibly stop arbitrary handler goroutines. Their
owners must coordinate completion before dependencies are released. See
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md).

## Rollout Checks

Before directing production traffic to a release:

- Verify the built artifact, environment, debug setting, and log format on the
  target platform rather than relying only on a successful local build.
- Apply schema changes using the reviewed [migration deployment](https://arkoren.dev/docs/guides/migrations/deployment.md)
  process; driver installation and startup do not create tables automatically.
- Confirm startup and readiness succeed, and that a dependency failure removes
  readiness without turning the dependency into a liveness restart condition.
- Send the platform's termination signal under representative load and verify
  requests drain, drivers stop, and the process exits within the allotted grace.
- Check that secrets are absent from artifacts and logs, and that diagnostics
  and operator details are inaccessible to unauthorized clients.

For the lifecycle and listener contracts behind these checks, see the
[`application` API reference](https://pkg.go.dev/arkoren.dev/core/application)
and the [`server` API reference](https://pkg.go.dev/arkoren.dev/core/http/server).

---

Source: https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers)

# Facades and Drivers

A report handler should not need Redis commands to cache a report's status.
Arkoren separates the service your code calls from the backend that stores the
data. You select the backend when constructing the service, then pass the
application-facing object to its consumers.

## Three Roles

A **facade** is the application-facing service, such as `*cache.Cache`. It adds
shared behavior around backend operations: the cache facade encodes Go values
as JSON, decodes reads, and records telemetry when enabled.

A **driver** performs the backend operations. For example, `memorycache` stores
bytes in the current process, while `rediscache` sends commands to Redis. A
[contract](https://arkoren.dev/docs/guides/core-concepts/contracts.md) defines the interface and behavioral rules between the
facade and driver, including expiry, copying, atomicity, and errors.

```text
Your handler -> Cache facade -> Cache driver -> Stored bytes
               typed values    raw operations
```

This separation does not make every backend equivalent. Portable service calls
can remain unchanged when you replace a driver; connectivity, supported
operations, sharing, and durability still need review.

## Typed Service Calls

For a process-local cache, import `arkoren.dev/core/service/cache` and
`arkoren.dev/core/service/cache/memorycache`, then construct the facade:

```go
driver := memorycache.New()
store := cache.New(driver)
```

The facade uses that driver; constructing another facade does not create a
separate backend unless you also supply a separate driver. Pass `store` to the
handlers or services that need caching, rather than making them select drivers.

To cache a report status, define the value your application wants to read:

```go
type ReportStatus struct {
	Ready bool `json:"ready"`
}
```

A `cache.Key[T]` associates a physical entry name with a Go value type and write
policy. Import `time` and give this entry a ten-minute lifetime:

```go
key := cache.NewKey[ReportStatus]("reports:42:status", func(entry *cache.KeyBuilder) {
	entry.TTL(10 * time.Minute)
})
```

With the operation's `ctx context.Context`, write the value through the facade:

```go
err := store.Put(ctx, key, ReportStatus{Ready: true})
```

On success, the driver holds JSON bytes with the configured expiry. A key with
no TTL setting has no expiry; reading it does not renew its lifetime.

The same key determines the read's result type:

```go
status, err := store.Get(ctx, key)
```

On success, `status` is a `ReportStatus`. Missing and expired entries return an
error matching `cache.ErrNotFound`; malformed stored JSON returns a decoding
error. The Go type does not create a separate storage namespace: two keys with
the same name address the same entry even if their type arguments differ. Keep
the type consistent for each name; [Cache Keys](https://arkoren.dev/docs/guides/cache/keys.md) explains key
identity and shared prefixes.

## Backend Changes

To move this cache to Redis, use the installation, connection, and lifecycle
instructions in [Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md). The composition boundary
supplies a `rediscache.Driver` instead of a `memorycache.Driver`; the `Put` and
`Get` calls above stay the same.

An in-memory cache belongs to one process and loses its contents on restart.
Redis can provide a shared backend, but its persistence depends on your Redis
deployment. Choosing a driver does not migrate existing entries.

Construction is not a connectivity check. The Redis cache constructor performs
no network I/O; `Start` checks connectivity, and `Stop` closes its client. Other
drivers have different lifecycle requirements. Register only the methods the
selected driver supplies, under one owner, using
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md).

Several facades can coexist. For example, a service can use one cache for
sessions and another for report status. The cache facade's `Name` setting labels
telemetry; it does not isolate entries. Use separate backend configuration or
explicit key prefixes when you need separate data.

## Separate Roles

Some services separate the sending and receiving sides of their contract.
Queues have `DispatcherDriver` and `ConsumerDriver`; broadcasting has
`PublisherDriver` and `ConsumerDriver`. Telemetry has separate trace, metric,
and log driver interfaces.

A web process that only dispatches jobs needs the dispatcher role, not a running
consumer. A worker needs the consumer role. One concrete driver can implement
both, but the service guide determines which facade and lifecycle each process
needs.

## Unsupported Operations

A method's presence does not establish backend support. Contracts can specify
an `ErrUnsupported` result when a backend cannot provide the required semantics.
Read the service's driver comparison before depending on such an operation.

For example, `cache.Driver.IncrementWithTTL` must update the counter and expiry
atomically. A backend that cannot do that returns `cache.ErrUnsupported`; it
must not substitute separate reads and writes. The rate limiter propagates
that failure rather than counting unsafely. Use `errors.Is` to classify a
documented sentinel, and do not weaken an atomic operation as a fallback.

## Native Client Access

If a required operation exists only in the vendor API, keep that dependency in
a narrow backend-specific part of your application. With a Redis-backed `store`,
import `fmt` and `arkoren.dev/driver/redis/rediscache` to print the native pool's
current connection count:

```go
if backend, ok := store.DriverAs[*rediscache.Driver](); ok {
	fmt.Println(backend.Client().PoolStats().TotalConns)
}
```

`DriverAs` finds the requested concrete type, including through wrappers that
expose `Unwrap() Driver`. It returns `false` when no match exists. Direct driver
and native-client calls bypass facade behavior, including its telemetry, and
must be reviewed when changing backends.

If you already have a native client, check the driver's `NewFrom` ownership
contract before wrapping it. Redis cache's `NewFrom` transfers ownership:
`Stop` closes the supplied client too. Other drivers can borrow clients instead.
Never infer ownership from the constructor name or close a borrowed client
while another component still needs it.

For the facade and accessors used here, see the
[`cache` API reference](https://pkg.go.dev/arkoren.dev/core/service/cache).

---

Source: https://arkoren.dev/docs/guides/core-concepts/contracts.md
[HTML documentation](https://arkoren.dev/docs/guides/core-concepts/contracts)

# Contracts

Two caches can expose the same `Get` method yet disagree about expiry, returned
bytes, or missing entries. A **contract** combines a service's driver interface
with its documented behavioral rules. Those rules tell you what application
code can rely on when selecting or replacing a driver.

## Behavioral Guarantees

Read the method documentation as well as its signature. The
[`cache.Driver` contract](https://pkg.go.dev/arkoren.dev/core/service/cache#Driver),
for example, establishes that:

- Zero write TTL means no expiry; a negative TTL fails with `ErrInvalidTTL`.
- Expired entries behave like missing entries.
- `Get` reports a missing entry with `ErrNotFound`, while `Has` returns `false`
  and deleting a missing entry succeeds.
- Stored and returned byte slices are independent copies, so caller mutation
  does not change cached data.
- An already-canceled context prevents the operation and leaves the cache
  unchanged.

These are distinct promises. A successful read does not imply that an entry
never expires, and a successful deletion does not imply that an entry existed.
The contract gives each result a meaning your code can use.

The [facade](https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers.md) adds its own documented behavior. Cache
`Get` decodes driver bytes into the key's Go type, so it can fail on invalid
JSON even after the driver successfully finds an entry. That is not a cache
miss; the stored entry remains unchanged.

## Error Identity

Classify documented conditions with `errors.Is`, not an error string. Drivers
can wrap a sentinel with additional context without changing its identity.
Sentinels belong to their service: a cache miss and a missing database row do
not share one universal not-found error.

Suppose `store` is an existing `*cache.Cache`, `key` is a `cache.Key[ReportStatus]`
as in [Typed Service Calls](https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers.md#typed-service-calls), and `ctx`
is the operation context. Import `errors` and
`arkoren.dev/core/service/cache` to distinguish absence from a failed read:

```go
value, err := store.Get(ctx, key)
switch {
case errors.Is(err, cache.ErrNotFound):
	// Load from the authoritative store instead.
case err != nil:
	return err
default:
	// Use the cached value.
}
```

Only the missing-entry branch establishes that the cache has no value for this
key. A connection failure or decoding error must not silently become a cache
miss. Decide explicitly whether your application can continue without caching.

Contracts do not give every possible backend failure a portable sentinel.
Preserve unexpected errors for the caller or operational reporting; do not
invent a classification from vendor error text.

## Context Boundaries

Pass the request or job's context to I/O operations so cancellation and
deadlines can reach the driver. Use `errors.Is(err, context.Canceled)` or
`errors.Is(err, context.DeadlineExceeded)` when cancellation affects your next
decision. Local hashing and encryption APIs do not take a context merely to
match this shape.

Distinguish an already-canceled operation from cancellation during a write.
The cache contract guarantees no mutation for the former. Once a command has
reached a remote backend, an error does not necessarily establish whether the
backend applied it. For example, cache `CompareAndPut` explicitly documents
that a network error can leave its outcome unknown. A deadline is not a
rollback guarantee.

## Atomic Operations

An **atomic operation** makes a decision and its associated change indivisible
to competing callers. Cache `Add`, for example, stores a value only if its key
is absent; a rejected addition leaves the existing value and TTL untouched.
Calling `Has` followed by `Put` is not equivalent: another caller can write
between the two calls.

When a contract permits `ErrUnsupported`, a driver unable to meet the required
semantics returns that error instead of an unsafe approximation. Do not
replace an unsupported atomic call with separate operations. Select another
backend or redesign the operation, as explained in
[Unsupported Operations](https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers.md#unsupported-operations).

The portable contract is not a catalog of everything a vendor provides. Native
features accessed through a vendor client sit outside those portable guarantees.
Likewise, a shared API does not promise identical durability, deployment
requirements, or performance across drivers.

## Conformance Evidence

A **conformance suite** exercises a driver's behavior against its contract.
When writing or adapting a cache driver, call the public `cachetest.Run` suite
with a driver factory. In a `_test.go` file, import `testing`,
`arkoren.dev/core/service/cache`, and its `cachetest` and `memorycache`
subpackages:

```go
func TestCacheContract(t *testing.T) {
	cachetest.Run(t, func(t *testing.T) cache.Driver {
		return memorycache.New()
	})
}
```

The factory is called for each subtest. The cache suite uses unique keys, so an
external backend need not start empty. It checks expiry, byte copying,
conditional writes, counters, cancellation, and concurrent access. Server-owned
expiry needs the suite's `WithExternalClock` option; local expiry can use
simulated time.

Passing the suite is evidence for the tested behavior, not proof about every
production configuration or failure mode. The interface documentation remains
binding even when a rule has no test. Follow
[Writing a Driver](https://arkoren.dev/docs/guides/drivers/writing-a-driver.md) when implementing a backend,
or [Conformance Suites](https://arkoren.dev/docs/guides/testing/conformance-suites.md) when integrating its
tests.

## Application Tests

A **fake** records requested effects so an application test can inspect them
without a production provider. For example, a queue fake can establish that
your code dispatched the expected job through the real facade. It cannot prove
that a production broker persists or redelivers that job.

An in-process driver serves a different purpose: it executes the service's
behavior locally. Use the service's testing guide to choose between recording
an effect and executing it. The workflow fake records commands and accepts
scripted outcomes without executing workflow bodies. The scheduling fake
records work and can explicitly drain due jobs through real handlers; it does
not run timers on its own. Use memory drivers when you need workflow execution
or automatic timer behavior. [Fakes](https://arkoren.dev/docs/guides/testing/fakes.md) describes those
boundaries without requiring backend infrastructure.

For the concrete guarantees used throughout this page, see the
[`cache` API reference](https://pkg.go.dev/arkoren.dev/core/service/cache).

---

Source: https://arkoren.dev/docs/guides/core-concepts/arks.md
[HTML documentation](https://arkoren.dev/docs/guides/core-concepts/arks)

# Arks

Sending an email is a service operation. Deciding that a completed report needs
an email, who receives it, and what the message means is domain behavior. An
**ark** is an optional, reusable domain feature that combines Arkoren services
with those concepts and rules.

## Current Availability

Arkoren has no built-in arks yet. The feature below illustrates the boundary;
it is not an installable integration or a promised API.

An ark is an ordinary Go module that an application explicitly chooses and
connects. It is not an automatically discovered plugin. You can use Arkoren's
services directly in application-owned features without introducing an ark.

## Domain Boundaries

A [facade](https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers.md) exposes a service operation, such as storing
a file or submitting mail. Its driver performs the backend work. Neither needs
to decide whether a report is ready for delivery or which recipients are
entitled to receive it.

An ark adds that domain meaning. A reusable report-delivery feature might
define a delivery request, track its status, and prepare notification content.
It could use storage, queues, and mail without making those services aware of
reports.

| Responsibility | Owner | Report Example |
| --- | --- | --- |
| Portable service behavior | Facade and contract | Submit a message through the mail service. |
| Backend communication | Driver | Submit that message to a particular mail provider. |
| Reusable domain rules | Ark | Notify recipients when a report becomes ready. |
| Deployment and application policy | Application | Choose the provider and authorize delivery requests. |

The distinction is the meaning of the behavior, not its size. A component does
not become an ark merely because it uses several services. Conversely, a small
feature can be domain-specific if it defines what a report, membership, or
subscription means.

## Feature Composition

In the hypothetical report feature, your application supplies configured
services and connects the feature to its entry points:

```text
Application: authorizes delivery and supplies configured services
  -> Report feature: tracks readiness and prepares notification content
       -> Storage facade -> Selected storage driver
       -> Mail facade    -> Selected mail driver
```

The feature owns its domain rules. The application still chooses backends,
credentials, exposure, and resource ownership. When a feature needs tables,
routes, or background work, those are explicit integration responsibilities,
not a reason to hide construction or startup inside ordinary service calls.

Using portable [contracts](https://arkoren.dev/docs/guides/core-concepts/contracts.md) lets a domain feature work through
selected service implementations. It does not remove provider constraints:
an in-memory backend still loses state on restart, and unsupported operations
still need an explicit decision.

For your own features, use typed dependencies and explicit registration as
described in [Application Composition](https://arkoren.dev/docs/guides/application/composition.md). That
composition pattern applies whether the feature lives in your application or
is eventually shared as a separate module.

---

Source: https://arkoren.dev/docs/guides/core-concepts/common-pitfalls.md
[HTML documentation](https://arkoren.dev/docs/guides/core-concepts/common-pitfalls)

# Common Pitfalls

When a service starts but behaves differently than expected, first check the
boundary involved: who owns the resource, when configuration takes effect, and
which guarantees the selected backend provides. These mistakes can leave code
that compiles but does not do what its caller assumes.

## Resource Ownership

### Late Cleanup

If composition acquires a resource and then fails while constructing a
consumer, cleanup registered afterward is never reached. Register release as
soon as you own the resource, before the next fallible operation.

Application cleanup becomes active immediately and runs in reverse registration
order, even if startup never runs. A registered stop function must therefore
tolerate an unstarted resource. Keep that ownership in one place using
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md), rather
than duplicating lifecycle wiring in every feature.

### Unchecked Connectivity

A constructed driver is not necessarily connected. For example, Redis cache's
constructor creates its client without network I/O; `Start` checks connectivity.
Omitting startup moves that failure to the first operation instead of rejecting
startup.

Follow the selected driver's setup, including its available startup, cleanup,
and readiness methods. Do not assume every driver has all three: some are
stateless, and some memory drivers need no startup. Register lifecycle work
with either your existing host/container or Arkoren's application, not both.

### Broad Dependencies

Passing `*application.Application` into business services gives them host
control and runtime state they do not need. Passing a composer also permits
hidden lifecycle registration. Keep those objects at the composition boundary;
pass typed services, a logger, or an event bus explicitly.

An event subscription is also an owned resource. Register its unsubscribe
function for release instead of leaving listeners attached across repeated
composition or tests; [Own Subscriptions](https://arkoren.dev/docs/guides/application/composition.md#own-subscriptions)
shows the focused pattern.

## Construction Policies

### Override Order

Construction callbacks run in argument order on one draft. When a setting
replaces an earlier value, put shared defaults first and the local override
last. With an existing `cache.Driver` named `driver` and a shared
`cache.BuilderFunc` named `defaults`, import `arkoren.dev/core/service/cache`:

```go
store := cache.New(driver, defaults, func(store *cache.Builder) {
	store.Name("reports")
})
```

The final name is `reports`; reversing the callbacks lets `defaults` replace
it. This is construction-time configuration, not a runtime settings API.
Changing a retained draft cannot reconfigure the published facade. See
[Configuration](https://arkoren.dev/docs/guides/first-steps/configuration.md#shared-policies) for reusable
policies and validation boundaries.

### Missing Observations

Registering an event listener does not connect every facade to that bus.
Components that expose an `Events` setting publish only when given a bus; a
database listener on the application's bus sees nothing if the database facade
uses no bus. Application lifecycle events are separate from facade events.

Likewise, a facade with no explicit telemetry follows `telemetry.Default()`;
the initial process default records nothing. Setting a default does not start
its exporters. Check both the component's construction and the selected
telemetry's lifecycle before diagnosing a missing metric. Use
[Observe Outcomes](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md#observe-outcomes)
for lifecycle reporting and [Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md)
for diagnostic capture. A failed diagnostic-tool declaration does not
necessarily undo capture attachment.

## HTTP Boundaries

### Middleware Registration

Router middleware is applied when routes are registered. Calling `Use` after
registering a route through that router or one derived from it panics; existing
routes would otherwise silently miss the new policy. Add shared middleware
before routes, and choose a group or `With` for narrower policies as described
in [Choose Scope](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#choose-scope).

### Explicit Recovery

`router.New` does not install panic recovery. For a buffered handler, install
recovery before registration so an ordinary panic can become a `500` Problem
Details response. Import `arkoren.dev/core/http/router`; `status` below is your
existing `router.Handler`:

```go
routes := router.New()
routes.Use(router.Recover())
routes.Get("/status", status)
```

Recovery protects only the middleware and handlers inside it. Keep request
logging outside recovery so it observes the converted error; [Order Policies](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#order-policies)
explains the order. Once a response is committed, recovery cannot replace bytes
already sent with a fresh error document.

### Hidden Errors

Writing an error-shaped JSON body and returning `nil` tells outer middleware
that the handler succeeded. It also bypasses central Problem Details rendering
and its request-instance handling. Return the error instead of manually
writing its representation, using [Error Handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md).
Streams have a separate commitment boundary; follow
[Streaming Responses](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md) rather than
trying to render a second response after streaming starts.

### Optional Zero Values

Go-value validation treats scalar `0`, `""`, and `false` as absent. Ordinary
range and format rules skip absent values, so an integer schema with only
`Min(1)` accepts zero. Add `Required` when zero means missing. If zero is a
real supplied value that must be checked, use a pointer schema so a non-nil
pointer is present and its pointee is validated.

Decoding alone does not apply these rules. Follow
[Validate Input](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#validate-input) to run the
schema at the request boundary, and [Validation](https://arkoren.dev/docs/guides/validation/overview.md)
for presence semantics. Exact encoded JSON validation has different presence
rules from validation of Go zero values.

## Data Assumptions

### Dialect Selection

`query.Query.Build()` uses ANSI SQL with `?` placeholders. Sending that SQL
directly to PostgreSQL does not select PostgreSQL's `$1` placeholders; MySQL
also has different identifier quoting. Pass published queries to the database
facade so it compiles for the configured driver's dialect. If you need SQL
text, use `BuildFor` with the intended dialect, as shown in
[Query Builder](https://arkoren.dev/docs/guides/queries/query-builder.md).

### Stable Pagination

Cursor pagination needs a total order. Ordering only by a non-unique timestamp
can skip or repeat rows at page boundaries. For a `reports` table with a unique
`id`, import `arkoren.dev/foundation/query` and end the order with that key:

```go
reports := query.Table("reports", func(statement *query.Builder) {
	statement.OrderByDesc("created_at")
	statement.OrderByDesc("id")
})
```

Rows sharing `created_at` now have a deterministic tie-breaker. Use
[Pagination](https://arkoren.dev/docs/guides/queries/pagination.md) for cursor construction and
[Batching and Streaming](https://arkoren.dev/docs/guides/database/batching-and-streaming.md) for traversing
larger result sets; ordering alone does not choose a traversal strategy.

## Deployment Limits

### Process-Local State

Memory drivers are useful for tests, development, and intentionally ephemeral
state. Separate replicas do not share their contents, and a restart loses
them. Before deploying state that must survive or be shared, select a suitable
backend and verify its persistence configuration using
[Choosing a Driver](https://arkoren.dev/docs/guides/drivers/choosing-a-driver.md).

### Operator Exposure

`InfoHandler`, verbose readiness, and diagnostic tools can reveal internal
metadata. They do not become protected merely because they are mounted on a
second port. Add an actual network or authorization boundary before exposing
them. [Mount Probes](https://arkoren.dev/docs/guides/application/health-and-readiness.md#mount-probes)
distinguishes minimal probes from detailed reports.

### Shutdown Budgets

If the platform kills a process before draining and cleanup finish, connections
and in-flight work can be cut off. Keep the HTTP drain timeout inside the
application budget and allow additional platform time for cleanup and telemetry
finalization. Follow [Shutdown Budgets](https://arkoren.dev/docs/guides/first-steps/deploying.md#shutdown-budgets)
instead of copying one timeout to every layer.

Shutdown does not automatically cancel request contexts already in progress.
A long poll or server-sent event handler must also observe
`server.ShuttingDown(ctx)` and return cooperatively; otherwise it can hold the
drain open until the timeout. [Streaming Responses](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md)
shows that cancellation boundary.

---

Source: https://arkoren.dev/docs/guides/application/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/application/overview)

# Application Overview

An HTTP service often needs its database before accepting requests and must keep
that database open until requests finish. The `application` package coordinates
these dependencies across startup, running work, and shutdown. It does not
construct dependencies for you or make them available through a lookup container.

## Choose Coordination

Use an application when several components need one startup order, one owner for
cleanup, or shared supervision. If your existing host already manages those
responsibilities, keep using it: Arkoren's router, server, and service packages
work independently. You can [run an HTTP server directly](https://arkoren.dev/docs/guides/http-networking/server.md)
or [adopt individual packages](https://arkoren.dev/docs/guides/first-steps/installation.md#existing-applications)
without adding an application.

To create an idle coordinator, import `arkoren.dev/core/application`:

```go
app := application.New(func(app *application.Builder) {
	app.Name("inventory")
})
```

This resolves construction settings but starts no resources or listeners.
Choose [component settings](https://arkoren.dev/docs/guides/first-steps/configuration.md#component-construction)
when constructing the application; its builder is not a runtime control handle.

## Understand Phases

A **composer** is the temporary scope passed to your composition callback.
Composition constructs concrete dependencies, passes typed values to their
consumers, and registers work. Every composition callback finishes synchronously,
in argument order, before any startup callback runs.

Inside composition, `Startup` registers initialization, `Run` registers supervised
running work, and `Cleanup` registers resource release. These calls register
callbacks; they do not execute them. Startup callbacks run in registration order.
Only after all succeed do runners and background health checks launch. A
constructed dependency is therefore not necessarily connected or ready to use.

Application `Run` combines startup, waiting for runners or caller cancellation,
and shutdown. Shutdown cancels runners, waits within its budget, then releases
resources in reverse cleanup-registration order. Register cleanup immediately
when ownership begins: it also runs if later construction or startup fails,
including for components whose startup never ran.

[Composition](https://arkoren.dev/docs/guides/application/composition.md) shows how to construct and share typed values.
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md) explains rollback,
supervision, and cooperative shutdown. A successful startup does not establish
continuing availability; [Health and Readiness](https://arkoren.dev/docs/guides/application/health-and-readiness.md) keeps
that separate from lifecycle state.

## Share Handles

Services that need logging, telemetry, or events receive those specific handles,
not the application. At the top of a composition callback, bind only the handles
it uses, then pass them to constructors. The handles are stable: a logger obtained
before capture is attached still reaches that capture after startup. Keep the
composer out of request handlers and background goroutines. Its context belongs
to startup; runners use their supplied context instead.

The logger adds trace and span IDs from a valid context even before startup;
use context-aware methods such as `InfoContext`. Correlation does not enable
recording. Application telemetry records nothing without an explicit
[telemetry configuration](https://arkoren.dev/docs/guides/observability/telemetry.md) or
[diagnostic capture](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md). Recording starts after
composition, and telemetry finalizes after resource cleanup and final lifecycle
events. The application owns that lifecycle.

## Choose Handler Exposure

If the service needs a temporary public home page, mount the welcome handler on
your existing `routes *router.Router` (`arkoren.dev/core/http/router`):

```go
routes.Get("/", app.WelcomeHandler(), application.WelcomeOperation)
```

The exact-root route serves self-contained HTML containing the escaped application
name and Arkoren resource links. Choose a name suitable for public display; an
empty name uses a generic heading. It fetches no external assets, needs no
JavaScript, and exposes no runtime, configuration, or build details. Replace it
when you add your own home page; it is not a health probe. For a first request
through a running service, follow the [quickstart](https://arkoren.dev/docs/guides/first-steps/quickstart.md#first-request).

Operator information is different: `InfoHandler` exposes name, version,
environment, lifecycle state, log level, start time, uptime, and binary build
details. Restrict network access or add authentication and authorization before
mounting it. On an existing operator-only router:

```go
operatorRoutes.Get(application.InfoPath, app.InfoHandler(), application.InfoOperation)
```

`app.Info()` returns the same snapshot inside the process. Neither handler is
mounted automatically or controlled by debug mode. Operation annotations document
the registered routes; they do not protect them. Follow
[diagnostic endpoint protection](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md) before remote
exposure, and use dedicated [probes](https://arkoren.dev/docs/guides/application/health-and-readiness.md#mount-probes) for
traffic and restart decisions.

Focused fragments in these guides keep error results visible but omit routine
propagation. Checks that determine ownership or another consequential decision
remain explicit; complete application code must handle its errors.

For construction and runtime contracts, see the
[`application` API reference](https://pkg.go.dev/arkoren.dev/core/application).

---

Source: https://arkoren.dev/docs/guides/application/composition.md
[HTML documentation](https://arkoren.dev/docs/guides/application/composition)

# Application Composition

When catalog and stock handlers use the same database, construct one facade and
pass it to both. Composition is the synchronous code that connects these typed
values and registers their lifecycle; it is not a container that resolves
dependencies later. This page assumes an [application coordinator](https://arkoren.dev/docs/guides/application/overview.md)
and an existing router. Startup execution and resource release belong to
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md).

## Share Typed Dependencies

Inside your composition callback, `composer` is a `*application.Composer` from
`arkoren.dev/core/application`. Assume `dsn` is your configured PostgreSQL
connection string and `routes` is a `*router.Router` from
`arkoren.dev/core/http/router`. Import `database` from
`arkoren.dev/core/service/database` and `postgresdatabase` from
`arkoren.dev/driver/postgres/postgresdatabase`.

Construct the driver and immediately register its release, before any later
construction can fail:

```go
connection := composer.Must(postgresdatabase.New(dsn))
composer.Cleanup(connection.Stop)
composer.Startup(connection.Start)

db := database.New(connection)
```

`New` validates configuration and creates a lazy pool without network I/O.
`Start` verifies connectivity later. The facade does not start or stop its driver;
the acquiring composition code owns that resource. Do not issue queries during
composition when connectivity has not been established.

Your application-owned `mountCatalog` and `mountStock` functions each accept
`(*router.Router, *database.Database)`. They register handlers that borrow the
facade without managing its lifecycle:

```go
mountCatalog(routes, db)
mountStock(routes, db)
```

Both receive the same concrete facade. Their signatures make the dependency
visible to the compiler without tying handlers to PostgreSQL. Returning or
passing a value transfers no cleanup ownership.

## Stop Failed Construction

`Must` unwraps a successful `(value, error)` result. On error it aborts composition:
later construction and every startup callback are skipped. Earlier registered
cleanup runs before `Start` or `Run` returns the original error, joined with
cleanup or telemetry-finalization errors if needed. Use `errors.Is` or `errors.As`
to classify the returned cause.

This ordering matters when a server's startup registration precedes another
constructor: an invalid later constructor fails first, because no startup
callback runs until all composition succeeds. Cleanup follows registration
position, not the number of successful startup callbacks.

For an operation returning only an error, use `Require`. With an existing
`migrator *migrate.Migrator` and `definitions []migrate.Definition`, both from
`arkoren.dev/core/service/database/migrate`, validate registration during composition:

```go
composer.Require(migrator.Add(definitions...))
```

Invalid or duplicate definitions abort construction; `Add` executes no SQL.
[Migrations](https://arkoren.dev/docs/guides/migrations/overview.md) owns execution and deployment policy.
Use `composer.Context()` if synchronous construction needs the startup context.

`Must`, `MustDefer`, and `Require` are valid only during synchronous composition,
even for successful results. Return ordinary errors from startup, runners, and
cleanup. Do not recover their private abort: the first failure stays recorded,
and a factory cannot publish a result after swallowing it. Unrelated composition
panics remain programmer failures and re-panic unchanged after cleanup.

## Extract Owned Construction

If multiple executables need the same database setup, extract construction and
its lifecycle registrations together. A `Factory[T]` is a defined callback type
that publishes one typed result:

```go
func InventoryDatabase(dsn string) application.Factory[*database.Database] {
	return func(composer *application.Composer) *database.Database {
		connection := composer.Must(postgresdatabase.New(dsn))
		composer.Cleanup(connection.Stop)
		composer.Startup(connection.Start)

		return database.New(connection)
	}
}
```

The helper captures configuration; acquisition happens inside the returned
callback. Replace the direct construction with one invocation:

```go
db := composer.Use(InventoryDatabase(dsn))

mountCatalog(routes, db)
mountStock(routes, db)
```

`Use` invokes the callback synchronously and returns its result unchanged.
Calling it again creates another driver: there is no cache, cloning, registry,
or automatic lifecycle adoption. Nested factories share the enclosing context
and cleanup stack, not a separate lifecycle.

For several related outputs, return a small named struct and pass individual
fields to consumers. Avoid a global bundle of every dependency: it hides what
each consumer needs. If you use a dependency-injection container, resolve typed
values during composition and choose either the container or the application
to own each resource's startup and release—never both.

## Own Subscriptions

An event subscription returns an unsubscribe function rather than a fallible
stop method. At the top of composition, obtain the specific event handle, then
use `MustDefer` to acquire and own the subscription. Here `onStarted` is your
existing `func(context.Context, event.Void) error` callback, with `event` from
`arkoren.dev/core/event`:

```go
events := composer.App().Events()

composer.MustDefer(events.Listen(application.Started, onStarted))
```

The subscription lasts until rollback or shutdown cleanup, not until the
composition function returns. A failed acquisition registers no release, even
if it returns one: the acquiring operation must clean up partial results itself.
A successful acquisition must return a nonnil release.

For an already acquired `cancel context.CancelFunc`, use `composer.Defer(cancel)`.
`Defer` accepts `func()`; `Cleanup` accepts `func(context.Context) error`. They
share one once-only, reverse-registration-order stack. Nil releases panic with
`ErrNilFunc`; release panics follow cleanup's panic-to-error handling.

Cleanup removes this listener before `Stopped` or rollback `Failed`. Keep
subscriptions needed for those events outside the cleanup stack, as shown in
[outcome observation](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md#observe-outcomes).
`Defer` and `Cleanup` can also register releases inside an executing startup
callback; `MustDefer` cannot.

## Package Registration

Use an `Action` when a helper registers work without producing a value. Given an
existing `listener *server.Server` from `arkoren.dev/core/http/server`, whose
acquiring owner already registered `listener.Stop`, this helper borrows it:

```go
func ServeInventory(listener *server.Server) application.Action {
	return func(composer *application.Composer) {
		composer.Startup(listener.Start)
		composer.Run(listener.Serve)
	}
}
```

Invoke `composer.Use(ServeInventory(listener))` after wiring routes. The action
adds startup and supervised serving, not a second cleanup owner. It returns no
application value; call `Use` as a statement and ignore its empty completion
result. Root callbacks passed directly to application `Start` or `Run` remain
ordinary `func(*application.Composer)` functions.

## Select Optional Features

If a catalog is disabled, skip its dependency and its routes together. As an
alternative to the always-on database above, assume validated configuration with
these typed fields:

```go
type CatalogSettings struct {
	Enabled bool
	DSN     string
}
```

With `catalog CatalogSettings`, construct only when selected:

```go
db, selected := composer.UseWhen(catalog.Enabled, InventoryDatabase(catalog.DSN))
if selected {
	mountCatalog(routes, db)
}
```

When disabled, the callback never runs: no pool, cleanup, or catalog routes are
created. Keep acquisition inside the callback. Go evaluates helper arguments
before selection, so passing an already opened resource does not defer acquisition.

Selectors return `(T, bool)`. A skipped factory returns zero and false; a selected
factory returns its value and true, even when zero or nil is legitimate. Test
the boolean for presence, not success. Construction errors abort composition;
they never become false. Selectors also accept actions; their result pair can
be ignored when no later branch needs it.

### Choose Selection Policy

Use a configuration boolean for a product feature. For a developer aid, choose
debug selection when operators opt in independently of deployment, or environment
selection when deployment determines policy. Given an existing
`diagnostics application.Action`, choose one of these alternatives:

```go
composer.UseDebug(diagnostics)
composer.UseIn(application.Development, diagnostics)
composer.UseInAny([]application.Environment{application.Development, application.Staging}, diagnostics)
```

`UseDebug` reads the construction-time `Debug` flag, which defaults to false;
environment and runtime log-level changes do not alter it. `UseIn` and `UseInAny`
match the single application environment exactly, including custom values.
`UseInAny` runs at most once; duplicate matches do not repeat it, and an empty
slice skips it. See [environment settings](https://arkoren.dev/docs/guides/first-steps/configuration.md#environment-settings)
for configuration. Selection is not access control: protect
[diagnostic endpoints](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md) separately.

Call `Use` and its selectors only during synchronous composition, never from
startup, runners, cleanup, or spawned goroutines. Invalid phases panic with
`ErrInvalidPhase`. Nil or typed-nil operations panic with `ErrNilFunc` even on
skipped branches; skipping also cannot erase a recorded composition failure.
Do not retain the composer for later container callbacks.

For every composition method, see the
[`application` API reference](https://pkg.go.dev/arkoren.dev/core/application).

---

Source: https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md
[HTML documentation](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership)

# Lifecycle and Ownership

Your HTTP server must not accept requests before its database is usable or keep
using that database after it closes. The application coordinates this order, but
your composition code chooses the owner of every resource. Start from the
[typed dependencies](https://arkoren.dev/docs/guides/application/composition.md) you have constructed; register each release
where ownership begins, not where startup happens to succeed.

## Register Resource Ownership

Assume `connection` is an unstarted root `*postgresdatabase.Driver` from
`arkoren.dev/driver/postgres/postgresdatabase`, and `listener` is an unstarted
`*server.Server` from `arkoren.dev/core/http/server` whose handlers use that
database. This application is their only lifecycle owner. In composition,
register the database first and the server second:

```go
composer.Cleanup(connection.Stop)
composer.Startup(connection.Start)

composer.Cleanup(listener.Stop)
composer.Startup(listener.Start)
composer.Run(listener.Serve)
```

All composition finishes before startup. The database verifies connectivity,
then the server binds its listener; serving launches only after both startup
callbacks succeed. Cleanup reverses its own registration order: the server
stops before the database. Consumers borrow the facade or driver and register
no second cleanup. A facade constructor does not register driver lifecycle.

Cleanup becomes active immediately, independently of startup progress. If a later
constructor fails, both registered releases still run even though neither startup
ran. If database startup fails, server cleanup still runs before database cleanup.
Every release must therefore tolerate a never-started or partially started
component. Cleanup is not paired positionally with startup hooks.

Use `Cleanup` for `func(context.Context) error` and `Defer` for an infallible
`func()`. Both share the same once-only cleanup stack. Composition's
[`MustDefer`](https://arkoren.dev/docs/guides/application/composition.md#own-subscriptions) combines acquisition with owning
an infallible release. Unlike Go's `defer`, these registrations survive the
composition function's return.

Follow the resource's [driver ownership contract](https://arkoren.dev/docs/guides/drivers/choosing-a-driver.md),
especially when wrapping a native client: a wrapper does not necessarily borrow
it. If a container also manages lifecycle, choose it or the application as owner,
not both. Configured application telemetry already has an owner; do not register
another start or stop for its handle.

## Supervise Running Work

Use application `Run` for an ordinary service. With your service-lifetime context
`ctx` and existing composition callback `compose`, import
`arkoren.dev/core/application`:

```go
app := application.New()
err := app.Run(ctx, compose)
```

`Run` composes, starts, waits for the runner group or caller cancellation, then
shuts down. Handle `err` at the executable boundary. Caller cancellation alone
after successful startup is a normal stop; startup, runner, and shutdown failures
are returned.

Each `composer.Run` callback launches on its own goroutine after startup. Use it
for serving or a consumer loop instead of launching unmanaged goroutines. The
runner must return when its supplied context ends. Handle transient errors inside
the runner when they should not stop the service.

An unexpected nonnil runner error or panic records the first failure and cancels
siblings. Results returned after cancellation count as normal stops. A nil return
does not cancel siblings: application `Run` waits for the entire group, including
the background health loop when checks exist. With no runners or checks, it waits
for caller cancellation or shutdown.

Composition and startup share the startup context's values, deadline, and
cancellation. Runner contexts preserve values but detach that deadline and
cancellation; shutdown or runner failure cancels them. Do not capture
`composer.Context()` for running work. Synchronous composition and startup must
also cooperate with cancellation: a deadline cannot interrupt their code.

## Acquire During Startup

Some resources cannot be acquired until earlier dependencies have started.
`Cleanup` and `Defer` may register releases synchronously inside an executing
startup callback. Register the release immediately after successful acquisition,
before further initialization can fail.

For example, your application-owned
`subscribe func(context.Context) (func(), error)` acquires a subscription and
returns a nonnil release on success. Its implementation cleans up partial results
on failure. `initialize func(context.Context) error` performs the remaining setup:

```go
composer.Startup(func(ctx context.Context) error {
	release, err := subscribe(ctx)
	if err != nil {
		return err
	}
	composer.Defer(release)

	return initialize(ctx)
})
```

If initialization fails, rollback still releases the subscription. The new release
joins the stack at the time of acquisition: it runs before releases registered
earlier during composition, regardless of startup-hook position. Use `Cleanup`
instead if release takes a context or can fail.

Return ordinary startup errors. `Must`, `MustDefer`, `Require`, and child
composition remain composition-only. A startup callback cannot add startup hooks,
runners, checks, or diagnostic tools. Do not retain the composer or register from
spawned goroutines; permission lasts only for the executing callback.

## Handle Failed Startup

A construction failure through `Must`, `MustDefer`, or `Require` skips all later
composition and startup. A startup error or detected cancellation skips later
startup callbacks and launches no runners. Both paths release every registered
resource and finalize telemetry before returning. A failed application cannot
start again; create a new application for another attempt.

Startup, runner, and cleanup panics become errors with stack traces. An unrelated
composition panic remains a programmer failure: cleanup and telemetry finalization
are attempted, then the exact original value is re-panicked. Secondary cleanup
errors do not replace it.

Use `errors.Is` and `errors.As` on returned errors because cleanup and telemetry
failures can be joined with the original cause. The application already logs
lifecycle failures before telemetry finalization. Rollback logs the original cause
with cleanup errors; normal shutdown logs cleanup errors and runner-wait timeouts.
Use the returned error for exit status or recovery rather than duplicate logging
after `Run`, when telemetry has stopped. Failures before telemetry starts reach
the configured log handler only; telemetry-finalization failures go directly to
that handler rather than through the failed exporter.

## Budget Shutdown

Shutdown immediately makes liveness and readiness unavailable. It emits `Stopping`,
cancels runners, and waits within the shutdown context's budget before cleanup.
Every release is attempted even after another fails, panics, or exhausts the shared
cleanup context. Errors are joined; telemetry finalizes last, after the final
lifecycle log and event.

> [!WARNING]
> Cancellation does not forcibly stop callback code. If a runner ignores its
> context, cleanup can overlap that runner after the wait times out. Make runners
> cooperate and releases safe for that possibility. Cleanup or telemetry callbacks
> that ignore cancellation can also prevent shutdown from returning.

Application `Run` uses a 30-second shutdown budget by default. If your drain needs
more time, configure it when constructing the application (`time` is the standard
library package):

```go
app := application.New(func(app *application.Builder) {
	app.ShutdownTimeout(45 * time.Second)
})
```

Zero or negative selects the default. This is not a hard total-duration limit.
Cleanup receives a context detached from caller cancellation but preserving
values, with the remaining caller budget and a minimum of ten seconds. If runner
waiting exhausts the budget, cleanup still gets a fresh ten seconds. Telemetry
finalization independently receives the remaining caller budget with at least
a fresh ten seconds. Rollback uses the same minimum budgets. Account for these
extra phases in your platform's
[shutdown budget](https://arkoren.dev/docs/guides/first-steps/deploying.md#shutdown-budgets).

## Separate Host Control

Use `Start`, `Wait`, and `Shutdown` separately only when another host owns service
lifetime or startup needs a separate deadline. `Start` runs once. A failed `Start`
already rolls back; successful startup always requires eventual `Shutdown`, even
after runner failure. Canceling the successful startup context does not stop
runners.

`Wait` returns the first runner failure, not its waiting context's error. It can
return nil when that context ends while runners remain active. Give `Shutdown`
a live, bounded context, not the context that just ended. Concurrent shutdown
callers wait for the single cleanup pass and receive its result unless their own
waiting context ends. During startup, shutdown cancels startup and waits for its
owner to roll back; timing out that wait does not take rollback ownership away.
Shutdown after failed startup returns nil because rollback is already complete.

Do not call `Shutdown` synchronously from composition, startup, a runner, or
cleanup: it can wait on its own caller. See the
[`Shutdown` contract](https://pkg.go.dev/arkoren.dev/core/application#Application.Shutdown)
for exact concurrent-call behavior.

## Observe Outcomes

Use lifecycle callbacks when failure must affect startup or shutdown. Use event
listeners for observation: listener errors and panics are logged without changing
the lifecycle result. Obtain `events := composer.App().Events()` at the top of
composition when registering observers there, and pass the specific handle to
other components rather than passing the application.

`Starting` follows successful composition and telemetry startup, before user
startup hooks. Listeners registered during composition can log through active
telemetry then. Composition or telemetry startup failures can emit `Failed`
without `Starting`; there is no built-in `app starting` log. `Started` means
runners have launched, not that the first health pass succeeded.

`Stopping` precedes runner cancellation and cleanup. `Stopped` follows waiting
and cleanup but precedes telemetry finalization; its listeners see `StateStopping`
and can write final telemetry. Only after finalization is `StateStopped` published.
`Stopped` does not mean cleanup succeeded: inspect the returned error.
`Failed` carries the original startup or runner cause, not added cleanup or
telemetry errors.

Keep a subscription needed for `Stopped` or rollback `Failed` outside the
application cleanup stack. In the function calling `Run`, assume `app`, `ctx`,
and `compose` already exist. Import `event` from `arkoren.dev/core/event`:

```go
logger := app.Logger()

unsubscribe, err := app.Events().Listen(application.Stopped, func(ctx context.Context, _ event.Void) error {
	logger.InfoContext(ctx, "inventory cleanup completed")

	return nil
})
if err != nil {
	return err
}
defer unsubscribe()
```

Now run the application in that same function:

```go
err = app.Run(ctx, compose)
```

Handle or return `err`; the deferred unsubscribe runs after `Run`, not during
application cleanup. Listeners are synchronous and must return: error isolation
cannot interrupt a blocked observer. Health events have a separate
[background-transition contract](https://arkoren.dev/docs/guides/application/health-and-readiness.md#observe-transitions);
see the [event bus guide](https://arkoren.dev/docs/guides/events/overview.md) for listener behavior.

For lifecycle settings and runtime methods, see the
[`application` API reference](https://pkg.go.dev/arkoren.dev/core/application).

---

Source: https://arkoren.dev/docs/guides/application/health-and-readiness.md
[HTML documentation](https://arkoren.dev/docs/guides/application/health-and-readiness)

# Health and Readiness

A database outage can make requests fail while the service process remains
healthy enough to recover. Use **readiness** to control traffic, **liveness** for
restart decisions, and a **startup probe** to give initialization time to finish.
Restarting every replica does not repair an unavailable shared database.

This page assumes an [application coordinator](https://arkoren.dev/docs/guides/application/overview.md) with dependency
[startup and cleanup](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md) already registered. Add checks
and routes during synchronous composition, before startup begins. Import
`application` from `arkoren.dev/core/application`.

## Select Required Checks

Include a dependency when requests cannot succeed without it. Leave optional
integrations out: otherwise their shared outage can remove every replica from
traffic. For an existing root `connection *postgresdatabase.Driver` from
`arkoren.dev/driver/postgres/postgresdatabase`, register its connectivity check:

```go
composer.Check("inventory-database", connection.Ping)
```

Registration does not call `Ping`. After composition and all startup hooks
succeed, a background loop begins checking. `Start` does not wait for its first
result. Nil means passing; an error or panic means failing. A failed check
changes readiness, not startup success, liveness, or runner-failure supervision.
The loop continues so a later successful pass can restore traffic.

Check names must be nonempty and application-wide unique; callbacks must be
nonnil. Invalid registration panics with `ErrNilFunc` or `ErrDuplicateCheck`.
Register only during synchronous composition, not in startup, requests, or later
goroutines; invalid phases panic with `ErrInvalidPhase`. Custom network checks
must propagate their context and close response bodies. In-memory dependencies
need no connectivity check.

## Mount Probes

Probe handlers have no built-in authentication. Mount them on a listener with
restricted network access that your platform can reach. Kubernetes HTTP probes
normally connect to the pod IP; a different port does not make a loopback-only
listener reachable. Keep sensitive
[diagnostics](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md) on loopback with port forwarding
or behind authentication and authorization instead.

With `app := composer.App()` and an existing `routes *router.Router` from
`arkoren.dev/core/http/router`, mount the three application-owned handlers:

```go
routes.Get(application.ReadyPath, app.ReadyHandler())
routes.Get(application.LivePath, app.LiveHandler())
routes.Get(application.StartupPath, app.StartupHandler())
```

The paths are `/readyz`, `/livez`, and `/startupz`. Each returns `200` with
`ok\n` or `503` with `unavailable\n`. None calls a dependency on request; polling
readiness more often does not cause more database calls.

| Probe | Success Means | Platform Use |
| --- | --- | --- |
| Readiness | The application is live and its cached checks pass and remain fresh. Without checks, readiness equals liveness. | Admit traffic. |
| Liveness | The application is running, no runner failure is recorded, and the runner group has not finished. | Decide whether to restart. |
| Startup | Startup completed, without waiting for checks. | Protect initialization from premature restarts. |

Before startup completes, all return `503`. Shutdown makes liveness and readiness
unavailable immediately. The startup probe stays successful during and after
shutdown following successful startup, if its listener remains reachable. No
verdict keeps a stopped listener reachable. A database-check failure alone does
not change liveness, while a runner failure does even if cached checks pass.
Configure your platform's [health probes](https://arkoren.dev/docs/guides/first-steps/deploying.md#health-probes)
accordingly; never use readiness as a restarting probe.

If you include probes in OpenAPI, pass `application.ReadyOperation`,
`LiveOperation`, or `StartupOperation` as the corresponding route's third
argument. These [annotations](https://arkoren.dev/docs/guides/openapi/built-in-handlers.md) describe responses;
they do not mount handlers, enable details, or apply access control.

## Tune Check Timing

With checks registered, readiness remains pending until the first completed
passing pass. A failure makes it unavailable; a later passing pass restores it
without restarting. A cached result remains fresh through three intervals of age.
Older results make readiness unavailable even if their individual checks passed.
Results interrupted by shutdown are discarded.

The first pass starts immediately after startup. Later passes follow an interval
ticker, not a full interval after completion. A 400 ms pass on a one-second
interval normally starts its next pass one second after the first began, not
1.4 seconds later. Checks in a pass run concurrently; the pass waits for all of
them, and background passes never overlap. Slow callbacks delay later observations.

Keep the defaults unless dependency latency or your traffic-removal budget needs
different timing. The interval defaults to five seconds; the timeout defaults to
two fifths of the configured interval, or two seconds by default. To use a shorter
cadence, set both during construction (`time` is the standard library package):

```go
app := application.New(func(app *application.Builder) {
	app.ReadinessInterval(2 * time.Second)
	app.ReadinessTimeout(500 * time.Millisecond)
})
```

Zero or negative selects the respective default. Keep the timeout shorter than
the interval. These settings are not setters on a running application.

> [!WARNING]
> Timeouts cancel the pass context, not callback code. A check that ignores
> cancellation prevents completion and delays later passes and shutdown. Keep
> checks lightweight, honor cancellation, and return an error when canceled.
> Returning nil after the deadline is not automatically converted into failure.

## Inspect Cached Results

When traffic stops reaching an instance, request `/readyz?verbose` on the
restricted probe listener first. It returns JSON with the plain probe's status
and no new dependency calls. Presence of `verbose` selects JSON regardless of
its value, including `false`. A failed check normally appears as:

```json
{
  "status": "unavailable",
  "checked_at": "2026-11-01T09:00:00Z",
  "checks": {
    "inventory-database": {
      "status": "failed",
      "error": "reason withheld"
    }
  }
}
```

The timestamp identifies the completed observation, not the request time. Reports
also explain `not live`, `no health pass completed`, and `health result stale`.
A stale report can still show passing checks; inspect the overall status and
reason. Plain and verbose probes set `Cache-Control: no-store` and
`X-Content-Type-Options: nosniff`.

Keep `HealthDetails` disabled unless every affected endpoint is operator-protected.
For an operator-restricted deployment, enable details during construction:

```go
app := application.New(func(app *application.Builder) {
	app.HealthDetails(true)
})
```

This includes backend error text, which can reveal addresses and connection
details. Names remain visible when details are disabled. Actual errors are logged
on failure transitions regardless of this setting; protect those logs too.

The setting also governs cached and fresh MCP health reports, `healthy` results,
and check details in fail-fast condition failures. It does not redact trusted
in-process `Checks`, `RunChecks`, or `CheckHealth` results, or event payloads.
Protect all affected remote endpoints before enabling it.

Inside the process, inspect cached typed records without contacting dependencies.
With the existing `app` and the standard library `fmt` package:

```go
for _, status := range app.Checks() {
	fmt.Println(status.Name, status.Err == nil, status.CheckedAt)
}
```

`Checks` returns copied records in registration order; error values remain shared.
It returns nil before the first completed pass and when no checks exist.
`CheckedAt` is the latest pass's completion time; `Since` records the first
observation or latest passing/failing transition. Use
[application diagnosis](https://arkoren.dev/docs/guides/diagnostics/application-diagnosis.md) for remote
investigation, and the
[`ReadyHandler` contract](https://pkg.go.dev/arkoren.dev/core/application#Application.ReadyHandler)
for report edge cases.

## Run Fresh Checks

An admin command or test may need the dependency's current state rather than the
cached traffic verdict. With an already started `app` and a caller context `ctx`
carrying a deadline:

```go
err := app.CheckHealth(ctx)
```

Nil means every registered check passed. Otherwise the error joins failed checks
in registration order, each prefixed with `health` and its name. This is the
command's dependency verdict, not an application startup error. Use
`app.RunChecks(ctx)` if you need individual passing and failing `CheckStatus`
records; their `Since` is zero because a fresh run has no transition history.

Both methods run checks concurrently with the configured readiness timeout and
the caller's deadline, and wait for every callback to return. They start no
dependencies, update no readiness cache, and publish no transition events. Calling
before composition cannot see checks not yet registered. A fresh success does
not restore traffic; the next passing background observation does.

Test callbacks directly for their behavior. For wiring tests, compose and start
the application before fresh checks and arrange shutdown even if assertions fail.
For cached transitions, use `testing/synctest` rather than fixed sleeps: a fresh
check assertion does not exercise the readiness cache.

## Observe Transitions

Background logs record failure and recovery transitions, not every repeated
verdict. `CheckFailed` fires for a first failing observation or a change from
passing to failing; `CheckRecovered` fires after a failing check passes.
`Ready` and `Unready` report the first completed pass's verdict and later changes
to whether all checks pass. These events describe background observations,
independently of lifecycle liveness.

Staleness, lifecycle changes, and explicit fresh checks publish none of these
events, so events alone cannot explain every `503`. Health payloads are trusted
in-process data; send errors only to authorized destinations, and do not mutate
shared payload slices. Listener failures are logged without changing lifecycle
success. Choose [observers rather than hooks](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md#observe-outcomes)
when alerts must not control startup or shutdown.

For check records, probe handlers, and settings, see the
[`application` API reference](https://pkg.go.dev/arkoren.dev/core/application).

---

Source: https://arkoren.dev/docs/guides/drivers/choosing-a-driver.md
[HTML documentation](https://arkoren.dev/docs/guides/drivers/choosing-a-driver)

# Choosing a Driver

A driver connects a service facade to its backend. Choose it from the behavior
your application needs, not only from whether it implements the interface.
[Facades and Drivers](https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers.md) explains which
responsibilities stay portable and which belong to the backend.

## State Scope

Start with where state must live. A memory cache is useful for isolated tests or
disposable data in one process. It cannot share a receipt between two API
replicas: each replica has its own cache. A shared Redis cache gives both replicas
access to the same keys, but persistence and failover durability still depend on
Redis configuration. Sharing state is not the same as preserving it indefinitely.

Local SQLite and filesystem storage can suit a single-host deployment. They do
not become shared services merely because several application replicas use the
same driver type. Choose a backend all relevant replicas can reach when they
must observe the same records, objects, counters, or work.

## Backend Requirements

Reuse a service your team can operate when it meets the contract you need. Then
check the linked provider guide for credentials, permissions, supported server
versions, resource provisioning, and startup requirements. A driver constructor
is not proof of connectivity or permission to perform an operation.

Compare the service's compatibility matrix before committing to a provider.
An interface method can return `ErrUnsupported`; vendor-native capabilities do
not automatically become portable operations. For example, a durable scheduler
can preserve timers and coordinate their delivery without guaranteeing
exactly-once application side effects. Keep handlers idempotent where repeated
delivery or an uncertain external result could duplicate work.

## Service Drivers

These are the available choices and their main prerequisites. Each Docs link
opens the canonical provider setup, including its compatibility evidence and
limits; this catalog does not duplicate those matrices. In-process drivers are
included with core. Vendor modules are installed from your application's module
directory as shown in the linked guide.

Hatchet is pending and unpublished. It is not an installable choice; its
development material belongs to the contributor archive, not application setup.

| Service | Driver | Pick when | Guide |
| --- | --- | --- | --- |
| Cache | Memory | Development and tests; a single process. | [Docs](https://arkoren.dev/docs/guides/cache/drivers.md#memory) |
| | Redis | Shared entries, counters, and locks; requires Redis 6.2 or later. | [Docs](https://arkoren.dev/docs/guides/cache/drivers.md#redis) |
| Feature flags | Memory | Development and tests; flags set in code. | [Docs](https://arkoren.dev/docs/guides/feature-flags/drivers.md#memory) |
| | OpenFeature | Flags in flagd, GO Feature Flag, ConfigCat, Flagsmith, or any service with an OpenFeature Go provider. | [Docs](https://arkoren.dev/docs/guides/feature-flags/drivers.md#openfeature) |
| | LaunchDarkly | Flags in LaunchDarkly, evaluated with its official Go SDK. | [Docs](https://arkoren.dev/docs/guides/feature-flags/drivers.md#launchdarkly) |
| Search | Memory | Development and tests; a single process. | [Docs](https://arkoren.dev/docs/guides/search/drivers.md#memory) |
| | PostgreSQL | Full-text search in an existing database; application-owned table migrations. | [Docs](https://arkoren.dev/docs/guides/search/drivers.md#postgres) |
| | Meilisearch | Dedicated prefix search; requires Meilisearch 1.15 or later. | [Docs](https://arkoren.dev/docs/guides/search/drivers.md#meilisearch) |
| | OpenSearch | Large indexes on an OpenSearch cluster. | [Docs](https://arkoren.dev/docs/guides/search/drivers.md#opensearch) |
| Database | PostgreSQL | A PostgreSQL server and pgx connection pool. | [Docs](https://arkoren.dev/docs/guides/database/drivers.md#postgresql) |
| | MySQL | An existing MySQL deployment. | [Docs](https://arkoren.dev/docs/guides/database/drivers.md#mysql) |
| | SQLite | An embedded database, tests, or an in-memory database. | [Docs](https://arkoren.dev/docs/guides/database/drivers.md#sqlite) |
| File storage | Memory | Development and tests; a single process. | [Docs](https://arkoren.dev/docs/guides/file-storage/drivers.md#memory) |
| | Filesystem | Files on a local disk, on a single host. | [Docs](https://arkoren.dev/docs/guides/file-storage/drivers.md#filesystem) |
| | Amazon S3 | S3 and S3-compatible object stores. | [Docs](https://arkoren.dev/docs/guides/file-storage/drivers.md#amazon-s3) |
| Queues | Memory | Development and tests; a single process. | [Docs](https://arkoren.dev/docs/guides/queues/drivers.md#memory) |
| | RabbitMQ | Durable AMQP work queues, publisher confirms, and manual acknowledgements. | [Docs](https://arkoren.dev/docs/guides/queues/drivers.md#rabbitmq) |
| | NATS | JetStream streams and durable pull consumers, not Core NATS alone. | [Docs](https://arkoren.dev/docs/guides/queues/drivers.md#nats) |
| | Amazon SQS | AWS queues, credentials, and a logical-name-to-URL mapping. | [Docs](https://arkoren.dev/docs/guides/queues/drivers.md#amazon-sqs) |
| | Cloudflare | Cloudflare Queues with HTTP pull consumers and an API token. | [Docs](https://arkoren.dev/docs/guides/queues/drivers.md#cloudflare) |
| Broadcasting | Memory | Development and tests; a single process. | [Docs](https://arkoren.dev/docs/guides/broadcasting/drivers.md#memory) |
| | Redis | Redis pub/sub. | [Docs](https://arkoren.dev/docs/guides/broadcasting/drivers.md#redis) |
| | NATS | NATS subjects. | [Docs](https://arkoren.dev/docs/guides/broadcasting/drivers.md#nats) |
| | RabbitMQ | An AMQP topic exchange. | [Docs](https://arkoren.dev/docs/guides/broadcasting/drivers.md#rabbitmq) |
| | MQTT | An MQTT 5 broker; check session and delivery limits. | [Docs](https://arkoren.dev/docs/guides/broadcasting/drivers.md#mqtt) |
| | Amazon SNS | Publishing to SNS and consuming from SQS. | [Docs](https://arkoren.dev/docs/guides/broadcasting/drivers.md#sns) |
| Mail | Log | Development: writes messages to the log instead of sending them. | [Docs](https://arkoren.dev/docs/guides/mail/drivers.md#log) |
| | SMTP | An SMTP server; encrypted delivery is the default. | [Docs](https://arkoren.dev/docs/guides/mail/drivers.md#smtp) |
| | Resend | A Resend API key and an authorized sender. | [Docs](https://arkoren.dev/docs/guides/mail/drivers.md#resend) |
| | Mailgun | A sending domain and private API key. | [Docs](https://arkoren.dev/docs/guides/mail/drivers.md#mailgun) |
| | Postmark | A server token and an authorized sender. | [Docs](https://arkoren.dev/docs/guides/mail/drivers.md#postmark) |
| | Amazon SES | AWS credentials and SES sending permissions. | [Docs](https://arkoren.dev/docs/guides/mail/drivers.md#amazon-ses) |
| Push notifications | Web Push | Visible browser notifications through standard encrypted Web Push, with application-owned subscriptions and service workers. | [Docs](https://arkoren.dev/docs/guides/push/drivers.md#web-push) |
| Payments | Memory | Local simulation without financial effects. | [Docs](https://arkoren.dev/docs/guides/payments/drivers.md#memory) |
| | Stripe | Stripe credentials; hosted checkout, existing recurring Prices, and card SDK preparation. | [Docs](https://arkoren.dev/docs/guides/payments/drivers.md#stripe) |
| | PayPal | Client credentials; Orders, explicit capture, and existing subscription Plans. | [Docs](https://arkoren.dev/docs/guides/payments/drivers.md#paypal) |
| | Mollie | An API key and enabled creditcard profile; one-off hosted card payments and explicit refunds. | [Docs](https://arkoren.dev/docs/guides/payments/drivers.md#mollie) |
| Scheduled tasks | Memory | Development and tests; a single process. | [Docs](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md#memory) |
| | Temporal | A Temporal service and worker host; persisted schedules and deferred workflows. | [Docs](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md#temporal) |
| | Restate | A Restate deployment and registered endpoint; persisted tick chains and deferred jobs. | [Docs](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md#restate) |
| Workflows | Memory | Development and tests; a single process. | [Docs](https://arkoren.dev/docs/guides/workflows/drivers.md#memory) |
| | Temporal | Temporal workflows and activities, with signals and timers. | [Docs](https://arkoren.dev/docs/guides/workflows/drivers.md#temporal) |
| | Restate | Restate workflow services, with signals and timers. | [Docs](https://arkoren.dev/docs/guides/workflows/drivers.md#restate) |
| Secrets | Memory | Development and tests. | [Docs](https://arkoren.dev/docs/guides/secrets/drivers.md#memory) |
| | Vault | A Vault KV v2 mount and token or login method with access to it. | [Docs](https://arkoren.dev/docs/guides/secrets/drivers.md#hashicorp-vault) |
| | AWS | AWS Secrets Manager and credentials with secret access. | [Docs](https://arkoren.dev/docs/guides/secrets/drivers.md#aws-secrets-manager) |
| | Google Cloud | Secret Manager resources and Google credentials with secret access. | [Docs](https://arkoren.dev/docs/guides/secrets/drivers.md#google-cloud-secret-manager) |
| | Azure | A Key Vault URL and an identity with secret access. | [Docs](https://arkoren.dev/docs/guides/secrets/drivers.md#azure-key-vault) |
| Encryption | AES | AES-GCM, the usual choice on hardware with AES support. | [Docs](https://arkoren.dev/docs/guides/encryption/drivers.md#aes-gcm) |
| | ChaCha20 | ChaCha20-Poly1305, for hardware without AES support. | [Docs](https://arkoren.dev/docs/guides/encryption/drivers.md#chacha20-poly1305) |
| Hashing | Argon2 | Passwords. The recommended choice. | [Docs](https://arkoren.dev/docs/guides/hashing/drivers.md#argon2id) |
| | bcrypt | Passwords, when you need compatibility with existing bcrypt hashes. | [Docs](https://arkoren.dev/docs/guides/hashing/drivers.md#bcrypt) |
| Telemetry | Memory | Trace IDs in logs and in-process inspection, without a backend. | [Docs](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md#memory) |
| | OpenTelemetry | Exporting traces, metrics, and logs over OTLP. | [Docs](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md#opentelemetry) |
| | Prometheus | Metrics scraped by Prometheus. | [Docs](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md#prometheus) |
| Language models | OpenAI | OpenAI credentials, or a compatible Responses/Chat Completions endpoint; capabilities vary by server and model. | [Docs](https://arkoren.dev/docs/guides/llm/drivers.md#openai) |

## Protocol Adapters

Some driver modules adapt a protocol rather than implement a service's `Driver`
interface. The MCP adapter serves application-owned `mcp.Tool` definitions from
`arkoren.dev/core/mcp`
over HTTP using the official Go SDK; it is not a storage backend or a language
model driver. [Your First MCP Server](https://arkoren.dev/docs/guides/mcp/first-server.md) shows registration
and HTTP serving; [Securing MCP](https://arkoren.dev/docs/guides/mcp/securing-mcp.md) explains the authentication
and authorization boundaries.

The [`mcpserver` API](https://pkg.go.dev/arkoren.dev/driver/mcp/mcpserver)
provides the adapter's constructor and transport settings.

For browser login or OIDC bearer verification, use the
[OAuth adapter](https://arkoren.dev/docs/guides/authentication/oauth.md). It integrates with HTTP
authentication rather than implementing a service `Driver` interface. Provider
registration, callback URLs, and accepted token audiences remain explicit setup.

Webhook adapters authenticate provider requests and expose typed event definitions.
They do not start receivers when you construct an outbound mail, payment, or LLM
driver. Bind the events you accept and mount the endpoint yourself:

| Adapter | Notifications | |
| --- | --- | --- |
| Stripe and PayPal | Payments and subscriptions. | [Docs](https://arkoren.dev/docs/guides/payments/drivers.md) |
| Resend | Email lifecycle and inbound email. | [Docs](https://arkoren.dev/docs/guides/mail/drivers.md#resend) |
| Mailgun | Email delivery and engagement. | [Docs](https://arkoren.dev/docs/guides/mail/drivers.md#mailgun) |
| Postmark | Email delivery, engagement, and inbound email. | [Docs](https://arkoren.dev/docs/guides/mail/drivers.md#postmark) |
| OpenAI | Background response lifecycle. | [Docs](https://arkoren.dev/docs/guides/llm/drivers.md#response-notifications) |

Follow [Receiving Webhooks](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md) for acceptance,
retries, and deduplication, and [Provider Adapters](https://arkoren.dev/docs/guides/webhooks/provider-adapters.md)
for authentication boundaries. The provider guides above own credential setup.
Outbound calls do not require a webhook endpoint.

CAPTCHA adapters also live in vendor modules without a shared service contract.
Choose the provider before its browser integration; tokens and acceptance
policies differ:

| Adapter | Pick when | |
| --- | --- | --- |
| Cloudflare Turnstile | You need token validity with hostname and action checks. | [Docs](https://arkoren.dev/docs/guides/abuse-prevention/captcha.md#cloudflare-turnstile) |
| Google reCAPTCHA | You need a CreateAssessment risk score and an application-owned threshold or step-up policy. | [Docs](https://arkoren.dev/docs/guides/abuse-prevention/captcha.md#google-recaptcha) |

Both include optional route middleware. The [CAPTCHA guide](https://arkoren.dev/docs/guides/abuse-prevention/captcha.md)
shows decode-once handler verification and header-based middleware separately.
For adapter options, see the
[`cloudflarecaptcha` API](https://pkg.go.dev/arkoren.dev/driver/cloudflare/cloudflarecaptcha)
and [`gcpcaptcha` API](https://pkg.go.dev/arkoren.dev/driver/gcp/gcpcaptcha).

## Driver-Free Features

A few features need no driver of their own. [Sessions](https://arkoren.dev/docs/guides/http-capabilities/sessions.md)
store their data in any cache, so they use whichever cache driver you configured.
[Events](https://arkoren.dev/docs/guides/events/overview.md) are delivered in process by the application's
event bus. [Secure links](https://arkoren.dev/docs/guides/http-capabilities/secure-links.md) are computed from
a secret and need no storage unless they are single-use.

After choosing a backend, follow its Docs link for construction, lifecycle
ownership, and a first operation. If no available driver meets the contract you
need, [write an application-owned driver](https://arkoren.dev/docs/guides/drivers/writing-a-driver.md).

---

Source: https://arkoren.dev/docs/guides/drivers/writing-a-driver.md
[HTML documentation](https://arkoren.dev/docs/guides/drivers/writing-a-driver)

# Writing a Driver

If your backend is not among the [available drivers](https://arkoren.dev/docs/guides/drivers/choosing-a-driver.md), you
can implement its service contract in your own Go module. A driver translates
that contract into backend operations; the facade continues to handle typed
application calls and its ordinary events and telemetry.

A wrapper is a driver too. It forwards to another implementation while changing
one responsibility, such as key naming. Neither kind requires an Arkoren
checkout or a contribution to the toolkit.

## Contract First

Start with the public interface and its method comments. For the cache example,
inspect [`cache.Driver`](https://pkg.go.dev/arkoren.dev/core/service/cache#Driver):

```sh
go doc arkoren.dev/core/service/cache.Driver
```

The contract describes missing values, expiry, cancellation, byte ownership,
and atomic operations. Its test package checks many of those rules, but tests
are evidence of covered behavior, not a replacement for the contract.

Decide whether you need a whole driver before implementing one. If an existing
driver already handles ordinary calls and you only need a provider-specific
operation, use its [native client access](https://arkoren.dev/docs/guides/core-concepts/facades-and-drivers.md#native-client-access).
For cache naming alone, [typed keys and namespaces](https://arkoren.dev/docs/guides/cache/keys.md) can avoid a
wrapper. A wrapper is useful when an existing raw-driver integration needs the
same naming rule applied to every operation.

## Wrap Existing Behavior

Suppose your application needs a fixed prefix on a raw cache connection. Create
an `internal/prefixcache` package in your application module. The following
fragments use `context` and `arkoren.dev/core/service/cache`; they illustrate
selected methods, not a complete `cache.Driver` implementation.

Keep the underlying driver private rather than embedding it. Embedding would
silently forward methods you forgot to adapt, potentially writing unprefixed
keys. A constructor with no optional settings needs no construction builder:

```go
type Driver struct {
	next   cache.Driver
	prefix string
}

func New(next cache.Driver, prefix string) *Driver {
	return &Driver{next: next, prefix: prefix}
}
```

This wrapper borrows `next`: its caller owns that driver's startup and shutdown.
Choose a complete, unambiguous prefix such as `reports:tenant-42:` and encode
variable identities consistently. Prefixing is not authorization or isolation
from code that can access the underlying driver.

For reads, change the key and leave the returned value and error semantics to
the underlying implementation:

```go
func (driver *Driver) Get(ctx context.Context, key string) ([]byte, error) {
	return driver.next.Get(ctx, driver.prefix+key)
}
```

Forward every interface method, applying the same prefix to every key or lock
name. Keep values, lock tokens, durations, and counter deltas unchanged. In
particular, `IncrementWithTTL` must use the prefixed identity too, or counter
windows would disagree with ordinary reads.

### Guarded Writes

A guarded write can compare one key and replace another. Transform both keys
before forwarding the *single atomic operation*:

```go
func (driver *Driver) CompareAndPut(ctx context.Context, match cache.Match, entry cache.Entry) (bool, error) {
	match.Key = driver.prefix + match.Key
	entry.Key = driver.prefix + entry.Key
	return driver.next.CompareAndPut(ctx, match, entry)
}
```

The records are passed by value, so changing their key fields does not change
the caller's records. A mismatch returns `false, nil` without changing either
entry or its TTL. Do not replace the call with a read followed by a write:
another writer could change the guard between them. Preserve `ErrUnsupported`
when the underlying driver cannot provide the required atomicity, and preserve
uncertain transport errors rather than falling back to an unconditional write.
[Guarded Writes](https://arkoren.dev/docs/guides/cache/overview.md#guarded-writes) shows the application use.

### Native Access

Expose the wrapped contract through `Unwrap` so the facade's `DriverAs` helper
can search through your wrapper:

```go
func (driver *Driver) Unwrap() cache.Driver {
	return driver.next
}
```

Native calls bypass your prefix transformation and the facade's operation
instrumentation. Give callers that escape hatch deliberately, not as a promise
that native operations obey wrapper policy.

After implementing all methods, add a compile-time interface check:

```go
var _ cache.Driver = (*Driver)(nil)
```

This catches missing methods and incorrect signatures. It cannot detect a
forgotten prefix or a non-atomic implementation; those need behavioral tests.

Your application can then pass the completed wrapper to the ordinary facade.
With an existing backend named `backend` whose lifecycle you already own:

```go
store := cache.New(prefixcache.New(backend, "reports:tenant-42:"))
```

Facade calls now use the wrapper's naming rule without changing application
key definitions. Keep the wrapper chain acyclic so `DriverAs` can finish its
search.

## Adapt a Backend

A backend driver performs the same contract operations through a vendor client.
Preserve distinctions that affect recovery: missing data is not an outage, a
rejected comparison is not a failed request, and a request error does not always
prove the backend performed no write.

For example, a Redis-backed `Get` can map native absence to `cache.ErrNotFound`
without hiding other failures. With a private `client *redis.Client` field,
import `errors` and `github.com/redis/go-redis/v9` in addition to the cache and
context packages:

```go
func (driver *Driver) Get(ctx context.Context, key string) ([]byte, error) {
	value, err := driver.client.Get(ctx, key).Bytes()
	if errors.Is(err, redis.Nil) {
		return nil, cache.ErrNotFound
	}
	return value, err
}
```

Callers can distinguish absence with `errors.Is`. A connection failure remains
a connection failure. If you wrap errors for context, preserve their identity
with `%w`; never include credentials or cached contents in the message.

The remaining implementation must satisfy the whole interface. For cache, that
includes independent byte copies, concurrent calls, cancellation before
mutation, nonmutating integer overflow, and atomic counters, pulls, and lock
ownership checks. Backend transactions, scripts, or native atomic commands may
be necessary. An unsupported result is valid only where the contract permits
it; do not advertise a capability your backend cannot implement safely.

## Construction and Ownership

Keep configuration separate from runtime operations. Put required values with
no safe default in constructor arguments. For optional settings, expose a
construction draft named `Builder` and its matching alias:

```go
type BuilderFunc = func(*Builder)
```

Use `New(required..., configure ...BuilderFunc)`. Invoke supplied callbacks
directly, synchronously, once in order on one fresh draft before ordinary
validation. A supplied nil callback panics; an expanded nil or empty slice adds
no edits. Draft methods return that draft. Publish detached private settings so
a retained draft or input container cannot reconfigure the driver. Native
clients, functions, and their captures need explicit ownership rules rather
than a claim of recursive copying.

The shipped Redis cache driver demonstrates this public shape. Import
`arkoren.dev/driver/redis/rediscache`, installed as described in
[Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md#redis):

```go
var connection rediscache.BuilderFunc = func(store *rediscache.Builder) {
	store.Addr("cache.internal:6379")
}
backend := rediscache.New(connection)
```

This constructs a driver without contacting Redis. `Start(ctx)` checks
connectivity, `Ping(ctx)` provides a readiness check, and `Stop(ctx)` closes its
client. Your application registers lifecycle work; the driver does not register
itself. [Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md)
explains the wiring.

### Native Resources

If you offer `NewFrom(client)`, document whether it borrows or takes ownership
and which optional settings still apply. Settings-free conversions need no
callback. Do not infer lifetime from the constructor name: Redis cache
`NewFrom` transfers ownership and `Stop` closes the supplied client, whereas
Redis broadcasting borrows its supplied client.

Configure native connection policy before sharing or transferring a client.
An accessor such as `Client()`, or `DB()`/`SQL()` for database drivers, exposes
native operations without transferring ownership. Explain when that accessor
is usable and whether native calls bypass facade or transaction behavior.

Acquire connections and start background loops at the documented lifecycle
boundary, not incidentally in ordinary configuration callbacks. Cloud SDK
configuration and credential loading can still require a constructor context.
Release owned resources even if the stop context has expired; document whether
shutdown waits for active work and how long the caller controls that wait.
Borrowing a shared resource must not let one driver close it behind another.

Keep credentials out of diagnostic output. Redact any exported settings record
in its `String`, `GoString`, and `slog.LogValuer` implementations, and avoid
logging native clients, payloads, or raw configuration. Redacting your record
does not sanitize native errors or caller-owned SDK objects.

### Schema Ownership

If the driver needs database tables, expose `Migrations()` for the application's
migrator. Do not run DDL during construction, startup, or the first operation.
The application applies schema changes through its
[migration deployment](https://arkoren.dev/docs/guides/migrations/deployment.md) process. Readiness can report
missing tables without trying to create them.

## Check the Contract

Once the prefix wrapper implements the complete interface, run the cache suite
against it over a fresh memory driver. In your application's wrapper test,
import `testing`, the cache package and its `cachetest` and `memorycache`
subpackages, plus your own `internal/prefixcache` package:

```go
func TestPrefixContract(t *testing.T) {
	cachetest.Run(t, func(t *testing.T) cache.Driver {
		return prefixcache.New(memorycache.New(), "reports:tenant-42:")
	})
}
```

The cache suite creates drivers per subtest and runs cases in parallel. This
memory driver needs no `Start` for reads or expiry, and no janitor is started
here. Add wrapper-specific tests that inspect underlying keys: a consistently
unprefixed wrapper could pass the portable suite while failing its own purpose.

For a vendor driver, initialize the required backend in the factory and register
cleanup on its supplied `*testing.T` before returning. Do not defer shutdown
inside the factory: it would close the driver before the suite uses it. If the
backend controls expiry, call the suite with `cachetest.WithExternalClock()`.
Use isolated disposable resources for [integration tests](https://arkoren.dev/docs/guides/testing/integration-tests.md).

Other suites have different factory shapes, lifecycle needs, and capability
options. Follow [Driver Conformance](https://arkoren.dev/docs/guides/testing/conformance-suites.md), especially
its [coverage limits](https://arkoren.dev/docs/guides/testing/conformance-suites.md#interpret-coverage), before
treating a pass as evidence. Test backend failures and application-specific
behavior separately; suite options do not relax required semantics.

For the interface and test entry points used here, see the
[`cache` API](https://pkg.go.dev/arkoren.dev/core/service/cache) and
[`cachetest` API](https://pkg.go.dev/arkoren.dev/core/service/cache/cachetest).

---

Source: https://arkoren.dev/docs/guides/http-fundamentals/routing.md
[HTML documentation](https://arkoren.dev/docs/guides/http-fundamentals/routing)

# Routing

A router connects an HTTP method and path to the handler that answers the
request. Arkoren uses Go's `http.ServeMux` matching rules and adds error-returning
handlers, route groups, and middleware. Create the router in your existing
application; [Running the Server](https://arkoren.dev/docs/guides/http-networking/server.md) owns serving it.

## Register Handlers

Import `net/http`, `arkoren.dev/core/http/router`, and
`arkoren.dev/core/http/response`. A status endpoint can return its response
helper's error directly:

```go
routes := router.New()
routes.Get("/status", func(w http.ResponseWriter, r *http.Request) error {
	return response.String(w, http.StatusOK, "ready")
})
```

`GET /status` returns `200` with the text `ready`. An Arkoren handler returns an
error instead of writing an error response at every failure site. Before the
final response starts, the router renders that error, normally as Problem
Details. After output starts, it cannot replace the response. Keep work that can
fail before the final response helper; [Error Handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md) explains
status selection and client-safe detail.

Returning `nil` without a final response produces `204 No Content`. An adapted
`net/http` handler instead owns its output and defaults to `200` when it writes
nothing. `router.New` does not install panic recovery; add
[`router.Recover()`](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md#panics) before routes when needed.

Use `Post`, `Put`, `Patch`, or `Delete` for other ordinary methods. With an
existing `invoiceStatus` handler, register a chosen set together:

```go
routes.Methods([]string{http.MethodGet, http.MethodHead}, "/invoices/status", invoiceStatus)
```

`Any` covers standard methods except TRACE and CONNECT; `Method` accepts an
explicit method string. GET matching also follows the standard library's HEAD
rules. Optional route metadata describes an [OpenAPI contract](https://arkoren.dev/docs/guides/openapi/overview.md),
not runtime decoding or validation.

Finish registration before serving requests, even for an empty router. Keep the
pointer from `New`; use `Clone`, `With`, or `Group` rather than copying a router
value. Focused fragments in these guides keep error results visible but omit
routine propagation; complete handlers must handle errors before using results.

## Capture Parameters

A wildcard captures one path segment. With your existing `showInvoice` handler:

```go
routes.Get("/invoices/{id}", showInvoice)
```

Inside that handler, use `r.PathValue("id")` for a string. For a numeric ID,
import `arkoren.dev/core/http/request` and parse before looking anything up:

```go
id, err := request.Param[int64](r, "id")
if err != nil {
	return err
}
```

A malformed or missing value returns a `400` problem. Parsing establishes a
value, not permission to access the invoice; authorize the resource separately.
[Requests](https://arkoren.dev/docs/guides/http-fundamentals/requests.md) covers query and body inputs.

A final wildcard such as `/assets/{path...}` captures the remaining path,
including slashes. Treat captured paths as untrusted input, not safe filesystem
paths. A catch-all also serves its parent path unless an explicit parent route
already exists; register that parent first.

Non-root routes match both trailing-slash forms. The root `/` matches only the
root. A literal route such as `/invoices/new` can coexist with
`/invoices/{id}` because it is more specific. Duplicate or ambiguous patterns
panic during registration; patterns containing `..` segments are rejected too.

## Group Routes

A group gives related handlers a common prefix and middleware scope. Here
`requireBilling` is your `router.Middleware`, and `listInvoices` and
`showInvoice` are existing `router.Handler` functions:

```go
routes.Group("/billing", func(billing *router.Router) {
	billing.Use(requireBilling)
	billing.Get("/invoices", listInvoices)
	billing.Get("/invoices/{id}", showInvoice)
})
```

Both routes run through `requireBilling`; routes outside the group do not.
`With(requireBilling).Get(...)` scopes extra middleware to one registration.
`Grouped` scopes middleware without adding a prefix, while `Clone` returns a
sub-router with an independent middleware stack and the same route registry.

Call `Use` before registering routes through that router or any derived router.
A later `Use` panics rather than silently leaving earlier routes unprotected.
Sub-router middleware does not alter its parent's stack. See
[Middleware](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md) for ordering and shared-state rules.

## Unmatched Requests

A missing path produces a `404` problem. A matching path with an unsupported
method produces a `405` problem with `Allow`. Both pass through the served
router's middleware, so rejection responses can carry security and CORS headers.
The fallback middleware chain is built on its first request and reused; its
captured values must support concurrent requests.

## Mount Native Handlers

For an existing `http.Handler` named `metrics`, adapt one endpoint:

```go
routes.Get("/metrics", router.HTTP(metrics))
```

The native handler writes its own errors: there is no returned error for the
router to render. Native middleware has a separate
[adapter boundary](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#native-middleware).

If an existing handler `tools` has its own paths, mount the entire handler:

```go
routes.Mount("/tools", tools)
```

The exact prefix and everything below it reach `tools`, with the prefix removed:
`/tools` and `/tools/` become `/`, and `/tools/jobs` becomes `/jobs`. Mounts
inherit middleware and cover the same methods as `Any`.

The local mount prefix must contain literal segments, not wildcards. Parent
groups may contain single-segment wildcards; those path values remain available
to the handler. Mounting strips the whole group prefix and preserves query
parameters and escaped path-segment boundaries. It copies the request and URL,
not the body or mutable context values. A mount remains an opaque catch-all for
OpenAPI: omit operation metadata or mark it hidden.

## Inspect Routes

`Routes` returns a detached eager collection in registration order, including
registrations from the whole router tree. To inspect only GET registrations:

```go
gets := routes.Routes().Filter(func(_ int, route router.Route) bool {
	return route.Method == http.MethodGet
})
```

The snapshot does not change with later registrations, and editing it cannot
change dispatch. Implicit trailing-slash variants are not separate entries.
Handler names and source locations can disclose application layout; restrict
access to any published inventory.

To exercise dispatch and middleware without starting a server, import
`net/http/httptest`:

```go
res := routes.Record(httptest.NewRequest(http.MethodGet, "/status", nil))
defer res.Body.Close()
```

You own the returned body. [HTTP Tests](https://arkoren.dev/docs/guides/testing/http-tests.md) shows assertions;
`Record` tests the in-memory pipeline, not network delivery.

For all registration and lookup methods, see the
[`router` API reference](https://pkg.go.dev/arkoren.dev/core/http/router).

---

Source: https://arkoren.dev/docs/guides/http-fundamentals/requests.md
[HTML documentation](https://arkoren.dev/docs/guides/http-fundamentals/requests)

# Requests

Client input needs parsing before your handler can use it. The `request`
package adds typed URL values and bounded body readers to `*http.Request`.
Input failures return problems that the router can render; operational failures
remain errors for server-side handling.

Import `arkoren.dev/core/http/request`. These fragments run inside an
[error-returning handler](https://arkoren.dev/docs/guides/http-fundamentals/routing.md#register-handlers), where `r` is the incoming
request. Return errors before writing a response or using a failed result.

## Read URL Values

For `/invoices/{id}`, parse the captured value into the type your application
uses. Named scalar types work too:

```go
type InvoiceID int64

id, err := request.Param[InvoiceID](r, "id")
```

`Param` returns a `400` problem for missing, empty, malformed, or out-of-range
values. It supports strings, booleans, integer types other than `uintptr`, and
floating-point types, including named versions. For unparsed text, use
`r.PathValue("id")`. A parsed ID still needs resource authorization.

Query parameters use the same scalar conversion. A required count should fail
when malformed; an optional display choice can use a fallback:

```go
limit, err := request.Query[int](r, "limit")
sort := request.QueryOr(r, "sort", "issued_at")
```

`Query` reads the first value and returns a `400` problem on failure.
`QueryOr` falls back for absent, empty, and invalid values; `ParamOr` does the
same for path values. Use `HasQuery(r, "preview")` to test presence independently
of value, including `?preview` or `?preview=`. Use an allowlist before turning a
sort choice into a database identifier. [Pagination](https://arkoren.dev/docs/guides/queries/pagination.md)
owns `request.Page` and `request.Cursor`.

URL decoding follows `net/url`, including omission of malformed query pairs.
Conversion follows `strconv`: booleans accept `1`, and floats accept `NaN` and
infinities. Apply [validation](https://arkoren.dev/docs/guides/validation/overview.md) for finite numbers,
ranges, or stricter grammar. OpenAPI parameter descriptions do not enforce them.

## Read Request Metadata

To read an optional client language preference:

```go
language := request.HeaderOr(r, "Accept-Language", "en")
```

`Header` returns an empty string when missing; `HeaderOr` also falls back for
empty values. These read text, not a verified identity or negotiated locale;
[locale selection](https://arkoren.dev/docs/guides/http-capabilities/localization.md#select-languages) owns language negotiation.

`Cookie` returns the whole cookie or `nil`; `CookieValue` returns its value,
and `CookieValueOr` supplies a fallback. Cookie and header values remain
untrusted. Use the [session capability](https://arkoren.dev/docs/guides/http-capabilities/sessions.md) rather
than treating an arbitrary cookie value as authentication.

## Decode JSON

JSON-looking bytes can arrive as `text/plain`, including in a cross-site
browser post. Body decoders do not check `Content-Type`; apply
[Content Types](https://arkoren.dev/docs/guides/http-fundamentals/content-type.md) before decoding, and use
[CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md) where cookies authorize
browser requests.

Use a request-specific struct so the endpoint exposes only intended writable
fields. For an invoice draft:

```go
type CreateInvoice struct {
	CustomerID string `json:"customer_id"`
	Reference  string `json:"reference"`
}
```

Decode the struct, rejecting misspelled or extra members when the endpoint has
a fixed input shape:

```go
input, err := request.JSON[CreateInvoice](r,
	request.Strict(),
	request.MaxSize(64<<10),
)
```

This reads at most 64 KiB under the helper's cap. Without options, body helpers
use 10 MiB or the limit stored by [Body Limits](https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md). An upstream
reader's ceiling cannot be raised by a per-call option. Limits bound bytes
read, not decoded memory or later application work. Call one body decoder per
request: the body is a consumed stream, not a reusable value.

The struct limits writable fields; it does not authorize the supplied customer.
Check that the caller may create invoices for `input.CustomerID` before writing
application data.

`JSON` returns `request.ErrInvalidJSON` (`400`) for native input failures and
`request.ErrBodyTooLarge` (`413`) when reading exceeds a cap. Unknown members
are ignored without `Strict`. JSON-v2 defaults make names case-sensitive,
reject duplicate members and a second top-level value, and reject fractions,
exponents, or overflow for native integer fields.

Ordinary custom decoder and body-read errors are not automatically bad input.
They keep their causes; existing HTTP problems retain their identity. A custom
decoder must deliberately identify client-input failures rather than exposing
arbitrary callback diagnostics. [Plain Errors](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md#plain-errors)
explains the router's private default rendering.

## Validate Input

Decoding establishes a Go value, not a valid invoice. With an existing
`validate.Schema[CreateInvoice]` named `invoiceInput`, combine decoding and
schema preparation:

```go
input, err := request.ParseJSON(r, invoiceInput,
	request.Strict(),
	request.MaxSize(64<<10),
)
```

`ParseJSON` returns the schema's prepared copy, including normalizations and
defaults. `ValidateJSON` checks the value but returns the decoded value instead.
Define schemas during application setup; [Validation](https://arkoren.dev/docs/guides/validation/overview.md)
owns their construction and [definition errors](https://arkoren.dev/docs/guides/validation/errors.md#definition-errors).

Schema violations become `request.ErrValidationFailed` (`422`) with an `errors`
member keyed by JSON Pointer. Pointers use wire member names, not Go selectors:
`json:"a/b~name"` becomes `/a~1b~0name`. Native semantic decoding failures also
become `422` violations with code `decode`; strict unknown members use `unknown`.
Syntax failures, including duplicate names, remain `400`.

Native decoding stops at its first semantic failure. Schema checks run on the
partially decoded value with presence-rule violations suppressed, so a response
is not an exhaustive list of malformed members. Fix the reported value and
resubmit. Schema infrastructure errors take precedence over field feedback;
ordinary application decoder failures remain operational errors.

A `decode` violation has no `params.expected`. Its optional
`params.default_representation` describes the Go type's default, not the required
wire format: an integer tagged `,string` still needs a quoted number. Custom
JSON methods, unsupported kinds, and `time.Duration` have no such hint. Keep
native causes in server diagnostics, not client detail. Use the current
[message catalog guidance](https://arkoren.dev/docs/guides/validation/errors.md#translate-messages) to translate
`decode`, and the [diagnostic ownership rules](https://arkoren.dev/docs/guides/validation/errors.md#diagnostic-ownership)
when recording causes.

Clients migrating from older native-decoding feedback must replace assumptions
about code `type` and `params.expected` with `decode` and the qualified default
hint. This does not change schema-generated codes, the `unknown` code, the
top-level validation problem identity, or its `422` status.

JSON-v2 supports `omitzero`, `omitempty`, `string`, `case`, and `embed` options;
the omission options do not change decoding, and `string` applies to a field's
top-level number, not composite elements. It does not support `format:*` tags.
`time.Duration` needs an explicitly represented input plus application conversion.
Native decoding support does not imply schema support: explicit validation field
registration rejects `embed`, `inline`, and `unknown` layouts. See
[struct validation](https://arkoren.dev/docs/guides/validation/structs-and-reuse.md).

## Read Other Bodies

For a small XML acknowledgment, choose the wire shape explicitly:

```go
type Acknowledgment struct {
	Reference string `xml:"reference"`
}

ack, err := request.XML[Acknowledgment](r, request.MaxSize(16<<10))
```

`XML` decodes only the first element and ignores trailing data. Native XML
syntax and numeric-conversion errors remain `request.ErrInvalidXML` (`400`),
including through `ValidateXML` and `ParseXML`; there is no native JSON Pointer
for field feedback. Operational decoder and read errors retain their causes.

`Bytes` and `String` read without decoding; `String` does not validate UTF-8.
`Strict` is unsupported for XML and raw bodies and returns
`request.ErrStrictUnsupported` before reading. A nil body is empty input: raw
reads return no data, while JSON and XML return their malformed-body problems.
For signed payloads, use the [webhook receiver](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md)
rather than decoding before signature verification.

## Parse Uploads

Allow `multipart/form-data` on upload routes before using `Multipart`. Its
size argument caps the whole body, not each file:

```go
form, err := request.Multipart(r, 12<<20)
if err != nil {
	return err
}
attachment := form.File("attachment")
```

`attachment` is a `*multipart.FileHeader`, or `nil` when absent.
`form.Value("reference")` reads the first text value. `Values` and `Files` keep
all submitted values in order. Validate content and size with
[file upload rules](https://arkoren.dev/docs/guides/validation/available-methods.md#file-uploads); storage
transfer belongs in [File Storage](https://arkoren.dev/docs/guides/file-storage/overview.md).

Up to 8 MiB of file content stays in memory; larger files spill to temporary
disk storage. Oversized, non-multipart, and malformed input produce `413`, `415`,
and `400` problems respectively. Transport and temporary-file I/O failures
remain operational errors. Existing HTTP problems preserve their identity.

When this helper parses inside a router handler lifecycle, it registers one
after-response cleanup hook. Outside that lifecycle, call `form.RemoveAll()`
after use. A form parsed elsewhere retains its existing cleanup owner. Successful
later calls reuse the parsed form and ignore their size argument; never retry
after a parsing failure because the stream has been consumed.

Consume and close opened uploads before cleanup; persist them before giving
work to a background job. `RemoveAll` deletes disk-backed temporary files, not
memory-backed data, and is safe to repeat. Removal failures can leave files
behind. Automatic cleanup is best-effort and does not return its error to the
handler; explicitly call `RemoveAll` before commitment if your application must
observe that failure. If query parsing fails after successful body parsing,
temporary-file cleanup is attempted and any cleanup failure stays in the error.

For all readers and body options, see the
[`request` API reference](https://pkg.go.dev/arkoren.dev/core/http/request).

---

Source: https://arkoren.dev/docs/guides/http-fundamentals/responses.md
[HTML documentation](https://arkoren.dev/docs/guides/http-fundamentals/responses)

# Responses

Once a handler has finished its work, send the result through its supplied
`http.ResponseWriter`. The `response` helpers set the format and status and
return any error, so the helper can be the handler's final call. Finish fallible
application work first: a returned error cannot replace a response already started.

Import `net/http` and `arkoren.dev/core/http/response`. These fragments assume an
Arkoren handler with writer `w` and incoming request `r`. For data delivered over
time rather than one complete result, use [Streaming Responses](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md).

## Return JSON

Give clients a deliberate response shape rather than serializing an internal
model with private fields. For an existing invoice ID:

```go
type InvoiceCreated struct {
	ID     string `json:"id"`
	Status string `json:"status"`
}

return response.JSON(w, http.StatusCreated, InvoiceCreated{
	ID:     invoiceID,
	Status: "draft",
})
```

The helper encodes using JSON-v2, sets `Content-Type: application/json`, and
writes the document followed by a newline. Encoding finishes before the final
status is written. An encoding failure therefore leaves the router free to
render an error response; a later network-write failure does not.

Native collection and pagination values use the same helper. Lazy collections
are materialized for encoding: they must be finite and fit in memory. A source
error leaves the response uncommitted, but work already performed by that source
is not rolled back. `response.XML` likewise buffers its encoded document before
writing and uses `application/xml`.

## Send Text

For a plain acknowledgment, use an explicit text response:

```go
return response.String(w, http.StatusOK, "Invoice accepted")
```

This sets `text/plain; charset=utf-8`, replacing an existing media type. It does
not validate or transform the supplied text. If no body is needed, write only
the final status:

```go
return response.Status(w, http.StatusNoContent)
```

The `204` is still a committed response. An informational status such as `103`
is not final; send a final response afterward. An Arkoren handler returning
`nil` without a final response defaults to `204`, unlike an adapted native
handler's `200` default.

## Render HTML

Use `html/template` when a page contains user-provided values. With an existing
parsed `*html/template.Template` named `invoicePage`, pass values as data, not
as template source:

```go
type InvoicePage struct {
	Reference string
}

return response.HTMLTemplate(w, http.StatusOK, invoicePage,
	InvoicePage{Reference: reference},
)
```

Contextual HTML escaping protects the reference when the template uses ordinary
strings. Do not mark untrusted values as trusted template HTML. Rendering
finishes in a buffer before commitment; execution failures return without
starting the response. Success sets `text/html; charset=utf-8`.

`response.HTML` writes its input unchanged: reserve it for trusted HTML you
produced yourself, never concatenated user input. `StringTemplate` uses
`text/template` without HTML escaping and sets `text/plain; charset=utf-8`.
Both template helpers replace a pre-existing media type. For another format,
render into your own buffer and send the bytes with `Raw`.

## Send Bytes

For already encoded content, set a known media type and use `Raw`. With an
existing PNG byte slice `preview`:

```go
w.Header().Set("Content-Type", "image/png")
return response.Raw(w, http.StatusOK, preview)
```

`Raw` keeps an explicit `Content-Type`; otherwise it uses
`application/octet-stream` rather than sniffing bytes. `Bytes` always sets
`application/octet-stream`, replacing an earlier media type. Do not let an
untrusted upload choose an active response type such as HTML. See
[file downloads](https://arkoren.dev/docs/guides/file-storage/uploads-and-downloads.md) for storage transfer
and download handling rather than loading large files into a response buffer.

## Redirect Safely

For a fixed application-owned path after a successful POST:

```go
return response.Redirect(w, http.StatusSeeOther, "/invoices")
```

`Redirect` sets `Location` and the supplied status without a response body. It
does not validate the target. A user-controlled return path instead needs
`SafeRedirect`; import `arkoren.dev/core/http/request` to read it:

```go
next := request.QueryOr(r, "next", "/invoices")
return response.SafeRedirect(w, http.StatusSeeOther, next)
```

Only a relative path starting with exactly one `/` is accepted. Absolute URLs,
network-path targets such as `//other.example`, backslashes, and URL parse
failures return `response.ErrUnsafeRedirect` (`400`) before commitment. The
router renders that error; the helper does not write the rejection itself.
`ValidateRedirectTarget` provides the same check when validation must happen
before another operation.

## Response Commitment

A response is **committed** when a final status is written, explicitly or
implicitly through a body write or flush. Informational `1xx` statuses other
than `101 Switching Protocols` leave the final response pending. `101` is final
for commitment tracking.

Buffered encoding and template preparation can fail while the response is still
pending. Once a final status is written, returned errors still reach outer
middleware and after-response hooks, but the router neither replaces the
response nor appends a problem body. Accepted bytes cannot be undone, even if a
later write fails. [Error Handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md#committed-failures) explains
panic recovery at this boundary; [Streaming Responses](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md#committed-failures)
explains live-output failures.

Always use the supplied writer for normal HTTP output. Writing through an
unwrapped native writer bypasses hooks and commitment tracking. A successful
connection hijack suppresses further error rendering and the default `204`;
a failed hijack does not itself commit. You own a hijacked connection, including
closing it and arranging shutdown. Router recovery does not take it back.

For all output helpers, see the
[`response` API reference](https://pkg.go.dev/arkoren.dev/core/http/response).

---

Source: https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md
[HTML documentation](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses)

# Streaming Responses

A live progress feed should deliver each update without waiting for the entire
job. Streaming helpers write and flush values as they arrive, rather than
preparing one buffered response. That changes failure handling: the initial
`200 OK` is sent before production finishes, so a later error cannot become a
replacement problem response.

Use these helpers inside an existing Arkoren handler with `w` and `r`. Import
`arkoren.dev/core/http/response`; [Responses](https://arkoren.dev/docs/guides/http-fundamentals/responses.md) owns ordinary buffered
results. Authenticate and authorize access before starting a live feed.

## Choose Event Output

For browser `EventSource` clients, use server-sent events (SSE). Each yielded
value becomes one event. With an existing `iter.Seq2[InvoiceProgress, error]`
named `updates`, define the client-visible shape and stream JSON:

```go
type InvoiceProgress struct {
	Stage   string `json:"stage"`
	Percent int    `json:"percent"`
}
```

Pass the iterator directly; the helper handles JSON encoding and event framing:

```go
return response.SSEJSON(w, r, updates)
```

The response uses `text/event-stream` and `Cache-Control: no-cache`. A native
fallible lazy collection can supply the iterator with `updates.Iter()` instead;
that conversion consumes nothing and transfers no cancellation responsibility.
[Streaming Replies](https://arkoren.dev/docs/guides/llm/streaming.md) applies this format to model output.

For pre-encoded event data, `response.SSE` accepts
`iter.Seq2[[]byte, error]`. Each payload line becomes a `data:` field. CRLF and
bare CR normalize to LF, including trailing line breaks; empty payloads emit
empty events. Payload line breaks cannot inject `event:` or `id:` fields.

If the client expects raw bytes instead, use `response.Stream`. With an existing
byte iterator `chunks`, select its actual media type before starting:

```go
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
return response.Stream(w, r, chunks)
```

Without an explicit type, raw streaming uses `application/octet-stream`.
`Stream` and `SSE` copy each yielded byte slice before `yield` returns, so the
producer may reuse its buffer. `SSEJSON` completes each value's encoding before
releasing the yield callback.

## Stop Iterator Work

Iterator helpers start a producer goroutine and always wait for it to exit.
When consumption stops, `yield` returns false: return immediately without
yielding again, including from cleanup. A blocked operation outside `yield`
must also have a way to finish, or the handler can wait indefinitely.

For a periodic status feed, this iterator fragment uses `time` and
`arkoren.dev/core/http/server`. It runs inside the handler and captures its
request `r`:

```go
ticks := func(yield func(time.Time, error) bool) {
	ticker := time.NewTicker(time.Second)
	defer ticker.Stop()
	for {
		select {
		case <-r.Context().Done():
			return
		case <-server.ShuttingDown(r.Context()):
			return
		case tick := <-ticker.C:
			if !yield(tick, nil) {
				return
			}
		}
	}
}
```

Send the timestamps with the same SSE helper:

```go
return response.SSEJSON(w, r, ticks)
```

Serve through the [Arkoren server](https://arkoren.dev/docs/guides/http-networking/server.md) to supply its
shutdown signal. Graceful shutdown does not cancel the request context, so
watching only `r.Context().Done()` is insufficient. The helper ends cleanly
with nil on server shutdown and returns the context error on request cancellation.

A write or flush failure does not cancel the captured request context either.
The ticker above has a bounded wait even without cancellation. Give network
calls or other blocking producer work their own bounds as well. Network write
deadlines do not interrupt arbitrary producer code. Producer panics, including
cleanup panics, are re-raised on the request goroutine after joining, not
converted to iterator errors; request recovery can then handle them.

## Own Channel Producers

If your application already supplies a `<-chan []byte`, use `StreamChan` or
`SSEChan`. With an existing channel `events`:

```go
return response.SSEChan(w, r, events)
```

Channel closure ends consumption, as do cancellation, shutdown, or I/O failure.
These helpers neither start nor join your goroutine. You own its stop signal,
cleanup, and join. Arrange a consumer-exit signal in addition to request and
server cancellation: an I/O failure can leave the request context active and
the producer blocked on a send nobody will receive. Do not assume that returning
from this helper proves your channel producer has stopped.

## Bound Slow Writes

The helpers renew the write deadline before the initial header flush and every
data write. Supported, enforced deadlines let a long stream outlive the server's
overall write timeout while bounding a stalled client write or flush. They do
not bound how long joining the producer takes.

The per-write timeout defaults to 30 seconds. For a feed that should drop slow
clients sooner, import `time` and set a shorter bound:

```go
return response.SSEJSON(w, r, updates,
	response.WithWriteTimeout(10*time.Second),
)
```

Zero or negative values select the default. A deadline-setting error matching
`http.ErrNotSupported`, even when wrapped, leaves the writer's existing timeout
policy in effect. That fallback does not guarantee bounded writes. Other
deadline-setting errors return unchanged: an initial failure is before
commitment; a later failure stops an already started response.

## Committed Failures

The supplied writer must implement `http.Flusher`. Otherwise the helper returns
`response.ErrNonFlushableWriter` before committing or starting an iterator
producer. Capability advertisement does not guarantee later success: flush
errors are returned, including `http.ErrNotSupported`. A writer exposing
`FlushError() error` can report failures; one exposing only `Flush()` cannot.

The initial flush happens after `WriteHeader(200)`, so even an initial flush
failure is committed. Iterator errors, JSON encoding errors, and write/flush
failures return to outer middleware and hooks without appending a problem body.
Bytes already accepted by a write remain accepted even if flushing them fails.
Use [request logging](https://arkoren.dev/docs/guides/observability/request-logging.md) to record the failure,
and design the client protocol to recognize completion rather than assuming
that a `200` means all updates arrived.

A recovered panic after commitment aborts the HTTP response with
`http.ErrAbortHandler`; see [Error Handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md#committed-failures).
Hijacked connections such as WebSockets are outside these helpers' lifecycle:
you must close them and arrange their shutdown yourself.

For all stream helpers and options, see the
[`response` API reference](https://pkg.go.dev/arkoren.dev/core/http/response).

---

Source: https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md
[HTML documentation](https://arkoren.dev/docs/guides/http-fundamentals/error-handling)

# Error Handling

An Arkoren handler returns a failure instead of formatting an error response at
each call site. While the final response is pending, the router renders the
error, normally as [Problem Details](https://www.rfc-editor.org/rfc/rfc9457).
Keep fallible work before the response helper: after commitment, returning an
error reports the failure but cannot replace output already sent.

This applies to error-returning handlers, including router-generated `404` and
`405` problems. Adapted `net/http` handlers write their own errors. Import
`net/http`, `errors`, and `arkoren.dev/foundation/problem` for the examples below.

## Plain Errors

You can return an ordinary service error unchanged. The router does not send
its text to the client, since diagnostics can disclose data or implementation
details. Without an error-owned renderer, status selection follows this order:

1. An error in the chain implementing `HTTPStatus() int` supplies its status.
2. Otherwise, a finished request context selects `499`, even for its deadline.
3. Otherwise, an error matching `context.DeadlineExceeded` selects `504`.
4. All other errors select `500`.

`499` is a nonstandard classification; a disconnected client might receive no
response. An error implementing both `error` and `http.Handler` instead renders
itself before commitment. That implementation owns negotiation and client-safe
output; it is not wrapped in another generated problem. `problem.Details`
implements both rendering and status interfaces.

Raw validation errors have no HTTP meaning by themselves. Use
`request.ValidationProblem` or the mapping below to convert violations to
`422`; do not map infrastructure failures into field advice. See
[Classify Failures](https://arkoren.dev/docs/guides/validation/errors.md#classify-failures).

## Return Problems

For a failure the client can act on, return a problem with an explicit safe
explanation. Inside an invoice handler with a loaded `invoice`:

```go
if invoice.Paid {
	return problem.Conflict().Derive(func(details *problem.Builder) {
		details.Detail("A paid invoice cannot be edited.")
	})
}
```

The status helper supplies `409 Conflict`. `Derive` produces a modified value
without changing the template. The response includes `type`, `title`, `status`,
and your `detail`; rendering fills an absent `instance` from the request path.
Other status helpers include `BadRequest`, `NotFound`, `Forbidden`, and
`UnprocessableContent`. `Unauthorized` takes the authentication challenge string.

If another failure caused this result, retain it as a diagnostic cause, not
as the client explanation. Here `err` is an existing application error:

```go
return problem.Conflict().Derive(func(details *problem.Builder) {
	details.Detail("The invoice changed. Reload it before editing.")
	details.Error(err)
})
```

The cause is excluded from the response but remains reachable through
`errors.Is` and `errors.As`. `Error(nil)` clears a cause. Problem `Error()` text
can include that cause, so it belongs in restricted server diagnostics, not in
another response's detail.

## Define Stable Problems

If clients distinguish an application failure, define its identity once:

```go
var ErrInvoiceClosed = problem.New(func(details *problem.Builder) {
	details.Type("https://api.example.com/problems/invoice-closed")
	details.Title("Invoice closed")
	details.Status(http.StatusConflict)
})
```

Return `ErrInvoiceClosed` or derive occurrence-specific details from it. Clients
can branch on `type` rather than parsing a sentence. The type must be a unique
URI reference; it need not resolve to a page. Arkoren's own failures use stable
`tag:arkoren.dev,...` identifiers and can be remapped the same way as domain errors.

`errors.Is(err, ErrInvoiceClosed)` compares stored type, status, and title.
Detail, extensions, headers, and cause do not alter this identity; editing an
identity member does. Cause traversal remains independent. For `errors.As`, a
value problem matches `problem.Details`, while a pointer matches
`*problem.Details`, following ordinary Go rules.

`New` defaults missing status to `500`, type to `about:blank`, and an empty
title to the final status phrase. `Derive` inherits stored members: changing
status alone does not reset an existing title. Callbacks run directly in order;
omitting callbacks makes no edits, but a supplied nil callback panics. Retaining
a builder cannot alter the published value. Outer extension maps and header
slices are copied; nested mutable values remain shared.

## Map Domain Errors

Central mapping keeps HTTP choices out of domain services. With an existing
router `routes` and application sentinel `ErrInvoiceVersion`, import
`arkoren.dev/core/http/middleware/errormap` and
`arkoren.dev/core/http/request`:

```go
routes.Use(errormap.Middleware(
	errormap.Map(ErrInvoiceVersion, problem.Conflict()),
	errormap.Func(request.ValidationProblem),
))
```

Register this before routes and before middleware whose errors it must see.
`Map` uses `errors.Is` and attaches the original error as the replacement's
cause. `Func` performs a custom conversion and owns cause preservation itself;
`ValidationProblem` already retains the cause and leaves non-validation errors
unchanged. Request `Parse*` and `Validate*` helpers perform that conversion already.

Mappings try the original error in order; the first replacement wins. Return
nil to decline reliably. Returning the unchanged input also declines, but
separately constructed equal value errors can be indistinguishable. Do not
mutate inputs, and use pointer errors when replacement identity matters.
Matching through `errors.Is` alone does not make a derived replacement unchanged.
The ordered slice is copied at construction; functions and captures stay shared.

A mapper cannot rewrite a response already rendered by a
[native middleware adapter](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#native-middleware). Put a mapper
inside that adapter if it must choose the rendered HTTP result; it can then see
only failures from layers inside it.

## Add Client Metadata

Use extensions for small, client-safe fields, not arbitrary diagnostic objects:

```go
return ErrInvoiceClosed.Derive(func(details *problem.Builder) {
	details.Extension("invoice_id", invoice.ID)
	details.Header("Cache-Control", "no-store")
})
```

Extensions appear at the top level of JSON. `Header` copies the supplied values;
passing no values removes that header. Accessors such as `Extension(name)`
report presence separately from its value; `Extensions()` copies the outer map.

For metadata applying to any problem from a request, the router supplies a
request-local response context. With a client-safe application `requestID`:

```go
if metadata, ok := problem.FromContext(r.Context()); ok {
	metadata.SetProperty("request_id", requestID)
}
```

Explicit problem extensions win over context properties; context properties
cannot replace standard members. Context methods synchronize outer maps and
header slices, and rendering snapshots them. Nested maps and slices still need
caller coordination during encoding. Keep the context as a pointer: it contains
a mutex and must not be copied after use.

## Negotiate Error Output

The renderer prefers `application/problem+json`, then `application/json`, then
`text/plain; charset=utf-8`. Missing or wholly invalid `Accept` uses that order.
A valid header excluding all supported representations produces `406` without
the original problem representation.

For each representation, exact ranges outrank type wildcards, which outrank
`*/*`; more matching parameters then outrank fewer, followed by quality. A
selected `q=0` excludes that representation even if a broader range permits it.
Across representations, higher quality wins, then matching-header order, then
server preference. Invalid ranges are ignored individually. An omitted `q`
means `1`; explicit weights must be `0` or `1`, optionally with up to three
fractional digits, all zeros after `1.`.

Parameters must match what is emitted. Text matches UTF-8 charset
case-insensitively, not another charset; other parameter values match exactly.
The JSON representations declare no parameters, so a required `charset` or
`profile` does not match them. For example,
`text/plain;charset=UTF-8;q=0, text/plain;q=1` still excludes UTF-8 text.

The renderer owns `Content-Type`. Problem headers replace context headers of
the same name; afterward `Vary` gains `Accept` unless it already contains that
token case-insensitively or contains `*`. Other surviving tokens remain,
including on `406` responses.

HTTP rendering completes missing standard members on a copy and changes stored
statuses outside `100`–`999` to `500` in that copy only. Stored accessors, plain
JSON, matching identity, and cause remain unchanged. Plain JSON omits an absent
status; decoding preserves absence as zero, replaces the destination only on
success, and clears previous causes and headers. `Defaulted` fills missing
members but does not validate a nonzero status.

Encoding finishes before commitment. If JSON encoding fails, text is used only
when acceptable under the same negotiation rules; otherwise the result is `406`.
A body-write failure stops rendering, not a second representation attempt.

## Panics

Install recovery before registering the handlers and middleware it must protect.
Import `arkoren.dev/core/http/router`:

```go
routes.Use(router.Recover())
```

Before output starts, an ordinary panic becomes an error for normal rendering,
usually a generic `500`. `RecoverBuilder.Handler` supplies a custom conversion;
`Handler(nil)` chooses the default. A converter returning nil still produces
an error matching `router.ErrRecoverUnexpected`, never success.

`RecoverBuilder.Events` publishes `router.PanicRecovered` through your event
bus. Its error and stack contain sensitive diagnostics. Restrict listeners and
sanitize forwarded reports. Recovered errors expose `Stack() []byte` through
`errors.As`; each call copies the stack, separately from the event's storage.
Listeners of the same event share its bytes, so copy before mutating. Copying
does not redact data. `http.ErrAbortHandler` is re-raised without a recovery event.

## Committed Failures

A final status, body write, or flush commits the response. Informational `1xx`
except `101` does not. A successful hijack also prevents further HTTP rendering;
a failed hijack does not itself commit. Use the supplied writer to preserve
[commitment tracking](https://arkoren.dev/docs/guides/http-fundamentals/responses.md#response-commitment).

After commitment, ordinary returned errors still reach outer middleware and
after-response hooks, but no replacement problem is appended. A recovered panic
after earlier output instead raises `http.ErrAbortHandler` so `net/http` can
abort the response. This is distinct from a native adapter rendering a recovered
error before any earlier output: that newly generated problem is not mistaken
for output preceding the panic. Wrapping such an error retains `errors.Is` and
`errors.As`; a new panic is not handled merely because its cause contains an
older rendered recovery.

Hijacked connections remain yours to close and drain. Recovery does not return
ownership to the server. See [Streaming Responses](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md) for
cooperative producers, write deadlines, and live-output failures.

For all constructors and contracts, see the
[`problem`](https://pkg.go.dev/arkoren.dev/foundation/problem),
[`router`](https://pkg.go.dev/arkoren.dev/core/http/router), and
[`errormap`](https://pkg.go.dev/arkoren.dev/core/http/middleware/errormap) API references.

---

Source: https://arkoren.dev/docs/guides/http-fundamentals/middleware.md
[HTML documentation](https://arkoren.dev/docs/guides/http-fundamentals/middleware)

# Middleware

Middleware wraps a handler to apply a policy to several requests: it can reject
input, establish request state, or observe the result. Choose both its scope and
its position. A wrapper sees only the handlers and middleware inside it, so an
incorrect order can hide failures or leave work unprotected.

These examples use `router` from `arkoren.dev/core/http/router` and a fresh
router `routes`. Call `Use` before registering routes through that router or
its derived routers; a later call panics. Configuration alternatives below are
independent, not calls to append after earlier route examples.

## Choose Scope

For behavior shared by every route, attach middleware to the router. Import
`arkoren.dev/core/http/middleware/secureheaders`:

```go
routes.Use(router.Recover(), secureheaders.Middleware())
```

Recovery protects the layers inside it; secure headers apply to the requests
that reach that middleware. At router scope, unmatched `404` and `405` requests
also traverse the stack.

For a billing-only policy, group existing `listInvoices` and `createInvoice`
handlers under `requireBilling`, your `router.Middleware`:

```go
routes.Group("/billing", func(billing *router.Router) {
	billing.Use(requireBilling)
	billing.Get("/invoices", listInvoices)
	billing.Post("/invoices", createInvoice)
})
```

The group inherits parent middleware without adding its policy to unrelated
routes. To protect one registration, use `With` instead:

```go
routes.With(requireBilling).Post("/invoices", createInvoice)
```

`With` leaves the parent's stack unchanged. All handler and middleware captures
remain shared across concurrent requests; scoping a stack does not copy captured
services or mutable state.

## Order Policies

Middleware runs in the order passed to `Use`: the first wrapper sees the
request first and the returned result last. For example, with existing
`[]errormap.Mapping` named `mappings`, import `errormap`, `requestlog`,
`bodylimit`, and `contenttype` from `arkoren.dev/core/http/middleware/`:

```go
routes.Use(
	errormap.Middleware(mappings...),
	requestlog.Middleware(),
	router.Recover(),
	secureheaders.Middleware(),
	bodylimit.Middleware(),
	contenttype.Middleware(),
)
```

The mapper sees errors from all inner layers. Logging sees recovered errors;
recovery sees panics from the policy middleware and handler. Body limits precede
anything that reads the body. An outer logger in this arrangement sees the
original error before the mapper replaces it; choose observation placement
deliberately if you need mapped errors instead.

Place [request tracing](https://arkoren.dev/docs/guides/observability/request-tracing.md) outside logging
when logs need its trace context. Place [CORS](https://arkoren.dev/docs/guides/browser-security/cors.md)
before authentication or body policies when preflights should terminate before
those checks. A policy outside security headers can reject a request before the
headers run, so choose that boundary consciously.

## Write Policies

A `router.Middleware` takes a handler and returns a handler. Construction runs
when the chain is built, not per request. Keep request-local values inside the
returned function. This example uses `net/http`,
`arkoren.dev/core/http/request`, and `arkoren.dev/foundation/problem`:

```go
func requireWorkspace(next router.Handler) router.Handler {
	return func(w http.ResponseWriter, r *http.Request) error {
		if request.Header(r, "X-Workspace") == "" {
			return problem.BadRequest().Derive(func(details *problem.Builder) {
				details.Detail("A workspace header is required.")
			})
		}
		return next(w, r)
	}
}
```

Register it before the routes it protects. Missing input returns a `400`
problem without calling the handler; otherwise the handler's error passes
outward. This checks presence only. Validate the workspace and enforce
[authorization](https://arkoren.dev/docs/guides/authorization/overview.md) before using it as a trusted tenant.

A wrapper can inspect or replace the error from `next`. It cannot replace an
already committed response by returning a different error. For shared domain
translation, prefer [error mapping](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md#map-domain-errors).

## Use Response Hooks

Header edits made after `next` can be too late. To add timing at the last safe
moment, register a hook before calling the handler. Inside a middleware's
returned function, import `fmt` and `time` alongside the request package:

```go
started := time.Now()
request.Hooks(r).BeforeWriteHeader(func(w http.ResponseWriter, _ int) {
	w.Header().Set("Server-Timing",
		fmt.Sprintf("app;dur=%d", time.Since(started).Milliseconds()))
})

return next(w, r)
```

`BeforeWriteHeader` runs once before the final status, not for informational
`1xx` except `101`. A body write or flush can commit an implicit `200`.
`BeforeWrite` observes an attempted body write, which may still fail afterward.
Do not record raw body contents merely to measure writes.

For work that observes handling completion, register an after-response callback.
With an existing application metrics function `recordOutcome`:

```go
request.Hooks(r).AfterResponse(func(err error) {
	recordOutcome(err)
})
```

Callbacks run in reverse registration order when router handling ends,
including panic unwinding. Their argument is the handler's returned error, not a
delivery acknowledgment; an unrecovered panic can leave it nil. A hijacked
connection can outlive these callbacks and remains the handler's responsibility.

`request.Hooks` requires the context installed by `router.Handler.ServeHTTP`,
including normal routed requests. Outside that lifecycle it panics with
`request.ErrNoHooksMiddleware`; there is no non-panicking public lookup. Only
register hooks where wiring guarantees that lifecycle.

## Preserve Context

Middleware can resolve application state with `router.Provide`. Here
`workspaces.FromHost` validates a hostname against trusted configuration and
`withWorkspace` derives a context containing that result. Import `context`:

```go
routes.Use(router.Provide(func(w http.ResponseWriter, r *http.Request) (context.Context, error) {
	workspace, err := workspaces.FromHost(r.Context(), r.Host)
	if err != nil {
		return nil, err
	}
	return withWorkspace(r.Context(), workspace), nil
}))
```

An error stops the chain. Derive successful contexts from `r.Context()`, not
`Background` or `TODO`; replacing the parent loses cancellation, lifecycle
hooks, and upstream adapter state. The same rule applies to custom middleware
calling `next` with a changed request. Static values can use `ProvideStatic`,
but mutable captured values still need concurrency-safe ownership.

## Native Middleware

For existing `func(http.Handler) http.Handler` middleware named `compress`, use
the adapter:

```go
routes.Use(router.HTTPMiddleware(compress))
```

Creating the adapter does not call `compress`. Its native constructor runs
once per wrapping, normally during registration, not once per request or once
globally. A single `Methods` or `Any` registration shares a chain across its
methods; separate registrations build separate chains. The unmatched-request
fallback builds one additional chain on first use and reuses it. Native handlers
and captures must therefore support concurrent use.

Returned errors are rendered *inside* native middleware while output remains
uncommitted, so it can observe the error status and body. The error also returns
outward for observation. To map that HTTP response, put the mapper inside the
adapter:

```go
routes.Use(
	router.HTTPMiddleware(compress),
	errormap.Middleware(mappings...),
)
```

An outer mapper still receives the returned error but cannot rewrite the
adapter's committed output. An inner mapper sees only the layers it wraps.
A recovered error rendered before earlier output is not treated as a
post-commit panic; an actual panic after earlier output still follows the
[abort rules](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md#committed-failures).

The adapter tracks writes through `next` and commitment at enclosing native
boundaries, including wrappers hiding `Unwrap`. Output in a native wrapper's
private buffer is not visible until forwarded through a tracked boundary.
Use the supplied writer for normal output; unwrapping it to write bypasses
tracking and hooks. Use `http.ResponseController` for capability operations,
and preserve the incoming context when forwarding requests.

Individual policies belong with their capabilities: [Body Limits](https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md),
[Content Types](https://arkoren.dev/docs/guides/http-fundamentals/content-type.md), [secure headers](https://arkoren.dev/docs/guides/browser-security/secure-headers.md),
[rate limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md), and
[authentication](https://arkoren.dev/docs/guides/authentication/overview.md) explain their setup and boundaries.
For composition and adapters, see the
[`router`](https://pkg.go.dev/arkoren.dev/core/http/router) and
[`request`](https://pkg.go.dev/arkoren.dev/core/http/request) API references.

---

Source: https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md
[HTML documentation](https://arkoren.dev/docs/guides/http-fundamentals/body-limit)

# Body Limits

A body cap bounds how many input bytes a request can make your application
read. Choose a ceiling for the endpoint before decoding: a small invoice JSON
request should not inherit the space allowance needed by an upload route.
The `bodylimit` middleware applies that ceiling before handlers and other
body-reading middleware.

## Set a Ceiling

With a fresh [router](https://arkoren.dev/docs/guides/http-fundamentals/routing.md) named `routes`, import
`arkoren.dev/core/http/middleware/bodylimit` and allow at most 64 KiB:

```go
routes.Use(bodylimit.Middleware(func(limit *bodylimit.Builder) {
	limit.Limit(64 << 10)
}))
```

Register routes afterward. Without a configured limit, the middleware uses
10 MiB, matching the request body helpers' default. Zero and negative
middleware limits also select 10 MiB; they do not disable the cap or require
an empty body.

The limit bounds input bytes, not decoded memory, temporary-file usage from
later processing, or application work. Use validation and processing limits
where decoded input can expand or trigger expensive operations.

## Handle Rejections

If `Content-Length` exceeds the ceiling, the middleware returns
`request.ErrBodyTooLarge` (`413`) before calling the next layer. Otherwise it
wraps the body with `http.MaxBytesReader`: unknown-length or understated bodies
fail only when downstream code reads far enough. The middleware does not
pre-read or verify an unread remainder.

The [request helpers](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#decode-json) convert a native limit-read error
to the same `413` problem. If your code reads `r.Body` directly, handle
`*http.MaxBytesError` yourself. In a raw-reader fragment using `io`, `errors`,
`net/http`, and `arkoren.dev/core/http/request`:

```go
data, err := io.ReadAll(r.Body)
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
	return request.ErrBodyTooLarge
}
```

Other read failures remain errors to handle before using `data`. Returning the
raw `MaxBytesError` alone does not give it the helper's HTTP problem identity.
Return parsing and size errors before committing a response; after output
starts, neither a `413` nor a malformed-input problem can replace it.

A nil body is empty input, including with request helpers: raw reads return
no data, while JSON and XML return their malformed-body problems. The
declared-length check still runs first, even for a nil body.

Middleware puts the cap in context and replaces the body only on a shallow
request copy. It does not replace the caller's context/body fields, read the
body, or close it. The underlying non-nil stream remains shared: downstream
reads consume it, and closing the wrapper closes that stream.

## Scope Larger Inputs

An outer ceiling cannot be raised by another `bodylimit` middleware,
`request.MaxSize`, or `request.Unlimited`. Put larger routes outside the
smaller scope from the start. On a fresh router without the global cap above,
with existing `createInvoice` and `uploadAttachment` handlers:

```go
routes.Group("/invoices", func(invoices *router.Router) {
	invoices.Use(bodylimit.Middleware(func(limit *bodylimit.Builder) {
		limit.Limit(64 << 10)
	}))
	invoices.Post("/", createInvoice)
})
```

The ordinary endpoint now has its own cap. Add the upload as a separate scope,
not a child of that group:

```go
routes.With(bodylimit.Middleware(func(limit *bodylimit.Builder) {
	limit.Limit(12 << 20)
})).Post("/attachments", uploadAttachment)
```

That route allows 12 MiB, subject to any additional parser limit. Apply the
middleware before anything that reads the body. A webhook receiver also has
its own [body policy](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md#body-limits); any surrounding
middleware remains a ceiling above it.

## Tighten One Read

Inside a cap, a handler can ask its decoder to accept less. With an existing
`CreateInvoice` input struct:

```go
input, err := request.JSON[CreateInvoice](r, request.MaxSize(16<<10))
```

`MaxSize(0)` permits only empty input, unlike middleware `Limit(0)`.
A negative per-call size selects the 10 MiB default. Options apply in order;
a later size option replaces an earlier per-call setting, but no option
removes the upstream reader's cap.

`Unlimited()` removes the helper's own cap and ignores a stored context limit,
not a ceiling already installed on the body. Use it only when memory use is
bounded another way; uncapped raw reads let a hostile client exhaust memory.

For configuration and per-call options, see the
[`bodylimit`](https://pkg.go.dev/arkoren.dev/core/http/middleware/bodylimit) and
[`request`](https://pkg.go.dev/arkoren.dev/core/http/request) API references.

---

Source: https://arkoren.dev/docs/guides/http-fundamentals/content-type.md
[HTML documentation](https://arkoren.dev/docs/guides/http-fundamentals/content-type)

# Content Types

A JSON decoder accepts JSON-looking bytes without checking the request's
`Content-Type`. That allows a mislabeled `text/plain` request to reach the same
decoder. A hostile page can send some form or plain-text browser requests
without a CORS preflight, sometimes with your cookies depending on their scope
and `SameSite` policy. Reject formats your endpoint does not accept before
reading the body.

The `contenttype` middleware checks the declared media type, not the contents
or the caller's identity. It complements decoding, [body limits](https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md),
and [CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md); it replaces none
of them.

## Require JSON

On a fresh [router](https://arkoren.dev/docs/guides/http-fundamentals/routing.md) named `routes`, import
`arkoren.dev/core/http/middleware/contenttype`:

```go
routes.Use(contenttype.Middleware())
```

Register JSON routes after the policy. By default it allows `application/json`
and any media type ending in `+json`, such as `application/merge-patch+json`.
Missing, malformed, or disallowed headers on requests carrying data return
`contenttype.ErrUnsupportedMediaType` (`415`) without calling the next layer.
The router renders that problem if output has not started.

Matching ignores case and valid media parameters such as `charset=utf-8`.
Malformed parameters are rejected. An allowed header does not prove valid JSON;
the [request decoder](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#decode-json) still checks bytes and structure.

## Check Empty Requests

The decision uses `ContentLength`, not a speculative read from the body:

- A nonzero length, including unknown length `-1`, requires an allowed header
  regardless of method.
- At length zero, POST, PUT, PATCH, DELETE, and QUERY reject a supplied header
  when it is malformed or disallowed. A missing header passes.
- Other methods at length zero pass without this media-type check.

A bodyless GET therefore does not require `Content-Type`, while an empty POST
declaring `text/plain` is rejected. This prevents a disallowed form format from
bypassing the policy merely because its body is empty. Do not use method or
media type as a substitute for authorization.

## Allow Other Formats

If one endpoint receives multipart uploads, give it a separate policy rather
than adding multipart to every JSON route. On a router without the global JSON
policy above, with an existing `uploadAttachment` handler:

```go
routes.With(contenttype.Middleware(func(content *contenttype.Builder) {
	content.Allowed("multipart/form-data")
})).Post("/attachments", uploadAttachment)
```

`Allowed` replaces the default list. Use `application/x-www-form-urlencoded`
for a form endpoint, or list both it and `application/json` when both formats
are intentional. An entry starting with `+`, such as `+xml`, matches every
media type with that suffix. An empty allowed list restores the JSON defaults;
it does not create a deny-all policy. Narrow defaults to only `application/json`
when the endpoint should not accept other JSON-based media types.

Allowing form media types allows cross-site form requests through this check.
Keep CSRF protection on cookie-authorized browser routes. The multipart parser
still validates boundaries and input separately; see
[Parse Uploads](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#parse-uploads).

## Customize Failures

For a more useful client explanation without changing status or identity,
derive the default problem. Import `arkoren.dev/foundation/problem`:

```go
failure := contenttype.ErrUnsupportedMediaType.Derive(func(details *problem.Builder) {
	details.Detail("Send application/json for invoice drafts.")
})
```

Pass it in place of the earlier middleware registration, before routes:

```go
routes.Use(contenttype.Middleware(func(content *contenttype.Builder) {
	content.ErrorResponse(failure)
}))
```

Only the complete zero `problem.Details{}` selects the default problem. A
nonzero custom problem with an absent status retains its members and receives
the renderer's default `500`, not `415`. Choose an explicit status if changing
the HTTP outcome. [Error Handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md#negotiate-error-output)
owns negotiation and rendering defaults.

## Observe Rejections

With an existing application `app`, add its bus to the same policy construction
callback. Do not stack another policy merely to enable events:

```go
routes.Use(contenttype.Middleware(func(content *contenttype.Builder) {
	content.Events(app.Events())
}))
```

The default nil bus publishes nothing. When configured, every rejection emits
`contenttype.Rejected` (`http.contenttype.rejected`), regardless of a custom
problem's status. Its payload contains method, path, and the lower-case parsed
base media type without parameters. Missing or unparseable base types produce
an empty value; malformed parameters can still leave a parsed base type.

The client controls this metadata, and paths can contain personal identifiers.
Review it before forwarding to external systems. Treat it as untrusted text,
keep listeners cheap, and put [rate limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) before this
policy when rejection traffic needs a bound. For event payloads, see
[HTTP Events](https://arkoren.dev/docs/guides/events/http-events.md#request-rejections).

For all policy options, see the
[`contenttype` API reference](https://pkg.go.dev/arkoren.dev/core/http/middleware/contenttype).

---

Source: https://arkoren.dev/docs/guides/http-capabilities/sessions.md
[HTML documentation](https://arkoren.dev/docs/guides/http-capabilities/sessions)

# Sessions

A cart selection or signed-in identity must survive the next HTTP request. A
**session** keeps that state on the server, in your [cache](https://arkoren.dev/docs/guides/cache/overview.md);
the browser carries only a random session ID in a cookie. Possession of that
cookie grants access to the session, so treat it as a bearer credential.

## Enable Sessions

Use an existing `*cache.Cache` named `store` and router named `routes`. Choose
the cache's [driver and lifecycle](https://arkoren.dev/docs/guides/cache/drivers.md) first. An in-process cache
is sufficient for local development; replicas need shared storage to see the
same records. Sharing a cache does not make its data durable or prevent eviction.

Import `arkoren.dev/core/http/session` and install the middleware before routes
that use it:

```go
routes.Use(session.Middleware(store))
```

The middleware loads a session before the handler and saves changed state before
the first final response headers. A new, unused session normally creates neither
a cache record nor a cookie; automatic renewal can itself mark state changed.
Missing, malformed, unknown, or retired IDs start fresh sessions. A cache read
failure instead fails the request, rather than silently replacing known state.

The default cookie is `__Host-arkoren.session`: `Secure`, `HttpOnly`, `Path=/`,
no `Domain`, and `SameSite=Lax`. The host prefix prevents sibling subdomains from
setting that cookie in conforming browsers. Setting a domain or narrower path
changes that trust boundary and, with an automatically selected name, drops the
host prefix. Explicit prefix names must satisfy their browser rules or
construction panics.

Register [CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md) before
authentication and state-changing handlers. `HttpOnly` limits script access;
`SameSite=Lax` is not a replacement for CSRF protection. Do not log cookie values,
`Session.ID()`, or `OriginalID()`.

For local plain HTTP only, use this alternative registration:

```go
routes.Use(session.Middleware(store, func(cookies *session.Builder) {
	cookies.AllowInsecure(true)
}))
```

This selects the unprefixed default cookie name. Never enable it in production:
HTTP exposes the bearer ID in transit. Do not rely on browser-specific localhost
exceptions for secure cookies.

## Store Values

Share a **field**, an immutable declaration of a storage name and Go type,
between handlers that read and write the same value. For a cart selected through
[validated request input](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#validate-input), define
the application's writable shape and field:

```go
type CartSelection struct {
	CartID string `json:"cart_id"`
}

var CartID = session.NewField[string]("cart_id")
```

In the handler, `input` is an already decoded, validated `CartSelection`; verify
that the visitor may use that cart before storing its ID. With
`arkoren.dev/core/http/request` imported and incoming request `r`:

```go
current := request.MustSession(r)
if err := current.Put(CartID, input.CartID); err != nil {
	return err
}
```

`Put` changes this request's session, not the cache. It does not authorize the
cart or confirm persistence. `MustSession` panics if middleware is missing; use
`request.Session(r)` and its `ok` result for code that also runs without sessions.

In a later handler, read the same field. Import `errors` to distinguish absence
from an incompatible value:

```go
current := request.MustSession(r)
saved, err := current.Get(CartID)
switch {
case errors.Is(err, session.ErrKeyNotFound):
	// No cart has been selected.
case err != nil:
	return err
default:
	// saved is the stored string; authorize access before loading the cart.
}
```

Use `current.Delete(CartID)` to remove the selection and check its error before
continuing. Deleting an absent valid field succeeds and still marks state
changed. Empty or zero-value fields return `ErrInvalidField` without mutation.

Field names are literal: no trimming, normalization, or automatic prefixing.
Different declarations with the same name address the same entry, even with
different types; they do not create a storage schema. `Name()` reveals the name,
while ordinary formatting and structured logging redact it. Keep the reserved
`arkoren:auth:user` name for authentication.

### Value Ownership

Runtime values must already have the field's Go type. A differently typed number
or map returns `ErrInvalidValueType`; supplying `encoding/json/jsontext.Value`
does not turn runtime input into persisted JSON. Only fields loaded from stored
records permit typed JSON decoding, which preserves integer precision. Replacing
a loaded field with `Put` restores runtime type checking until the next save and
load. Use JSON-compatible values and share field declarations across readers.

A present nil interface is not an absent field: interface fields can return
`nil, nil`, while concrete fields reject that value. Typed nil pointers retain
their pointer type.

`Put` retains values without copying them. The mutex protects session state, not
nested maps, slices, or pointed-to objects. Do not mutate borrowed values
concurrently with access or persistence; call `Put` again after a mutation to
mark it changed. Fields have no separate TTL.

For standalone tests, `session.New(expiresAt, initial)` creates changed state but
does not attach or save it. Do not use a zero `Session`. `New` snapshots only the
outer input map; do not mutate that input during construction. A nil map creates
empty writable state, and nested values remain shared. `All` returns only a shallow
copy and exposes loaded values as borrowed `jsontext.Value` slices. Passing that
copy to `New` makes them runtime input again, not persisted fields.

## Group Fields

When a feature owns several fields of one type, declare a namespace rather than
repeating its prefix:

```go
cart := session.NewNamespace[string]("cart")
coupon := cart.Field("coupon")
err := current.Put(coupon, "WELCOME")
```

The field's physical name is `cart:coupon`; it behaves like any standalone
`Field[string]`. This adds neither expiry nor authorization. To adopt a namespace
without migrating data, preserve each existing physical name.

`Derive` creates an independent definition. If existing names use dots, an
alternative definition can retain that convention:

```go
legacy := cart.Derive(func(space *session.NamespaceBuilder) {
	space.Separator(".")
})
city := legacy.Subnamespace("shipping").Field("city")
```

`city.Name()` is `cart.shipping.city`. A subnamespace joins with its parent's
separator before its own callbacks configure future joins. Empty separators
concatenate names. Names are not escaped: `a:b` joined with `c` collides with `a`
joined with `b:c`. Validate or encode untrusted components and convert numeric
IDs explicitly. Invalid ancestors retain their first `ErrInvalidField` through
derivation; fields inherit it. Construction callbacks run once in order, nil
callbacks panic, and retained drafts cannot change published definitions.
Namespace formatting also redacts names; `Name()` deliberately reveals them.

This field namespace is distinct from the middleware's cache namespace, which
names whole records. To customize record names, pass `Builder.Namespace` a
`cache.Namespace[jsontext.Value]`, as explained under
[cache key policy](https://arkoren.dev/docs/guides/cache/keys.md). The default produces
`arkoren:session:ID`. Choose collision-free literal names; a zero or invalid
namespace panics after configuration. Session lifetime overrides its TTL.

## Change Identity

Rotate the ID after login or privilege changes to prevent **session fixation**:
an attacker arranging for a victim to authenticate into a known session. For
login, use the same `auth.SessionIdentity` as your authentication middleware;
[Session Login](https://arkoren.dev/docs/guides/authentication/session-login.md) owns the user loader,
credential checks, and complete identity setup.

After credentials succeed, with the configured `identity` and typed loaded `user`:

```go
if err := identity.Login(request.MustSession(r), user.ID); err != nil {
	return err
}
```

Login stores the identifier, then regenerates the ID only if that write succeeds.
For other privilege changes, call `current.Regenerate()` after successfully
changing the authentication-related session state. Regeneration keeps the fields
and original creation time; it does not log the visitor out.

To remove the identity but retain unrelated fields:

```go
if err := identity.Logout(request.MustSession(r)); err != nil {
	return err
}
```

Logout removes even a stale, wrongly typed identifier and rotates the ID. To end
the entire session instead, call `current.Destroy()` before response headers.
Destroy discards later changes. `Clear` only clears fields; it neither rotates
nor retires the ID.

None of these calls performs storage I/O. On a non-5xx response, middleware saves
a regenerated record and issues its cookie before retiring the original ID.
Destroy retires the original and expires the cookie, even if retirement fails.
A failed save leaves the previous session intact; a failed retirement can leave
the old ID usable. A successful login/logout HTTP response alone therefore does
not prove the storage transition succeeded.

## Save Changes

Finish session mutations before [response commitment](https://arkoren.dev/docs/guides/http-fundamentals/responses.md#response-commitment),
including before [streaming](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md).
Persistence runs before the first final status, not after every mutation. A
`5xx` status skips persistence; hijacked connections such as WebSocket upgrades
do not save changes. This does not roll back database effects or undo state
already saved before a later response failure.

Retirement replaces the old record with a **tombstone**, a marker that rejects
reuse. Loaded sessions re-read their original record, then atomically compare
those exact bytes and write the current ID using the cache driver's
`CompareAndPut`. A missing, retired, or subsequently changed original drops the
save without a cookie or retry. The same guard protects a proposed regenerated
ID. Guard rejection is not a persistence error.

Ten-second per-ID [cache leases](https://arkoren.dev/docs/guides/cache/locks.md) reduce conflicts; the atomic
guard, not the lease, prevents stale revival after lease expiry. Ordinary
concurrent requests otherwise save whole snapshots in lease order:
last-write-wins can lose independent field edits. Retirement neither revokes a
different ID committed earlier nor cancels a handler's completed effects.

> [!WARNING]
> Upgrade every session writer and drain older requests before relying on guarded
> retirement. An old unconditional writer or restoration of an earlier live
> record can bypass it. Custom cache drivers must implement atomic
> `CompareAndPut`, never emulate it with separate reads and writes or fall back
> to `Put`. The rollout changes no JSON format, key names, cookie names, or TTL
> settings.

Tombstones live for `MaxLifetime`, or the idle TTL when the maximum is disabled.
Missing originals also reject guarded writes; marker retention is not a promise
to revoke every independently issued session.

Primary persistence keeps request context values but ignores its cancellation
and deadline, applying `PersistenceTimeout` instead: five seconds by default.
Each best-effort lease release has a separate ten-second cleanup budget and
falls back to lease expiry on failure. Cleanup can delay headers beyond the
primary timeout. These bounds require a driver that honors context; synchronous
error handlers and event listeners are unbounded by them, so keep callbacks fast.

## Set Lifetimes

The idle lifetime defaults to two hours from creation or renewal. Before response
commitment, a live session renews when strictly less than 15 minutes remain,
not on every request. The absolute maximum is 24 hours from original creation.
With `time` imported, choose a different policy at construction:

```go
routes.Use(session.Middleware(store, func(cookies *session.Builder) {
	cookies.TTL(time.Hour)
	cookies.MaxLifetime(7 * 24 * time.Hour)
}))
```

A positive maximum caps initial expiry, renewal, and `Session.Extend` at the
original creation time plus that maximum. Regeneration does not reset it. The
cookie expiry and record TTL use the cap; capping unused state alone does not
create a record or cookie.

Equality counts as expired. If the absolute deadline is reached during a handler,
middleware discards its record write and new/refreshed cookie rather than
granting the same authenticated state a fresh lifetime. An expired loaded session
is replaced on the next request without its old fields. An idle deadline reached
during a handler can still trigger regeneration and renewal, subject to the
absolute maximum. No nonpositive remaining TTL is passed to the cache as its
zero-TTL/no-expiry policy.

Zero restores defaults for `TTL`, `MaxLifetime`, `ExpirationDelta`, and
`PersistenceTimeout`; session TTL zero never means forever. Negative
`MaxLifetime` disables the absolute cap; negative `ExpirationDelta` disables
near-expiry renewal. Final negative `TTL` or `PersistenceTimeout` panics after
all configuration callbacks. A later callback can replace an earlier invalid
setting before validation.

## Observe Failures

Save, verification, lease-acquisition, and retirement errors do not themselves
change the response status. They are discarded unless you configure observation.
Use your access-controlled `*slog.Logger` named `logger` and optional application
event bus `events` in this alternative registration:

```go
routes.Use(session.Middleware(store, func(cookies *session.Builder) {
	cookies.ErrorHandler(func(err error) {
		logger.Error("session persistence failed", "error", err)
	})
	cookies.Events(events)
}))
```

The error callback receives unredacted diagnostics, including backend causes
that can contain session keys. Apply your own redaction and retention policy.
This is different from the field-name formatting protections above.

Events report observed outcomes: `session.expired` on replacement of expired
state, `session.destroyed` only after successful retirement, and
`session.persist.failed` for persistence failures. A failed destroy retirement
emits the failure with operation `retire`, not the destroyed event. Payloads
contain no session IDs, data, or storage error.

Destroyed/failure event paths use the registered route pattern without its
method prefix, never expanded path parameters or queries. Host prefixes remain;
unmatched patterns produce an empty path. Keep secrets out of route literals.
Contexts still retain caller values, and callback diagnostics remain unredacted;
event payload limits do not sanitize either. See
[HTTP events](https://arkoren.dev/docs/guides/events/built-in-events.md#http) for payloads.

For all fields, lifecycle operations, and settings, see the
[`session` API reference](https://pkg.go.dev/arkoren.dev/core/http/session).

---

Source: https://arkoren.dev/docs/guides/http-capabilities/localization.md
[HTML documentation](https://arkoren.dev/docs/guides/http-capabilities/localization)

# Localization

A signup form can use the same validation rules for every visitor while showing
messages in each visitor's language. The `locale` middleware selects a language
once per request and puts it, with a **catalog** of message templates, in the
request context. Validation uses that context to render messages; your handlers
can use the language for their own text.

Arkoren does not provide a general translation engine, date/number formatting,
or plural handling. Keep application text in your own catalogs or translation
library. This page connects language selection to messages; the
[validation error guide](https://arkoren.dev/docs/guides/validation/errors.md#translate-messages) owns error
conversion and relocalization.

## Build Catalogs

A `validate.Catalog` names a default language and maps supported language tags to
`validate.Messages`. Each message maps a stable violation code, such as
`string.min`, to plain text with optional `{param}` placeholders. Import
`arkoren.dev/foundation/validate` and define a small Spanish translation:

```go
spanish := validate.Messages{
	"required":     "es obligatorio",
	"string.min":   "debe tener al menos {min} caracteres",
	"string.email": "debe ser un correo electrónico válido",
}
```

For `string.min`, `{min}` comes from the violation's `min` parameter. A missing
parameter leaves its placeholder visible, so test spelling. Lists render with
commas and times use RFC 3339. Templates do not include the field name; the
validation error's path identifies the input. Use number-independent phrasing
when a language needs plural forms the template system cannot express.

Add English as the default, using the independently mutable copy returned by
`validate.English()`:

```go
catalog := validate.Catalog{
	Default: "en",
	Locales: map[string]validate.Messages{
		"en": validate.English(),
		"es": spanish,
	},
}
```

Catalog keys determine which languages clients can select. The Spanish catalog
can be incomplete: missing messages fall back rather than disabling the language.

## Select Languages

With an existing router `routes`, import
`arkoren.dev/core/http/middleware/locale` and register middleware before routes
that validate or use translated text:

```go
routes.Use(locale.Middleware(func(language *locale.Builder) {
	language.Catalog(catalog)
}))
```

This uses `Accept-Language`; query and cookie selectors are disabled by default.
For example, `Accept-Language: es-MX,es;q=0.9,en;q=0.5` selects the catalog key
`es` and sets `Content-Language: es`. Without locale middleware, validation uses
built-in English and `locale.FromContext` returns an empty string.

If links should force a language and your interface remembers a visitor's choice,
use this alternative registration:

```go
routes.Use(locale.Middleware(func(language *locale.Builder) {
	language.Catalog(catalog)
	language.QueryParam("lang")
	language.Cookie("language")
}))
```

The middleware tries the first supported choice in this order:

1. The named query parameter, such as `?lang=es`.
2. The named cookie, such as `language=es`.
3. `Accept-Language`, ordered by client preference.
4. The catalog default, or `en` if it is empty.

An absent or unsupported query value falls through rather than blocking the
cookie/header. Empty selector names disable those sources. Cookie detection only
reads: your application writes the preference cookie and chooses its security
attributes and lifetime. There is no locale cookie-lifetime API. Client language
preferences are presentation input, not proof of identity or permission.

Matching ignores case, surrounding whitespace, and `_` versus `-`, then tries
the base language. A request for `es-MX` can select `es`; a request for `es` does
not select an `es-MX`-only entry. Add base-language entries when they should be
selectable. If nothing matches, the default remains usable even without an entry
in `Locales`.

### Header Weights

Header candidates use descending `q` weights; omitted weights are `1` and ties
retain header order. Thus `fr;q=0.5, es;q=0.9, en;q=0.9` tries `es`, `en`, then
`fr`.

Weights accept HTTP decimal syntax: `0` or `1`, optionally a decimal point and
up to three digits; digits after `1` must be zero. `0.9` and `1.000` are valid;
`.9`, `+0.9`, `1.001`, `0.1234`, exponent/hexadecimal forms, and nonfinite values
are not. Missing or duplicate `q` parameters invalidate that entry. Malformed
language tags, wildcards, invalid weights, and `q=0` entries are skipped, not
request errors. Skipping a candidate does not veto the configured fallback.

### Configuration Checks

The completed catalog and nonempty cookie name are checked after every
construction callback has run. Tags require one to eight ASCII letters followed
by optional one-to-eight-character alphanumeric subtags, separated with `-` or
`_`; surrounding ASCII whitespace is ignored for matching. This is a shape
check, not full BCP 47 validation or a registry lookup.

Invalid tags, duplicate normalized keys such as `en-US` and `EN_us`, and cookie
names that fail `net/http.Cookie.Valid` panic during construction. An empty
catalog/default is valid. Callbacks run once in order, so a later callback can
replace an earlier invalid setting; nil callbacks panic. Finish loading messages
before construction: middleware copies both catalog map layers, and later edits
or retained builders do not reconfigure it.

## Read Languages

Use the selected language for application-owned text. In a handler with incoming
request `r`, `w http.ResponseWriter`, and imports `net/http` and
`arkoren.dev/core/http/response`:

```go
switch locale.FromContext(r.Context()) {
case "es":
	return response.String(w, http.StatusOK, "Hola")
default:
	return response.String(w, http.StatusOK, "Hello")
}
```

The Spanish request above receives `Hola`. The default branch also covers tests
or callers without middleware. For a larger catalog, use your translation
library with this language rather than growing a switch for every message.

A match preserves the original catalog spelling. A key `ES_mx` becomes context
value `ES_mx`, but response header `Content-Language: ES-mx`: headers trim
whitespace and replace underscores without changing case. The default uses its
catalog spelling when an exact normalized key exists; otherwise it preserves
`Catalog.Default`, or uses `en`. Your application text lookup must handle that
fallback even when it has no validation catalog entry.

The middleware adds `Accept-Language` to `Vary`, and `Cookie` when cookie
selection is enabled. Error responses carry these headers too. Query selection
is part of the URL, not an additional `Vary` header. Ensure any proxy cache also
keys on the relevant query values; these headers do not make personalized
responses safe for shared caching.

## Translate Messages

Existing request validation renders violations with the request's catalog and
language. Pass `r.Context()` to schemas called directly rather than substituting
a background context. An existing signup route rejecting a short password can
produce this Spanish violation:

```json
{
  "code": "string.min",
  "params": {
    "min": 12
  },
  "message": "debe tener al menos 12 caracteres"
}
```

Only the message changes. Codes, params, paths, and the HTTP problem title stay
stable for programmatic clients. See
[translated validation errors](https://arkoren.dev/docs/guides/validation/errors.md#translate-messages) for
the response boundary and `Errors.Localize`; this page does not replace your
schema or [diagnostic ownership policy](https://arkoren.dev/docs/guides/validation/errors.md#diagnostic-ownership).

For an application rule with stable code `username.reserved` and public parameter
`name`, add its message before middleware construction:

```go
catalog.Locales["es"]["username.reserved"] = "{name} está reservado"
```

The parameter is inserted into the visible message. Never put credentials or
other private values in message params. Use application-specific code prefixes
to avoid future built-in collisions; [custom rules](https://arkoren.dev/docs/guides/validation/custom-rules.md)
owns producing the violation.

Conditional messages can use specialized keys such as `required_if:eq`. A plain
`required_if` translation is sufficient when every condition should use the same
wording: it takes precedence over the built-in English variant. Add specialized
translations when the condition needs a more precise sentence.

## Handle Fallbacks

Message lookup differs from language negotiation. For an ordinary code it tries:

1. The request language, then its base language.
2. The catalog default, then its base language.
3. Built-in English.
4. The code itself if no template exists.

A regional catalog can therefore override a few messages while its base catalog
supplies the rest. Lookup uses stored spellings, not negotiation's case folding:
keep regional/base keys consistent, such as `es-MX` with `es`. A partly translated
response can contain default-language or English messages even though its
`Content-Language` reports the selected language.

A missing custom translation can appear as `"message": "username.reserved"`.
The code remains valid for clients, but the reader sees an identifier. Test
custom codes as well as built-in ones rather than assuming English knows every
application message. Relocalizing existing errors preserves their original
message when no template is known.

## Load Translations

For larger catalogs, keep each language in a JSON object mapping codes to
templates. For example, the application file `locales/es.json` can contain:

```json
{
  "required": "es obligatorio",
  "string.min": "debe tener al menos {min} caracteres"
}
```

Embed the files with the standard `embed` package at package scope:

```go
//go:embed locales/*.json
var translations embed.FS
```

At startup, decode each file with `encoding/json/v2` before registering middleware:

```go
data, err := translations.ReadFile("locales/es.json")
if err != nil {
	return err
}
messages := validate.Messages{}
if err := json.Unmarshal(data, &messages); err != nil {
	return err
}
catalog.Locales["es"] = messages
```

Fail startup on missing or malformed files instead of silently dropping a
language. Middleware owns its copied catalog, not later mutations to these maps.

## Check Responses

Send requests through your configured router, not only directly to a schema, to
verify language selection and response headers together. With `net/http/httptest`
and the `/greeting` route using the handler above:

```go
incoming := httptest.NewRequest(http.MethodGet, "/greeting?lang=es", nil)
incoming.Header.Set("Accept-Language", "en")
result := routes.Record(incoming)
```

With query selection enabled, expect `Content-Language: es` and body `Hola`
despite the English header. The caller owns `result.Body` and must close it after
reading. Also test an unsupported query falling through to cookie/header, header
weights, and a missing translation. Compare your supported rule codes with each
language's templates; `validate.English()` supplies built-in keys, not your custom
code inventory.

For selection settings and message APIs, see the
[`locale` reference](https://pkg.go.dev/arkoren.dev/core/http/middleware/locale)
and [`validate` reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/http-capabilities/secure-links.md
[HTML documentation](https://arkoren.dev/docs/guides/http-capabilities/secure-links)

# Signed Links

A password-reset email needs a link that works without a login but rejects an
altered account ID. A **signed link** authenticates its path and query using a
secret key. An expiry limits how long the link is accepted; a cache can record
single use, and a server-side binding can invalidate links when account state
changes.

A valid signature proves that the URL matches what a key holder signed. It does
not identify the person opening it or independently authorize access to the
named resource. Decide what possession of the link permits, authorize issuance,
and recheck application constraints at the action boundary. Anyone who obtains
the URL can present it: keep signed queries out of logs, analytics, and referrers,
and deliver links over HTTPS.

## Create Signers

Provision a separate random 32-byte shared key using
[application key tooling](https://arkoren.dev/docs/guides/keys/overview.md). For example, generate the
application file `link-key.key` with `arkoren key generate shared --out
link-key.key`; keep it outside source control and restrict access. Import
`arkoren.dev/foundation/key` and `arkoren.dev/core/http/link`, then load the key
at startup:

```go
secret, err := key.LoadShared("link-key.key")
```

Handle loading failure before constructing the signer. For email links, give it
your configured public origin rather than deriving that origin from an untrusted
request host:

```go
signer, err := link.New(secret, func(links *link.Builder) {
	links.Origin("https://app.example.com")
})
```

Handle construction errors before issuing links or installing middleware. A zero
key returns `ErrInvalidSecret`. Origins must be HTTP(S) scheme/host origins,
without credentials, query, fragment, or an application path; invalid values
return `ErrInvalidOrigin`. Use HTTPS in production.

Without `Origin`, a signer accepts root-relative paths and returns relative URLs.
With an origin, it completes root-relative targets and accepts absolute URLs only
for that origin. Fragments, credentials, opaque URLs, and unsuitable targets
return `ErrInvalidURL`. A configured origin scopes link reconstruction; it is
not a substitute for your server's host-routing policy.

Construct once and share the signer. It detaches owned key material and
query-name containers, so retained drafts cannot reconfigure it. Cache/event
services remain shared and application-owned; the signer does not start or stop
them. Request-binding functions and captures must be safe for concurrent use.

## Issue Links

After the application authorizes a reset request, build the target from a typed
account reference. With `net/url` imported, avoid concatenating raw query input:

```go
type ResetTarget struct {
	UserID string
}

target := ResetTarget{UserID: "user-42"}
query := url.Values{}
query.Set("user", target.UserID)
path := "/reset?" + query.Encode()
```

The target identifies the account selected by your application, not arbitrary
request fields that the signing service trusts automatically. Use `Temporary`
with `time` imported to issue a short-lived link:

```go
signed, err := signer.Temporary(path, 15*time.Minute)
```

The result is an absolute URL under `https://app.example.com` with `expires` and
`signature` query parameters. The signature covers the path, ordinary query
values including `user`, and expiry; changing protected values invalidates it.
Signing replaces pre-existing reserved parameters. It does not encrypt query
values, so never put passwords or other secrets in them.

Temporary lifetimes must be positive; zero/negative returns `ErrInvalidExpiry`,
not a default. Expiry uses Unix-second precision and the verifier's local clock;
equality counts as expired. Synchronize verifier clocks rather than treating the
TTL as a cross-server timing guarantee. `Sign` creates permanent links, which
cannot be consumed and have no automatic expiry.

If an email system appends tracking parameters, an alternative signer can exclude
specific names:

```go
signer, err := link.New(secret, func(links *link.Builder) {
	links.Origin("https://app.example.com")
	links.IgnoreQuery("utm_source", "utm_medium")
})
```

Anyone can add, change, or remove ignored values without invalidating the link.
Never derive account selection, redirect destinations, authorization, or bindings
from them. Empty names and reserved `signature`/`expires` names return
`ErrInvalidIgnoreQuery`. Signing an arbitrary redirect destination does not make
that destination safe; validate allowed destinations before issuance and use.

## Protect Routes

Use `link.Middleware` on a display-only GET. With an existing router `routes`
and your application-owned `showResetForm` handler:

```go
routes.With(link.Middleware(signer)).Get("/reset", showResetForm)
```

The handler runs only after signature and expiry checks. Middleware returns
`403 Invalid Link` for invalid authentication and `403 Link Expired` for
authentic expired links. With a configured cache, it also returns
`403 Link Already Used` for a consumed temporary link. Cache-read failures pass
to application error handling; they are not proof that the link was forged.

The signature covers the URL, not the HTTP method. Register explicit methods,
not `Any`. If GET and POST intentionally share a path, GET must only display the
form; POST must independently verify and consume before changing state. This
also keeps email scanners and link previews from spending a reset link.

Verification uses the path as the router sees it; stripping a prefix can break
the signed path. Request verification reconstructs absolute links using the
configured origin, not forwarded host headers. Preserve the signed path and
query through proxy routing and form submission.

Outside middleware, `signer.Verify(signed)` checks authenticity and expiry only.
Use `errors.Is` to classify `ErrInvalidURL`, `ErrInvalidSignature`, or
`ErrExpired`. Expiry is reported only after authentication. `Verify` neither
checks consumption nor reserves a use, even when a cache is configured.

## Consume Links

Single use requires a cache shared by every server accepting the same links.
Choose [cache storage](https://arkoren.dev/docs/guides/cache/drivers.md) that retains markers for their entire
lifetime; early eviction, data loss, or an in-memory restart lets spent links
work again. Construct this alternative signer with the existing `*cache.Cache`
named `store`:

```go
signer, err := link.New(secret, func(links *link.Builder) {
	links.Origin("https://app.example.com")
	links.Cache(store)
})
```

Middleware only observes consumption. Spend the link in the POST handler after
decoding and validating the proposed password and applying your application's
reset policy, but before committing the password change. Register that
application-owned handler for POST explicitly:

```go
routes.Post("/reset", resetPassword)
```

Inside `resetPassword`, with incoming request `r`, keep all consumption failures
off the action path:

```go
err := signer.ConsumeRequest(r)
if errors.Is(err, link.ErrConsumed) {
	return link.ErrLinkUsed
}
if err != nil {
	return err
}
```

Import `errors` for classification. Only after this succeeds may your
application-owned password update run. The form must post to the URL carrying
the original signed query; `ConsumeRequest` reconstructs it without manual URL
assembly. Use `Consume(ctx, signed)` outside HTTP, or `ConsumeURL` with a parsed
URL.

Consumption verifies the signature, binding, and expiry, then atomically adds a
cache marker. Confirmed concurrent calls sharing that backend have one winner;
the others return `ErrConsumed`. A marker lives until signed expiry plus one
minute for clock differences. That margin retains markers longer; it does not
extend the link's validity or guarantee arbitrary clock-skew tolerance.

> [!WARNING]
> A cache error can occur after a marker was committed but its confirmation was
> lost. Do not perform the action after any consumption error, and do not assume
> a retry can succeed. Confirm application state through your recovery flow or
> issue a new link.

Consumption is not a transaction with the password update. If the update fails
after successful consumption, the link stays spent. If you need an atomic
consume-and-update guarantee, enforce that boundary in application-owned durable
storage; the cache marker does not provide it.

Only temporary links can be spent; authenticated permanent links return
`ErrNotTemporary`. Missing cache configuration returns `ErrNoCache` before
verification of a parsed target. `Consume` first parses its raw URL, so an
unparsable value returns `ErrInvalidURL` even without a cache; `ConsumeURL` and
`ConsumeRequest` check cache configuration before validating their target.

## Bind State

A still-unexpired link might need to stop working after a password change.
**Binding** adds current server-side bytes to the authenticated input without
putting them in the URL. Load the account identified by `target.UserID` as `user`;
its `PasswordHash` is a persisted `[]byte`. Issue a bound variant:

```go
signed, err := signer.Temporary(path, 15*time.Minute, link.Bind(user.PasswordHash))
```

Use the same binding policy for every verification, consumption, and inspection.
Read the current persisted bytes each time, not a stale issuance snapshot:

```go
err := signer.Verify(signed, link.Bind(user.PasswordHash))
```

This succeeds while the stored bytes match; after they change, it returns
`ErrInvalidSignature`. A wrong, missing, or unexpected binding has the same
error. An empty binding is distinct from no binding. `Bind` copies its input;
only a digest participates in the signature.

Choose stable bytes: a stored hash or a dedicated reset-version value, not a
freshly salted recomputation or an inconsistently formatted timestamp. Every
verifier must read the current state from an appropriate consistency boundary.
Replica lag or cached old values can delay rejection; reverting to old bytes
can make old links verify again. Binding is not an atomic lock against a state
change that races with the subsequent action.

For bound GET verification, configure a lookup. Here `users.Find(ctx, id)` is
your application-owned loader returning that account and an error:

```go
binding := func(policy *link.MiddlewareBuilder) {
	policy.BindRequest(func(r *http.Request) ([]byte, error) {
		user, err := users.Find(r.Context(), r.URL.Query().Get("user"))
		if err != nil {
			return nil, err
		}
		return user.PasswordHash, nil
	})
}
```

Import `net/http` for the callback. It runs **before** authentication: use the
query only for a bounded lookup, never as proof of permission, and perform no
state change there. Lookup errors reject with `403 Invalid Link`. The returned
problem retains the cause for application error handling; response detail and
rejection events do not expose it. Redact causes before diagnostic logging.

Install the bound display route instead of the earlier unbound one:

```go
routes.With(link.Middleware(signer, binding)).Get("/reset", showResetForm)
```

This does not configure POST automatically. In the POST handler, load the current
account from the signed query, fail closed on lookup errors, and replace the
earlier unbound consumption call with:

```go
err := signer.ConsumeRequest(r, link.Bind(user.PasswordHash))
```

Keep the same explicit error checks before the password update. Enforce any
required state-version condition in the application's update as well; neither
binding nor single-use markers authorize a different resource or transaction.

## Inspect Links

For a recovery page, inspect an authentic link without spending it. With a
context `ctx` and current bound account:

```go
details, err := signer.Inspect(ctx, signed, link.Bind(user.PasswordHash))
```

On success, `details.Expired` can be true: expiry is descriptive, not an
inspection error. `Previous` identifies a previous signing key; `Consumed`
reports a known cache marker. False does not guarantee usability: it is always
false without a cache or for permanent links, and another request can consume
immediately after inspection. All inspection errors return zero details,
including authentication failures and cache observation errors.

`details.URL` removes `signature` and `expires` but retains application paths,
query values, and ignored tracking values. It is not safe diagnostic text. If
your recovery policy permits reissuance, pass it to `Temporary` with the current
binding; inspection alone is not permission to extend a reset indefinitely.

## Rotate Keys

To accept existing links during rotation, load `current` and `previous` as
separate `key.Shared` values, then use an alternative construction:

```go
signer, err := link.New(current, func(links *link.Builder) {
	links.Origin("https://app.example.com")
	links.Previous(previous)
})
```

Retain any required cache, binding, or ignored-query policy in the replacement
configuration. New links use the current key; old keys verify only. Coordinate
all issuers/verifiers before removing a previous key. Removing it invalidates
its remaining links, including permanent links, which never expire naturally.
If a key leaks, continuing to accept it also accepts forged links made with it;
choose invalidation over a graceful overlap when that risk requires it.

## Observe Outcomes

Configure `Events(events)` on the signer with your application's event bus if
you need outcome counts. Confirmed consumption publishes `link.consumed`;
failures, including uncertain writes and repeat use, publish nothing. Middleware
publishes `link.rejected` for invalid authentication, not authentic expiry or
already-used outcomes. `Verify` and `Inspect` publish neither event.

Payloads omit paths, queries, bindings, and diagnostic errors. They carry
`Method`, which is empty for non-request consumption. Do not replace removed
`Path`/`Err` field reads with full request URLs or causes. Context values remain
available and custom HTTP methods are client-controlled, so select safe
attributes explicitly. HTTP problem `instance` can still contain a request
path; event privacy does not scrub it. See
[signed-link events](https://arkoren.dev/docs/guides/events/http-events.md#signed-links).

`link.Config` is a printable diagnostic record, not an input to `New`. Its
formatting redacts previous keys and omits service pointers, but displays origin
and ignored query names. Keep credentials out of those settings; this does not
sanitize arbitrary signers, URLs, returned errors, or inspection results.

For URL variants, inspection fields, and error contracts, see the
[`link` API reference](https://pkg.go.dev/arkoren.dev/core/http/link).

---

Source: https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md
[HTML documentation](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks)

# Receiving Webhooks

A webhook is an HTTP notification from another service. Unlike a browser request,
it usually has no logged-in user: verify the provider's credentials before acting
on the notification. The `webhook` package authenticates each delivery, decodes
the event you bind, and acknowledges application acceptance.

Three values separate those responsibilities:

- A **source** verifies requests for one provider account and returns its envelope,
  the provider metadata surrounding the still-encoded event payload.
- An **event** associates an exact provider event key with a payload decoder. It
  holds no credentials and works with multiple compatible sources.
- A **receiver** binds events to your application handlers and sets request policy.

Use a [provider adapter](https://arkoren.dev/docs/guides/webhooks/provider-adapters.md) when one is shipped. Constructing
an outbound mail, model, or payment driver neither creates a source nor registers
an endpoint. Your application owns the public route and provider registration.

## Binding Handlers

For Resend delivery notifications, import `arkoren.dev/driver/resend/resendwebhook`
and `arkoren.dev/core/http/webhook`. Configure the endpoint in Resend and load its
signing secret—not the sending API key. See the
[adapter setup](https://arkoren.dev/docs/guides/webhooks/provider-adapters.md#resend) for installation and verification.

```go
source, err := resendwebhook.New(signingSecret)
```

Construction checks the secret's format; verification happens on each request.
Focused fragments keep construction errors visible and assume success. Handle
those errors before using the returned values or registering routes.

With `context` imported and an existing application service `messages`, accept
only the delivered event:

```go
accept := func(ctx context.Context, delivery webhook.Delivery[resendwebhook.Email]) error {
	return messages.RecordDelivered(ctx, delivery.Payload.EmailID)
}
```

`messages.RecordDelivered` is application-owned and must tolerate duplicates.
Return nil only when you have accepted responsibility for the notification.
Every handler error answers 503, even if some work has already committed; core
does not roll back your work. A 503 permits redelivery but cannot make the
provider retry. Use [queued handling](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md#queued-handling) for
slow work rather than making the provider wait.

Bind that function while constructing the receiver:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[resendwebhook.Envelope]) {
	receiver.Handle(resendwebhook.EmailDelivered, accept)
})
```

The handler receives the authenticated delivery ID, exact provider event type,
and decoded email metadata. It does not receive the raw request or credentials.
Each key has one binding; registering it twice is an error, not fan-out.

With your existing router `r`, mount the returned `router.Handler` directly:

```go
r.Post("/webhooks/resend", handler)
```

After acceptance returns nil, the receiver writes 200 with an empty body.
Authenticated unbound events also receive 200 without calling your handler.
Acceptance is not proof that downstream work finished. Choose
[delivery policies](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md) before relying on redelivery,
deduplication, or queue durability.

## Custom Sources

If no adapter implements your provider's protocol, supply a verifier with
`webhook.NewSource`. Core supplies no signature algorithm or replay protection
for custom sources. Prefer the provider's supported verifier rather than inventing
a signature format.

For a hypothetical shipping provider in your application, define the envelope
using `encoding/json/jsontext`:

```go
type Envelope struct {
	ID   string         `json:"id"`
	Kind string         `json:"kind"`
	Data jsontext.Value `json:"data"`
}
```

The provider-specific `verifyAndParse` function below is your code, not an Arkoren
helper. It must authenticate the exact received bytes, implement the provider's
timestamp/replay rules, and return an envelope only after authentication. For a
known invalid request, wrap both `webhook.ErrRejected` and your native sentinel,
for example `fmt.Errorf("%w: %w", webhook.ErrRejected, ErrBadSignature)`.
Leave transient verification failures unclassified: they answer 503 rather than
rejecting a potentially genuine notification.

With `net/http` imported and your account's `secret` already loaded:

```go
source := webhook.NewSource(func(r *http.Request, body []byte) (webhook.Delivery[Envelope], error) {
	envelope, err := verifyAndParse(r, body, secret)

	if err != nil {
		return webhook.Delivery[Envelope]{}, err
	}

	return webhook.Delivery[Envelope]{
		ID:      envelope.ID,
		Type:    envelope.Kind,
		Payload: envelope,
	}, nil
})
```

The verifier receives the bounded body exactly as sent; `r.Body` is already spent.
Do not decode and re-encode the body before verifying a body signature. Headers,
method, URL, and `r.Context()` remain available for protocols that need them.
Do not select a business action or trust payload fields before verification.

The receiver reads at most 1 MiB by default and rejects larger bodies with 413
before the source runs. An outer body-limit middleware can lower that ceiling.
See [body limits](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md#body-limits) before increasing it.

`NewSource` retains the function without calling it or checking credentials.
Your provider constructor must validate credentials; `Source.Err` and `Handler`
only detect a missing verifier. Its function, credentials, and captured clients
are shared across requests and must support concurrent use.

## Typed Events

Keep reusable event definitions separate from configured sources. For the shipping
envelope above, use a small payload with the fields your handler needs:

```go
type Shipment struct {
	ID       string `json:"shipment_id"`
	Tracking string `json:"tracking_number"`
}
```

With `json` imported from `encoding/json/v2`, define the provider's exact key:

```go
var ShipmentDispatched = webhook.NewEvent("shipment.dispatched", func(envelope Envelope) (Shipment, error) {
	var shipment Shipment
	err := json.Unmarshal(envelope.Data, &shipment)

	return shipment, err
})
```

Only a bound, authenticated event reaches this decoder. Decoders must not perform
network I/O; keep remote lookups in acceptance or a worker. Ordinary decoder
errors answer 503, allowing a later schema fix to accept a redelivery. An
authenticated decoder can return `ErrSkip` to acknowledge without acceptance;
the [skip rules](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md#acknowledgement-outcomes) differ from errors
returned by handlers.

Event keys are case-sensitive provider strings, not queue names or diagnostic
categories. Empty keys, control characters, and nil decoders are invalid.
Types connect a handler to its payload, not to a particular account or secret.
Bind `ShipmentDispatched` through `receiver.Handle` in the same way as the shipped
Resend event above.

## Shared Bindings

A library can publish its source constructor and credential-free event
definitions. To share application bindings, return a `webhook.BuilderFunc` rather
than hiding endpoint policy inside the source. With an application-owned
`acceptShipment` of type `func(context.Context, webhook.Delivery[Shipment]) error`:

```go
bindings := func(receiver *webhook.Builder[Envelope]) {
	receiver.Handle(ShipmentDispatched, acceptShipment)
}
handler, err := webhook.Handler(source, bindings)
```

Additional callbacks run once in order on the same construction draft, so the
application can add receipt policy without redefining the binding. Publication
copies binding and rejection containers, not functions or captured dependencies.
There is no receiver lifecycle; the owners of those dependencies manage them.

`Handler` reports invalid sources, events, bindings, or policy with `ErrInvalid`.
At least one event must be bound. The first invalid setting is retained, later
callbacks still run, and a supplied nil callback panics. Fix construction errors
before serving requests rather than waiting for a provider delivery to reveal them.

For every function and option, see the
[`webhook` API reference](https://pkg.go.dev/arkoren.dev/core/http/webhook).

---

Source: https://arkoren.dev/docs/guides/webhooks/provider-adapters.md
[HTML documentation](https://arkoren.dev/docs/guides/webhooks/provider-adapters)

# Provider Adapters

A valid webhook proves only what its provider's protocol authenticates. A body
signature protects different data from Mailgun's signed token or Postmark's Basic
Auth. Select the matching adapter and credentials before treating a notification
as application input.

Each adapter returns a `webhook.Source` and publishes credential-free event
definitions. Bind those definitions using [Receiving Webhooks](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md#binding-handlers).
Outbound drivers do not register endpoints, and their API credentials are not
interchangeable with endpoint verification credentials.

Known authentication failures wrap `webhook.ErrRejected` and answer 400 by
default. A notification that passes verification can still fail envelope or
payload decoding with 503. Neither answer guarantees the provider will retry;
use the [acknowledgement rules](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md#acknowledgement-outcomes)
when deciding your endpoint's failure policy.

## Verification Boundaries

This comparison covers the shipped adapters, not every protocol the providers
offer. All except PayPal verify locally without fetching event resources.
`Delivery.ID` is the identity used by optional core deduplication, not necessarily
the identifier of the email, response, or payment affected.

| Adapter | Authentication | `Delivery.ID` |
| --- | --- | --- |
| [Resend](#resend) | Svix v1 HMAC covering ID, timestamp, and exact body | `svix-id` header |
| [Mailgun](#mailgun) | HMAC covering timestamp + token, not `event-data` | UTC event date + `":"` + event ID |
| [Postmark outbound](#postmark) | Endpoint Basic Auth, not a body signature | `X-PM-Webhook-Trace-Id`, or empty when absent |
| [Postmark inbound](#postmark) | Separate inbound source with endpoint Basic Auth | Inbound `MessageID` |
| [OpenAI](#openai) | SDK verification of exact body and webhook headers | `webhook-id` header, not JSON event ID |
| [Stripe](#stripe) | SDK verification of exact body and `Stripe-Signature` | Native event ID |
| [PayPal](#paypal) | Remote verification of original event and signing metadata | Native event ID, not transmission ID |

Use HTTPS at the public endpoint. Timestamp freshness checks do not prevent
replays inside the accepted window; configure [delivery policies](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md)
and make business effects idempotent. Empty IDs disable core duplicate suppression.
Do not substitute resource IDs or attempt IDs: they can collapse distinct events
or fail to recognize redeliveries.

Mollie's classic callback is an unauthenticated status hint, not an authenticated
source. Use the application-owned [Mollie callback](https://arkoren.dev/docs/guides/payments/drivers.md#mollie-callbacks)
and retrieve authoritative state through the payment API.

## Resend

Choose `resendwebhook` for Resend email notifications. Register your public webhook
URL and selected events in Resend, then load that endpoint's `whsec_` secret. No
sending API key is required. From your application's module directory:

```sh
go get arkoren.dev/driver/resend
```

Import `arkoren.dev/driver/resend/resendwebhook` and construct the source:

```go
source, err := resendwebhook.New(signingSecret)
```

The secret must contain nonempty standard base64 after `whsec_`, without line
breaks. Invalid secrets return `webhook.ErrInvalid`. The verifier requires unique
Svix authentication headers and accepts matching v1 signatures within five
minutes of local time in either direction. It verifies before parsing the envelope;
bad signatures wrap both `webhook.ErrRejected` and `resendwebhook.ErrInvalidSignature`.

Bind `EmailSent`, `EmailDelivered`, `EmailBounced`, `EmailComplained`, or
`EmailReceived` to receive typed `Email` metadata. `EmailDelivered` means acceptance
by the recipient's mail server, not inbox placement or reading. `EmailReceived`
does not fetch email bodies or attachments; perform those lookups in application
work after acceptance. See the [mail integration](https://arkoren.dev/docs/guides/mail/drivers.md#receiving-resend-notifications)
and [`resendwebhook` API](https://pkg.go.dev/arkoren.dev/driver/resend/resendwebhook).

## Mailgun

Choose `mailgunwebhook` for Mailgun's JSON event notifications. Configure the
webhook URL and events in Mailgun and load the account's **HTTP webhook signing
key**, not an outbound API key.

> [!WARNING]
> Mailgun signs only timestamp + token, not `event-data`. A successful token check
> does not prove payload integrity. Protect the HTTPS endpoint and account for
> this limitation before allowing a payload to authorize a consequential action.

Install the module:

```sh
go get arkoren.dev/driver/mailgun
```

With import `arkoren.dev/driver/mailgun/mailgunwebhook`, construct the source:

```go
source, err := mailgunwebhook.New(signingKey)
```

Construction rejects empty keys, whitespace, and control characters without
trimming or repairing them. The source checks HMAC and a local timestamp window
of thirty minutes in either direction. There is no token cache: a valid token
can verify more than once. Signature failures wrap `webhook.ErrRejected` and
`mailgunwebhook.ErrInvalidSignature`.

If delayed deliveries require a wider local window, choose that policy explicitly.
With `time` imported, this alternative construction allows one hour:

```go
source, err := mailgunwebhook.New(signingKey, func(signature *mailgunwebhook.Builder) {
	signature.Tolerance(time.Hour)
})
```

Zero restores thirty minutes; a negative tolerance is invalid. A larger window
also accepts older captured tokens. The window measures signature age, not event
age. Delivery identity instead uses the event's UTC date and native ID because
Mailgun event IDs are unique only within a day.

`Accepted`, `Delivered`, `Failed`, `Opened`, `Clicked`, `Unsubscribed`, and
`Complained` decode native Mailgun event payloads. `Failed` retains temporary or
permanent severity rather than treating every delivery failure as terminal.
See [Mailgun notifications](https://arkoren.dev/docs/guides/mail/drivers.md#receiving-mailgun-notifications)
and the [`mailgunwebhook` API](https://pkg.go.dev/arkoren.dev/driver/mailgun/mailgunwebhook).

## Postmark

Choose `postmarkwebhook` for outbound notifications or parsed inbound email.
Configure endpoint-specific Basic Auth credentials on the Postmark webhook URL;
the sending API token is not a receiver credential. Public HTTPS is mandatory,
though TLS can terminate at a trusted proxy. Apply Postmark IP allowlisting at
the trusted edge. Basic Auth neither signs the body nor prevents a credential
holder from submitting or replaying it.

Install the module:

```sh
go get arkoren.dev/driver/postmark
```

Import `arkoren.dev/driver/postmark/postmarkwebhook`. For outbound notifications:

```go
source, err := postmarkwebhook.New(username, password)
```

The constructor rejects empty credentials, control characters, and a colon in
the username. The source rejects missing, incorrect, or duplicate Authorization
headers before parsing the body. Authentication failures wrap `webhook.ErrRejected`;
malformed authenticated JSON answers 503 rather than a credential rejection.

Bind `Delivered`, `Bounced`, `SpamComplaint`, `Opened`, `Clicked`, or
`SubscriptionChanged` for their typed outbound payloads. Core deduplication uses
`X-PM-Webhook-Trace-Id` when present. Do not replace a missing trace ID with an
outbound `MessageID`: different opens or clicks can share the same message ID.

For inbound email, configure a separate URL and construct a separate source:

```go
source, err := postmarkwebhook.NewInbound(username, password)
```

Bind `Inbound` on that receiver. Inbound JSON has no `RecordType`, so the adapter
supplies the exact event key `Inbound`; its delivery ID is the inbound `MessageID`.
The same HTTPS and credential-replay limits apply. Treat email contents as
untrusted input even when endpoint authentication succeeds.

See [outbound notifications](https://arkoren.dev/docs/guides/mail/drivers.md#receiving-postmark-notifications),
[inbound email](https://arkoren.dev/docs/guides/mail/drivers.md#receiving-inbound-email), and the
[`postmarkwebhook` API](https://pkg.go.dev/arkoren.dev/driver/postmark/postmarkwebhook).

## OpenAI

Choose `openaiwebhook` for background response lifecycle notifications. Register
the endpoint and events in OpenAI and load its dashboard endpoint secret, not
your model API key. Install the module:

```sh
go get arkoren.dev/driver/openai
```

With import `arkoren.dev/driver/openai/openaiwebhook`:

```go
source, err := openaiwebhook.New(signingSecret)
```

The secret must use canonical `whsec_`-prefixed base64 with at least 24 decoded
bytes. Invalid secrets wrap `webhook.ErrInvalid`. The OpenAI SDK verifies the
exact body and authentication headers locally before envelope parsing. Signature
failures wrap `webhook.ErrRejected` and `openaiwebhook.ErrInvalidSignature`.
There is no network call or source lifecycle hook.

`ResponseCompleted`, `ResponseFailed`, `ResponseCanceled`, and `ResponseIncomplete`
produce a `Response` reference with an ID and event time, not generated output or
failure details. `ResponseCanceled` retains the native key `response.cancelled`.
Use the response ID for a later API lookup; preserve the distinct `webhook-id`
as `Delivery.ID`. See [response notifications](https://arkoren.dev/docs/guides/llm/drivers.md#response-notifications)
and the [`openaiwebhook` API](https://pkg.go.dev/arkoren.dev/driver/openai/openaiwebhook).

## Stripe

Choose `stripewebhook` for Stripe snapshot events, not thin notifications.
Register the endpoint and event selection in Stripe and load that endpoint's
`whsec_` signing secret separately from the payment API key.

```sh
go get arkoren.dev/driver/stripe
```

Import `arkoren.dev/driver/stripe/stripewebhook`:

```go
source, err := stripewebhook.New(signingSecret)
```

Construction requires a `whsec_` prefix, a nonempty suffix, and no whitespace or
control characters. The Stripe SDK verifies the exact body and `Stripe-Signature`
locally. Signature age defaults to five minutes; an optional construction callback
can set `stripewebhook.Builder.Tolerance`. Zero restores the default, and a
negative value is invalid. Signature failures wrap `webhook.ErrRejected`,
`stripewebhook.ErrInvalidSignature`, and the SDK cause.

Shipped event definitions cover checkout, PaymentIntent, refund, subscription,
invoice, and invoice-payment snapshots and decode to `payment.Observation`.
Bound snapshot decoders require an exact API-version match with the installed
Stripe SDK and the expected resource schema. Mismatches answer 503, not a bad
signature rejection. Authenticated unbound events do not undergo those bound
schema checks.

No Stripe resource is fetched and no payment state is changed by decoding.
Use [Stripe notifications](https://arkoren.dev/docs/guides/payments/drivers.md#stripe-notifications) for
event selection and reconciliation; checkout completion is not itself proof of
collected funds. The full definitions are in the
[`stripewebhook` API](https://pkg.go.dev/arkoren.dev/driver/stripe/stripewebhook).

## PayPal

Choose `paypalwebhook` for PayPal notifications whose original event and signing
metadata can be verified through PayPal's API. You need a client ID, client
secret, and registered webhook ID from the same environment. The source defaults
to sandbox independently of any payment driver.

```sh
go get arkoren.dev/driver/paypal
```

Import `arkoren.dev/driver/paypal/paypalwebhook`. With matching sandbox credentials:

```go
source, err := paypalwebhook.New(clientID, clientSecret, webhookID)
```

Construction makes no network call. Request verification does: PayPal OAuth or
verification outages can prevent acceptance of a genuine notification and answer
503. Missing signing metadata, invalid event JSON, and a confirmed verification
failure wrap `webhook.ErrRejected`. An unknown verification result remains a
processing failure. The `PAYPAL-CERT-URL` header is sent as data to PayPal, not
fetched as an application-selected URL.

For production, explicitly select live mode with matching live credentials and
registration in this alternative construction:

```go
source, err := paypalwebhook.New(clientID, clientSecret, webhookID, func(listener *paypalwebhook.Builder) {
	listener.Live(true)
})
```

To bound verification I/O, `HTTPClient` accepts an application-supplied
`*http.Client`. The source copies it, disables redirects, and uses thirty seconds
when its timeout is nonpositive; transports remain shared and are not closed by
the source. Choose a timeout within the provider's response deadline.

Shipped definitions cover order approval, captures, sales, refunds, and subscription
lifecycle events. They decode to `payment.Observation` without applying business
state. Delivery identity is the event ID, not the retry-specific transmission ID.
Follow [PayPal notifications](https://arkoren.dev/docs/guides/payments/drivers.md#paypal-notifications) for
bindings and reconciliation, and consult the
[`paypalwebhook` API](https://pkg.go.dev/arkoren.dev/driver/paypal/paypalwebhook).

## Boundary Tests

Test with valid provider fixtures, then supply incorrect credentials or required
authentication metadata and confirm rejection before your handler runs. For
body-signing protocols, also confirm that altering the body invalidates the
signature. Do not expect that protection from Mailgun's token signature or
Postmark's Basic Auth; test their stated limits rather than assuming body integrity.
Include authenticated unknown events and bound schema failures: authentication success
does not mean every payload is decodable. For SDK-backed verification, use the
SDK version your application actually ships.

A substitute source can test core outcomes and queue handoff, but it cannot prove
provider authentication, live PayPal verification, or actual provider retries.
Check those boundaries against your configured provider environment separately.
Then choose [Delivery Policies](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md) for receipts, queued work,
and receiver diagnostics; their categories are not provider event keys.

---

Source: https://arkoren.dev/docs/guides/webhooks/delivery-policies.md
[HTML documentation](https://arkoren.dev/docs/guides/webhooks/delivery-policies)

# Delivery Policies

A provider can redeliver after your server accepts an event, including when the
response is lost. It can also send distinct events about the same resource.
Choose what counts as acceptance and make its effects safe to repeat before
enabling duplicate suppression or moving work to a queue.

These examples assume the Resend source and typed `accept` function from
[Receiving Webhooks](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md#binding-handlers). Import
`arkoren.dev/core/http/webhook` and `arkoren.dev/driver/resend/resendwebhook`.
Each receiver construction below is an alternative policy, not a sequence of
redefinitions to paste into one scope. Your application handles construction
errors before mounting the returned handler.

## Duplicate Receipts

`Deduplicate` remembers successfully accepted delivery IDs in a configured
[cache](https://arkoren.dev/docs/guides/cache/overview.md). A recorded ID receives 200 without decoding or
acceptance. Deduplication is disabled by default and is best effort, not an
exactly-once guarantee: cache errors fail open and concurrent copies can both run.
Make the handler idempotent, for example by committing an event ID and its
business effect in the same application transaction.

With `cache` imported from `arkoren.dev/core/service/cache` and `time` from the
standard library, choose a namespace and retention for a new receiver:

```go
receipts := cache.NewNamespace[bool]("webhook.resend.primary", func(namespace *cache.NamespaceBuilder) {
	namespace.TTL(24 * time.Hour)
})
```

Twenty-four hours is an example policy, not a provider guarantee. Match retention
to your provider's retry and manual-redelivery window. Zero TTL means no expiry;
there is no automatic 72-hour fallback. Use separate, noncolliding physical names
for unrelated accounts or receivers sharing a cache.

With your existing `*cache.Cache` named `store`, attach the receipts:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[resendwebhook.Envelope]) {
	receiver.Handle(resendwebhook.EmailDelivered, accept)
	receiver.Deduplicate(store, receipts)
})
```

The receiver authenticates first, checks the ID before decoding, and records it
after acceptance succeeds but before writing 200. A failed acceptance does not
write a receipt. An empty ID bypasses suppression; use the adapter's stable event ID,
not an email ID or retry-attempt ID.

A failed cache check still runs acceptance, and a failed receipt write still
answers 200. Cache eviction, expiry, or process-local storage can therefore admit
duplicates. The receiver neither starts nor stops the cache; its owner manages
that lifecycle. Nil caches and invalid namespaces fail receiver construction.

### Preserving Receipts

Preserve deployed namespace names, separators, and retention when upgrading.
Changing a receipt key forgets prior acceptance and can admit replays. A route
rename does not change keys: namespaces are not derived from routes, provider
names, secrets, or diagnostic labels.

Only delivery IDs are encoded with unpadded base64url. Namespace names and
separators are literal, so different definitions alone do not guarantee isolated
physical keys. See [cache identity safety](https://arkoren.dev/docs/guides/cache/keys.md#identity-safety).
The default separator is `:`; explicitly set `namespace.Separator(".")` in the
namespace callback to retain receipts written with the former dot default.

If you are migrating from the former scope-based webhook API, reproduce its
prefix rather than choosing a new name. With `encoding/base64` imported,
`oldScope` and `oldRetention` are your deployed values:

```go
name := "webhook:v1:" + base64.RawURLEncoding.EncodeToString([]byte(oldScope))
receipts := cache.NewNamespace[bool](name, func(namespace *cache.NamespaceBuilder) {
	namespace.Separator(":")
	namespace.TTL(oldRetention)
})
```

Use this namespace in `Deduplicate` instead of the new-receiver namespace above.
Preserve the required replay window or explicitly migrate existing receipts;
there is no automatic lookup under old names. General namespace upgrades are
covered in [preserving cache names](https://arkoren.dev/docs/guides/cache/keys.md#preserving-names).

## Queued Handling

If acceptance needs slow email, API, or database work, hand a verified delivery
to a [queue](https://arkoren.dev/docs/guides/queues/overview.md). `Dispatch` acknowledges when the queue driver
accepts the message, not when a worker finishes. There is no built-in inbox or
outbox: that acknowledgement is only as durable as your chosen queue driver.
Workers must still tolerate duplicates, including work submitted before a lost
response or a failed receipt write.

Queued messages can outlive a deployment. Keep the payload in a small type you
control rather than depending on a provider SDK's future JSON encoding:

```go
type DeliveredEmail struct {
	EmailID string `json:"email_id"`
}
```

Use `Event.Map` to retain the provider decoder while projecting its result into
that application-owned type:

```go
delivered := resendwebhook.EmailDelivered.Map(func(email resendwebhook.Email) (DeliveredEmail, error) {
	return DeliveredEmail{EmailID: email.EmailID}, nil
})
```

The original event remains unchanged. Its decoder runs before conversion; a
decoder error prevents conversion. `Map` preserves the exact provider key and
does not replace the delivery ID or type. Keep conversion free of I/O and shared
payload mutation; functions and captured values must support concurrent calls.
An ordinary conversion error answers 503 at the decode stage. An authenticated
`ErrSkip` instead acknowledges without queueing or recording a receipt.

Import `arkoren.dev/core/service/queue`. With your configured `*queue.Dispatcher`
named `dispatcher`, declare a typed job and bind it instead of `Handle`:

```go
job := queue.NewJob[webhook.Delivery[DeliveredEmail]]("email-delivered")
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[resendwebhook.Envelope]) {
	receiver.Dispatch(delivered, dispatcher, job)
})
```

The queued payload includes the provider ID, type, and mapped payload under the
stable JSON members `id`, `type`, and `payload`. The queue creates its own message
ID. Evolve application payloads compatibly while old messages remain queued, and
remember that queue message-size limits can be smaller than the webhook body
limit. If the event already returns the payload type you want to store, dispatch
it directly without `Map`. A nil dispatcher, invalid job, or invalid mapped event
fails receiver construction.

With `context` imported, your existing `*queue.Consumer` named `consumer` can
perform the work later. Register before starting the consumer:

```go
err := consumer.Handle(job, func(ctx context.Context, delivery webhook.Delivery[DeliveredEmail]) error {
	return messages.RecordDelivered(ctx, delivery.Payload.EmailID)
})
```

Here `messages` is your idempotent application service, not an Arkoren helper.
Consumer failures follow [queue delivery and retry policy](https://arkoren.dev/docs/guides/queues/delivery-and-retries.md),
not a second HTTP answer to the provider. Queue verified deliveries, not raw
bodies: signature headers are not included in the job.

## Acknowledgement Outcomes

The receiver processes one event per request and writes its own response.
Provider retry rules remain provider-owned: neither 503 nor any other answer
guarantees redelivery. A handler can commit part of its work and then fail;
an error is not evidence of rollback.

| Outcome | HTTP Response | New Receipt |
| --- | --- | --- |
| Acceptance returns nil, including an intentional no-op | 200, empty body | Write attempted if enabled and the ID is nonempty |
| Previously recorded ID | 200, empty body | None |
| Authenticated unbound event | 200, empty body | None |
| Source, decoder, or conversion returns `ErrSkip` | 200, empty body | None |
| Source error matches `Reject` | Mapped 4xx | None |
| Source error wraps `ErrRejected` without a matching override | 400 | None |
| Body exceeds the effective limit | 413 | None |
| Malformed body read | 400 | None |
| Body read interrupted by cancellation or timeout | 503 | None |
| Other source error, ordinary decode/conversion error, or any acceptance error | 503 | None |

Return `ErrSkip` from a source only after authentication succeeds; otherwise a
forged request could be acknowledged as valid. Decoders and conversions already
run after authentication. In acceptance, return nil for an intentional no-op.
Every acceptance error—including `ErrSkip` and permanent queue errors—answers
503 and leaves the ID unrecorded.

Failure responses use fixed Problem Details messages without the underlying
cause, and the receiver never redirects. Oversized bodies retain the native
`request.ErrBodyTooLarge` 413 problem. Because these responses are already
committed, application-wide [error mapping](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md)
cannot replace them. Response-writer hooks still run.

Returned receipt errors still match their class (`webhook.ErrRejected` or
`webhook.ErrUnavailable`) and original cause through `errors.Is`. Oversized
receipts also match `request.ErrBodyTooLarge`. [Request logging](https://arkoren.dev/docs/guides/observability/request-logging.md)
can retain the cause without exposing it to the sender; an error mapper can
still affect log classification, not the committed response.

### Rejection Overrides

Use `Reject` only for known-invalid verification errors. Do not map a temporary
lookup or verification outage to 4xx: the provider might stop redelivering a
genuine event. For Resend, an alternative receiver can answer 401 for a known bad
signature, with `net/http` imported:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[resendwebhook.Envelope]) {
	receiver.Handle(resendwebhook.EmailDelivered, accept)
	receiver.Reject(http.StatusUnauthorized, resendwebhook.ErrInvalidSignature)
})
```

Mappings apply only to source errors and match through `errors.Is`. Calls
accumulate; the first matching mapping wins. Unmapped `ErrRejected` answers 400;
other source errors answer 503.

`Reject` requires at least one non-nil error and a 4xx status, but refuses 408
and 429 because they mean retry later, 406 because Mailgun stops retrying, and
410 because Standard Webhooks providers disable the endpoint. Invalid mappings
make `Handler` return `ErrInvalid` at construction.

## Body Limits

The receiver reads the body before authentication, so its size limit also bounds
unauthenticated requests. The default is 1 MiB. If provider notifications fit
within a smaller allowance, lower it while constructing the receiver:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[resendwebhook.Envelope]) {
	receiver.Handle(resendwebhook.EmailDelivered, accept)
	receiver.MaxBodySize(256 << 10)
})
```

This permits at most 256 KiB; larger requests answer 413 without calling the
source. Zero selects the 1 MiB default; negative sizes fail construction. An
outer [body-limit middleware](https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md) remains a ceiling:
the smaller limit wins. Raising `MaxBodySize` cannot raise that outer ceiling.
Choose a larger allowance only for a provider payload that needs it, and keep
ingress limits and queue capacity consistent with that choice.

## Receiver Diagnostics

A reachable endpoint can still reject every request after a secret change.
Connect your application's [event bus](https://arkoren.dev/docs/guides/events/overview.md) to distinguish
rejections, processing failures, and authenticated events without bindings.
With an existing bus named `bus`:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[resendwebhook.Envelope]) {
	receiver.Handle(resendwebhook.EmailDelivered, accept)
	receiver.Events(bus)
	receiver.Label("resend-primary")
})
```

`Events(nil)`, the default, publishes nothing. Labels default to empty and are
not validated or redacted: choose low-cardinality application values without
credentials, personal identifiers, or provider-controlled data. A label is not a
receipt identity and does not alter deduplication.

| Topic | Payload | When Published |
| --- | --- | --- |
| `webhook.Rejected` (`http.webhook.rejected`) | `RejectedEvent`: label, reason, status | Any 4xx outcome |
| `webhook.Failed` (`http.webhook.failed`) | `FailedEvent`: label, stage | Any 503 outcome |
| `webhook.Ignored` (`http.webhook.ignored`) | `IgnoredEvent`: label, reason | Authenticated unbound event |

Skips and duplicate acknowledgements publish none of these topics. Categories
describe local processing, not provider business outcomes:

| Field Type | Constants and String Values |
| --- | --- |
| `RejectionReason` | `RejectionReasonBodyTooLarge` = `"body_too_large"`; `RejectionReasonUnreadableBody` = `"unreadable_body"`; `RejectionReasonVerify` = `"verify"` |
| `FailureStage` | `FailureStageRead` = `"read"`; `FailureStageVerify` = `"verify"`; `FailureStageDecode` = `"decode"`; `FailureStageAccept` = `"accept"` |
| `IgnoreReason` | `IgnoreReasonUnbound` = `"unbound"` |

These constants belong to `webhook`. Unclassified source errors, including
remote verification outages and envelope-processing failures, report
`FailureStageVerify`; classified source rejections report `RejectionReasonVerify`.
`Map` errors belong to the decode stage; every acceptance error belongs to the
accept stage. Empty zero values remain unspecified, not default categories.

With `context` imported and your existing logger, subscribe for the receiver's
lifetime:

```go
unsubscribe, err := bus.Listen(webhook.Failed, func(ctx context.Context, failure webhook.FailedEvent) error {
	logger.WarnContext(ctx, "webhook processing unavailable",
		"receiver", failure.Label, "stage", string(failure.Stage))

	return nil
})
```

Handle the registration error and call `unsubscribe` when retiring the receiver,
not when a short-lived setup function returns. Listeners run synchronously after
the outcome is known and cannot change it; send slow alert work to a worker.

Diagnostic payloads omit bodies, headers, provider IDs, event types, and causes.
That omission does not sanitize request context values or returned receipt
errors, and normal HTTP problem instances can still contain a request path.
Select safe logging attributes instead of forwarding whole contexts or errors.

If upgrading from plain-string category fields, compare typed constants or use
explicit conversions such as `string(failure.Stage)` for string-only APIs.
String and JSON category bytes and member names remain unchanged. This change
does not require renaming provider keys, labels, or receipt namespaces.

## Operating Receivers

Meet the provider's response deadline by keeping acceptance short and bounding
remote verification I/O. Core handles one event per request; for provider batches,
decode a batch payload and split it in the consumer. Challenge responses,
interactive decisions, and decoding that fetches remote resources need
application-owned handling rather than this receiver pipeline.

Before deploying, check the boundary with signed provider fixtures:

- A valid delivery returns 200 and reaches the intended handler or queue job.
- An invalid signature never reaches acceptance; a verification outage answers 503.
- Repeating an ID suppresses the second sequential acceptance when receipts are enabled.
- Failed acceptance can run again; cache failures and concurrent deliveries remain safe.
- Body-limit and decoder failures produce the intended status and diagnostic stage.

Use `handler.Record(signedRequest)` to exercise a fixture without a network;
close the returned response body when the test finishes reading it.
For queue handoff, `queuetest.New()` from
`arkoren.dev/core/service/queue/queuetest` records dispatches without a broker;
`FailWith(err)` simulates acceptance failure. That fake proves submission, not
external queue durability. Test worker effects independently and use
`testing/synctest` for timestamp-sensitive fixtures. Monitor
[cache failures](https://arkoren.dev/docs/guides/cache/monitoring.md) as well as receiver outcomes, since
receipt failures intentionally do not fail the delivery.

For every function and option, see the
[`webhook` API reference](https://pkg.go.dev/arkoren.dev/core/http/webhook).

---

Source: https://arkoren.dev/docs/guides/http-networking/server.md
[HTML documentation](https://arkoren.dev/docs/guides/http-networking/server)

# HTTP Server

An HTTP listener needs to acquire its port before the application reports a
successful startup. It also needs a bounded shutdown that lets requests finish
without leaving their dependencies in use. The `server` package separates
binding, serving, and cleanup so you can give each step to the appropriate
application lifecycle phase.

The server accepts any `net/http.Handler`. Build an Arkoren router and
[register typed handlers](https://arkoren.dev/docs/guides/http-fundamentals/routing.md#register-handlers)
before constructing the listener; this page manages the network boundary, not
request decoding or response construction.

## Bind Locally

With an existing handler named `routes`, import
`arkoren.dev/core/http/server` and choose an explicit loopback address:

```go
srv := server.New(routes, func(listener *server.Builder) {
	listener.Addr("127.0.0.1:8080")
})
```

Construction opens no socket. Without the callback, the address is `:8080`,
which listens on all interfaces. Use that broader address only when the
deployment's network policy permits access. Without TLS settings the listener
serves plaintext HTTP; use TLS termination at your ingress or configure
[HTTPS](#serving-https) before exposing sensitive traffic.

Configure native fields through `srv.HTTP()` before `Start` if you need a Go
server setting that the builder does not expose. Server and TLS configuration
must not be mutated after startup.

## Register Lifecycle

Inside an existing application composition callback, give the composer the
server's cleanup, startup, and runner methods. The composer type comes from
`arkoren.dev/core/application`:

```go
compose := func(composer *application.Composer) {
	composer.Cleanup(srv.Stop)
	composer.Startup(srv.Start)
	composer.Run(srv.Serve)
}
```

Pass this callback to your application's `Start` or `Run`. `Start` acquires the
TCP listener; an occupied port fails startup rather than becoming a later runner
failure. `Serve` starts accepting requests only after all startup hooks succeed.
Register required dependency initialization as startup work too.

The cleanup registration matters even if serving never begins: another startup
hook can fail after this listener has bound. `Stop` also closes a bound listener
that `Serve` never took over. See
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md) for the
application's startup and cleanup ordering, and
[outcome handling](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md#observe-outcomes)
for observing a runner failure.

If you do not use an application, you own all three phases. With a caller-owned
context `ctx`, check binding before entering the serving loop:

```go
if err := srv.Start(ctx); err != nil {
	return err
}
err := srv.Serve(ctx)
```

Arrange `Stop` with a live, bounded cleanup context on every exit after a
successful `Start`, including serving failure. Canceling `ctx` initiates
`Serve`'s drain; merely returning from unrelated application code does not.

## Serving HTTPS

Prepare a `*tls.Config` named `settings` by
[loading a server identity](https://arkoren.dev/docs/guides/http-networking/certificates.md#load-identities). Select TLS when
constructing the listener:

```go
srv := server.New(routes, func(listener *server.Builder) {
	listener.Addr("127.0.0.1:8443")
	listener.TLS(settings)
})
```

Use the same lifecycle registrations. `Start` rejects missing credentials and
obviously invalid static certificates before binding. A nonnil configuration
needs a static certificate, `GetCertificate`, or `GetConfigForClient`.
`TLS(nil)` selects plaintext instead; an HTTPS failure never falls back to HTTP.

The native TLS configuration remains shared. Finish preparing its certificates,
trust pools, and callbacks before startup. Callbacks are not invoked by the
startup check, so their failures occur during handshakes. Binding successfully
does not prove that clients trust the identity or that its names and validity
match their requests. Native `ServeTLS` handles handshakes and HTTP/2 negotiation.

## Set Timeouts

The server supplies connection-level limits so a slow client cannot retain a
connection indefinitely under the default configuration:

| Setting | Default | Scope |
| --- | --- | --- |
| `ReadHeaderTimeout` | 10 seconds | Reading request headers. |
| `ReadTimeout` | 30 seconds | Reading the whole request, including its body. |
| `WriteTimeout` | 60 seconds | Writing a response. |
| `IdleTimeout` | 120 seconds | Waiting between keep-alive requests. |
| `MaxHeaderBytes` | 1 MiB | Request headers, with Go's additional parser buffer allowance. |

For an upload listener whose request bodies legitimately take longer, import
`time` and change its read budget during construction:

```go
uploads := server.New(routes, func(listener *server.Builder) {
	listener.Addr("127.0.0.1:8081")
	listener.ReadTimeout(2 * time.Minute)
})
```

These settings apply to the listener, not an individual route. A separate
listener can keep a long upload budget away from ordinary API requests. Zero
timeout values select Arkoren's defaults rather than disabling them.

Socket timeouts are not application-work deadlines. Pass request contexts to
downstream work and make that work honor cancellation. For long responses,
[bound slow writes](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md#bound-slow-writes)
rather than assuming the default write timeout renews after every message.
A negative `WriteTimeout` disables that connection write deadline; use it only
with an explicit streaming write policy.

## Shutting Down

When the runner context ends, `Serve` stops accepting new connections and asks
Go's HTTP server to drain tracked connections. The default drain budget is
15 seconds. Choose a different budget before startup with `time` imported:

```go
srv := server.New(routes, func(listener *server.Builder) {
	listener.Addr("127.0.0.1:8080")
	listener.ShutdownTimeout(10 * time.Second)
})
```

Keep this budget inside the application's and deployment's
[shutdown budgets](https://arkoren.dev/docs/guides/first-steps/deploying.md#shutdown-budgets). Zero selects
15 seconds; a negative value expires the drain deadline immediately. A clean
drain returns nil. An expired drain closes remaining tracked HTTP connections
and returns an error matching `context.DeadlineExceeded`.

`Stop` instead uses its caller's context as the drain budget; it does not use
`ShutdownTimeout`. Even an already-expired cleanup context releases the listener
and closes remaining tracked connections. Repeated calls are safe.

Neither operation proves that arbitrary handler goroutines have finished or that
the client received a complete response. Hijacked connections, including
WebSockets, are outside HTTP draining: their owners must close them and wait
for associated work before releasing resources that work still uses.

## Stop Streams

Beginning shutdown does not cancel active request contexts. This lets ordinary
requests finish, but a long poll or stream needs a separate signal to stop
waiting. `server.ShuttingDown` returns that signal for requests accepted by this
server; without it, the result is a nil channel and its select case is disabled.

Inside a handler, let a wait end for either request cancellation or server
shutdown. Here `incoming` is its `*http.Request`, `messages` is an
application-owned channel, and `sendMessage` writes one message and returns an
error:

```go
select {
case <-server.ShuttingDown(incoming.Context()):
	return nil
case <-incoming.Context().Done():
	return incoming.Context().Err()
case message, open := <-messages:
	if !open {
		return nil
	}
	return sendMessage(message)
}
```

The response streaming helpers watch shutdown too, but cannot stop a producer
blocked outside their yield or receive operation. Arrange cooperative
[iterator cancellation](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md#stop-iterator-work)
and stop channel producers before they block sending their next message.
Connection closure is not a substitute for joining application-owned work.
If bytes are already sent, follow the rules for
[committed failures](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md#committed-failures)
rather than trying to replace the response.

## Observe Outcomes

Set `listener.Events(app.Events())` in the construction callback to publish
listener events on your application's bus. `server.Listening` reports the bound
address; it does not mean serving has begun or an HTTPS handshake succeeded.
`server.ShutdownTimedOut` reports a `Serve` drain deadline that forced tracked
connections closed. It is not emitted for a clean drain or by `Stop`.

Keep subscriptions alive for the observed component's lifetime and
[own their cleanup](https://arkoren.dev/docs/guides/application/composition.md#own-subscriptions).
These synchronous notifications are not readiness probes or proof that all
handlers stopped. Mount the application's
[health probes](https://arkoren.dev/docs/guides/application/health-and-readiness.md#mount-probes) and
[observe readiness transitions](https://arkoren.dev/docs/guides/application/health-and-readiness.md#observe-transitions)
separately.

## Separate Listeners

If operational routes need a different exposure policy, construct another
server with its own handler and address:

```go
admin := server.New(operatorRoutes, func(listener *server.Builder) {
	listener.Addr("127.0.0.1:9090")
})
```

Register `admin.Stop`, `admin.Start`, and `admin.Serve` on the same composer as
the public listener. Both bind before runners begin, and each drains its own
connections. Decide which deployment endpoints reach these listeners in
[Health Probes](https://arkoren.dev/docs/guides/first-steps/deploying.md#health-probes); a second port alone
does not authenticate an operator.

## Test Listeners

Most handler tests need no socket; use [HTTP Tests](https://arkoren.dev/docs/guides/testing/http-tests.md).
When a test needs a real listener, let the OS assign a port:

```go
srv := server.New(routes, func(listener *server.Builder) {
	listener.Addr("127.0.0.1:0")
})
if err := srv.Start(ctx); err != nil {
	return err
}
endpoint := "http://" + srv.Addr()
```

`Addr` reports the assigned port after binding. It does not start serving;
run `Serve` separately and arrange `Stop` with a live cleanup context even if
the test fails before serving begins.

For native configuration and all options, see the
[`server` API reference](https://pkg.go.dev/arkoren.dev/core/http/server).

---

Source: https://arkoren.dev/docs/guides/http-networking/http-client.md
[HTML documentation](https://arkoren.dev/docs/guides/http-networking/http-client)

# HTTP Client

An outgoing request can fail before you know whether the receiving service
processed it. A retry gives transient failures another chance, but repeating a
write without the receiver's deduplication contract can duplicate its effects.
Arkoren's `client` package adds retry policy to a standard `http.RoundTripper`;
you still use Go's `http.Client` for requests, redirects, cookies, and responses.

If you are adding this to an existing project, follow
[existing application installation](https://arkoren.dev/docs/guides/first-steps/installation.md#existing-applications).
No separate client module is required.

## Create Clients

For a client used to read reports, import `net/http`, `time`, and
`arkoren.dev/core/http/client`. Set a whole-operation timeout when constructing
the client:

```go
outgoing := &http.Client{
	Transport: client.Transport(http.DefaultTransport),
	Timeout:   8 * time.Second,
}
```

The default permits three total attempts, including the first. It retries
transport errors and responses with status `408`, `429`, `502`, `503`, or `504`;
`500` is not retryable by default. Eligibility also depends on the request's
method and body, as described in [Permit Retries](#permit-retries).
In particular, the default considers `PUT` and `DELETE` eligible, not only reads.
Do not share this policy with writes unless the receiving API honors their
idempotent semantics.

Reuse clients across requests and goroutines. Configure the client and base
transport before sharing them. Construction opens no connections. Passing nil
as the base selects `http.DefaultTransport`; it does not disable retries.

The default backoff uses full jitter, starting with a 100-millisecond upper
bound, doubling up to 10 seconds. The client's timeout covers all attempts,
retry waits, and reading the final response body—not eight seconds per attempt.

## Read Responses

With your caller's `ctx` and report URL `endpoint`, create a request carrying
that context:

```go
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
```

Handle a construction error before passing the request to `Do`. Then inspect
the final response status before consuming its data. This fragment also uses
`fmt` and `io` inside a function returning an error:

```go
res, err := outgoing.Do(req)
if err != nil {
	return err
}
defer res.Body.Close()

if res.StatusCode != http.StatusOK {
	return fmt.Errorf("read report: unexpected status %s", res.Status)
}
data, err := io.ReadAll(res.Body)
```

Retries finish before `Do` returns. A final `503` is an HTTP response with a nil
Go error, so checking `err` alone does not establish success. Close the final
body on every successful `Do`, including non-2xx responses and body-read errors.
Reading to EOF before closing permits connection reuse when the native
transport can reuse it.

`ReadAll` is suitable only for a bounded response. Stream large bodies or impose
an application-specific size limit on untrusted data. A read error after `Do`
returns does not trigger another request; restarting a partial download is your
application's decision.

The retry wrapper closes the native body once and then releases the attempt
context. Repeated or concurrent `Close` calls return the first close result;
EOF and I/O errors alone do not release that ownership. This does not make
concurrent reads safe. Capture close errors too if your operation needs to
report them.

## Permit Retries

A request is eligible for another attempt only if both conditions hold:

- Its method is `GET`, `HEAD`, `OPTIONS`, `TRACE`, `PUT`, or `DELETE`, or it has
  a nonempty `Idempotency-Key` header.
- It has no body, or `Request.GetBody` can recreate that body.

These are transport rules, not proof that an endpoint is safe to repeat. For a
write API without a verified replay contract, use an explicitly single-attempt
client:

```go
writes := &http.Client{
	Transport: client.Transport(http.DefaultTransport, func(transport *client.Builder) {
		transport.Attempts(1)
	}),
	Timeout: 8 * time.Second,
}
```

This disables Arkoren's retries while retaining instrumentation. Inspect any
SDK, proxy, application, and native transport replay behavior separately; this
setting is not an exactly-once delivery guarantee.

Only opt a `POST` or `PATCH` into retries if the destination documents how it
deduplicates an operation using `Idempotency-Key`. The header itself does not
create that behavior. For a report-creation API with that contract, use an
application-persisted `operationID` for the same logical operation on every
retry. With `strings` imported and the operation's JSON in `payload`:

```go
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, strings.NewReader(payload))
if err != nil {
	return err
}
req.Header.Set("Idempotency-Key", operationID)
```

Send this request through `outgoing.Do` and handle its final response as above.
Do not generate a new key for an uncertain retry, and follow the receiver's key
retention and payload-matching rules. A body from `strings.Reader`,
`bytes.Reader`, or `bytes.Buffer` gets `GetBody` from Go's request constructor;
an arbitrary streaming reader does not. Without `GetBody`, a nonempty body is
sent once even if its method or key would otherwise permit retries.

Each attempt clones the request. Before a resend, the transport drains up to
4 KiB of the discarded response, closes it, waits, then recreates the request
body. Drain or close errors do not prevent a retry. A body-recreation failure
returns an error wrapping its cause without sending that attempt. You own only
the final response body returned to you.

## Respect Retry-After

For retryable `429` and `503` responses, a valid `Retry-After` replaces the
configured backoff. A wait greater than 10 seconds, or greater than the remaining
request deadline, stops retries and returns that response with its body still
yours to close. The ten-second cap is fixed; increasing the backoff maximum
does not increase it.

The header accepts unsigned whole decimal seconds or an HTTP date parsed by
`http.ParseTime`, after trimming whitespace. A past date means no wait.
Signed or fractional values such as `+2`, `-1`, and `1.5` fall back to backoff;
an oversized unsigned decimal instead exceeds the cap and stops retries.
Other statuses ignore this header, even if a custom policy permits retrying them.

There is no separate exhaustion error. The final outcome remains a response
or transport error:

| Final Outcome | Caller Receives |
| --- | --- |
| All attempts return `503` | The final `503`, with nil error from `Do`. |
| All attempts fail at the transport | An error from `Do`, normally wrapped in `*url.Error`. |
| A response is not retryable | That response immediately. |
| `Retry-After` exceeds the allowed wait | That response without another attempt. |
| The context ends during a retry wait | A context error, with no next attempt. |

## Bound Requests

Use a request context for an operation-specific budget instead of mutating a
shared client. With `context` imported:

```go
bounded, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()

req, err := http.NewRequestWithContext(bounded, http.MethodGet, endpoint, nil)
```

Keep this context alive until you have read and closed the response body.
Canceling immediately after `Do` can interrupt subsequent reads. The earliest
applicable deadline wins; the request context reaches every attempt and bounds
retry waits.

If one attempt must not consume the whole operation budget, set an additional
per-attempt limit:

```go
outgoing := &http.Client{
	Transport: client.Transport(http.DefaultTransport, func(transport *client.Builder) {
		transport.AttemptTimeout(2 * time.Second)
	}),
	Timeout: 8 * time.Second,
}
```

An attempt timeout includes reading its response body. Backoff is outside that
individual budget, so retain an overall deadline. Zero or negative attempt
timeouts set no per-attempt limit, which is the default. A timeout before
response headers can be retried while the overall context remains active;
a timeout reading the final body cannot.

Custom transports must implement context-aware I/O. The retry wrapper cannot
force an arbitrary blocked read, write, or close to return.

### Upgraded Streams

For `101 Switching Protocols`, the body retains `io.Writer` and
`io.ReadWriteCloser` only if the native body supports writing. Native byte counts
and errors pass through unchanged. Your protocol code owns closure, and the
attempt context stays active until `Close`.

If you need the writable interface through `http.Client`, leave its `Timeout`
at zero: Go's client timeout adds a read/close-only wrapper. Use a request
context or `AttemptTimeout` for the applicable budget, subject to the native
stream's cancellation behavior. Arkoren adds no write deadline or forced
interruption to upgraded I/O.

## Tune Policy

Choose retry settings from the destination's documented behavior. For an API
that explicitly treats `500` as transient, replace the outcome predicate while
retaining the other default outcomes:

```go
func retryReports(res *http.Response, err error) bool {
	if err != nil {
		return true
	}
	switch res.StatusCode {
	case 408, 429, 500, 502, 503, 504:
		return true
	default:
		return false
	}
}
```

Attach this function and, if needed, a different jittered backoff during
construction:

```go
transport := client.Transport(http.DefaultTransport, func(transport *client.Builder) {
	transport.RetryIf(retryReports)
	transport.Backoff(client.Backoff(200*time.Millisecond, 2, 2*time.Second))
})
```

`RetryIf` replaces rather than extends the predicate. Exactly one of its
response and error is nonnil. It cannot override method safety, body
replayability, the attempt limit, or cancellation. Nil restores the default
predicate; nil `Backoff` restores the default delay function.

`Attempts(0)` selects three total attempts; negative values send once. A custom
backoff receives the attempt about to run: `2` for the first retry. Nonpositive
delays mean no wait; avoid repeated immediate retries against a struggling
service. While the caller context is active, `RetryIf` runs after every attempt,
including the last. `Backoff` runs only when another attempt is eligible, before
`Retry-After` selection—even if that header ultimately stops the retry.

## Own Transports

If you already own a transport with proxy, TLS, or pool settings, pass it as
the base rather than replacing it with nil:

```go
outgoing := &http.Client{
	Transport: client.Transport(base),
	Timeout:   8 * time.Second,
}
```

Configure `base` before sharing it. Retain it for pool management:
`Transport` does not forward `CloseIdleConnections` to its base. When retiring
an owned native pool, call `base.CloseIdleConnections()` yourself; do not close
a shared pool on behalf of unrelated users.

If you want an owned copy of Go's default transport, use `NewTransport`. With
`crypto/tls` imported, keep normal server verification enabled:

```go
transport, err := client.NewTransport(&tls.Config{
	MinVersion: tls.VersionTLS12,
})
if err != nil {
	return err
}
outgoing := &http.Client{Transport: transport, Timeout: 8 * time.Second}
```

Construction clones the current nonnil `*http.Transport` default and preserves
its proxy, pooling, and HTTP/2 settings. Another default implementation returns
an error before configuration callbacks run; wrap that implementation with
`Transport` instead. Call `outgoing.CloseIdleConnections()` when retiring this
owned pool; active requests are not interrupted.

`NewTransport(nil)` preserves the default's TLS settings. A nonnil configuration
replaces them with a shallow clone. Prepare nested pools, certificates, and
callback state before sharing; neither the supplied configuration nor the
default is modified. For private roots and client identities, follow
[TLS Certificates](https://arkoren.dev/docs/guides/http-networking/certificates.md#choose-client-trust). Certificate errors can
be retried as transport errors, but retrying cannot fix invalid trust, expiry,
or names.

An SDK can use this `*http.Client` too, but choose one layer to own retries.
Three SDK attempts around three transport attempts can send nine requests.

## Trace Attempts

Every attempt that reaches the underlying transport gets a client span when
telemetry is active. Retries carry `http.request.resend_count`. Ordinary spans
include body reading and end on EOF, a read error, or closure; no-body and
upgrade spans end at headers. Rewind failures before network I/O create no
network-attempt span.

By default instrumentation follows `telemetry.Default()`. With an existing
application, choose its [shared handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle)
explicitly:

```go
transport := client.Transport(http.DefaultTransport, func(transport *client.Builder) {
	transport.Telemetry(app.Telemetry())
})
```

Do not also wrap the base with `telemetry.Transport`; the retry transport
already instruments attempts and a second wrapper can duplicate spans.

Trace injection defaults to `telemetry.PropagateInternal`: internal IP literals,
single-label hosts, and recognized internal suffixes qualify; resolving a
public-looking name to a private address does not. This is a destination-name
policy, not authentication. It does not sanitize trace headers you manually
put on the request. Keep sensitive headers out of caller-owned requests and
apply an explicit [outgoing policy](https://arkoren.dev/docs/guides/observability/request-tracing.md#outgoing-policy)
when a destination requires a different boundary.

For a client that must not inject trace context, import
`arkoren.dev/core/service/telemetry` and set the predicate:

```go
transport := client.Transport(http.DefaultTransport, func(transport *client.Builder) {
	transport.Propagate(telemetry.PropagateNone)
})
```

Spans still record the request. `Propagate` replaces injection policy, not retry
policy. Expanding it to a partner can disclose trace identifiers and baggage;
allow only destinations you trust to receive that data. Incoming remote-parent
acceptance is a separate [incoming trust](https://arkoren.dev/docs/guides/observability/request-tracing.md#incoming-trust)
decision.

## Observe Failures

Retries produce debug records on `slog.Default()` unless you supply a logger
with `transport.Logger(logger)`. Records contain the method, hostname, attempt,
delay, and status or error type—not paths, queries, bodies, or error text.
Enable debug level on that logger to see them. These are planned retry records,
not a log of every call.

To receive terminal outcome events, set `transport.Events(app.Events())` in the
construction callback and subscribe to `client.RequestFailed` on that bus.
Its payload includes `Method`, hostname-only `Host`, `Attempts`, `StatusCode`,
and `Err`. A transport failure has zero status; an HTTP response has nil error.
Keep listeners short because dispatch is synchronous, and
[own the subscription](https://arkoren.dev/docs/guides/application/composition.md#own-subscriptions) for the
observed client's lifetime.

The event reports terminal transport errors and responses that still match
`RetryIf` when attempts or allowed `Retry-After` waits run out. It does not
report every non-2xx response: default-policy `400` and `500` responses do not
emit it. No event is emitted if the caller context is already done at the
post-attempt check, during cancellation of a retry wait, or for body-read errors
after `Do`. Events are disabled without a bus and are not a durable retry queue.

Unlike retry logs, event `Err` retains the original error and can expose
sensitive diagnostics. Redact it before export; see
[Telemetry Privacy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md).

For backoff edge cases, construction rules, and every option, see the
[`client` API reference](https://pkg.go.dev/arkoren.dev/core/http/client).

---

Source: https://arkoren.dev/docs/guides/http-networking/certificates.md
[HTML documentation](https://arkoren.dev/docs/guides/http-networking/certificates)

# TLS Certificates

HTTPS needs two separate decisions: which identity a server presents, and which
issuers a client trusts. An **identity** contains a certificate chain and its
matching private key. A **certificate authority (CA)** signs certificates;
clients must independently trust that authority before accepting its identities.

The `certificate` package provisions local-development identities and imports
existing client or server credentials as native Go TLS values. Keep generic
[application key management](https://arkoren.dev/docs/guides/keys/overview.md) separate from certificate
issuance. For file creation and trust-store command syntax, use
[Certificate Commands](https://arkoren.dev/docs/guides/cli/certificates.md).

## Choose Trust

For a public production service, obtain a certificate from an issuer your
clients already trust, or terminate TLS at an ingress that manages those
certificates. This package's local CA is not publicly trusted and is not a
production PKI, ACME client, or renewal service. Generating a certificate for
a public hostname does not validate ownership of that name.

For local HTTPS, generate a dedicated development CA and configure a particular
client to trust its public certificate. A client-specific pool keeps that
decision out of the operating system's global trust store. Do not disable
certificate verification to make local requests succeed: fix the trusted issuer,
certificate names, or validity instead.

The library neither writes files nor changes system trust. The examples below
also do neither. If you later choose a CLI trust command, review its broader
scope and required permissions separately.

## Provision Locally

Add the independent module from your application's module directory:

```sh
go get arkoren.dev/foundation/certificate
```

It works without Arkoren core. The server and retrying-client integrations use
`arkoren.dev/core`; follow
[existing application installation](https://arkoren.dev/docs/guides/first-steps/installation.md#existing-applications)
if that dependency is not already present.

> [!WARNING]
> Anyone with a trusted CA's private key can issue identities accepted by its
> clients. Protect private exports and their storage directories, never commit
> them, and keep the CA key out of ordinary server deployments. Provision
> deliberately; do not regenerate missing credentials during application startup.

In an explicit provisioning operation, import
`arkoren.dev/foundation/certificate` and generate the development authority:

```go
authority, err := certificate.GenerateRoot("Reports Local Development")
```

This creates an in-memory ECDSA P-256 root with a default lifetime of
5 × 365 days. After checking the error, issue a server identity for the names
your local clients actually use:

```go
identity, err := certificate.GenerateServer(authority, []string{"localhost", "127.0.0.1", "::1"})
```

The server certificate also uses P-256 and has server-authentication purpose.
Its subject alternative names (SANs) are exactly the supplied DNS names and IP
addresses; no localhost name is inferred. Supply bare ASCII DNS names or IP
literals, not URLs or host-and-port pairs. A common name does not replace SANs.

The default server lifetime is 90 days, shortened to the issuer's remaining
lifetime. To choose a shorter lifetime, import `time` and use a construction
callback instead:

```go
identity, err := certificate.GenerateServer(authority, []string{"localhost"}, func(leaf *certificate.ServerBuilder) {
	leaf.Validity(7 * 24 * time.Hour)
})
```

An explicit lifetime beyond the issuer's expiry fails rather than being
shortened. Zero selects the default; negative or positive subsecond lifetimes
are invalid. `RootBuilder.Validity` similarly selects a root lifetime. The
issuer must be a currently valid, self-signed, certificate-signing root with
path-length-zero constraints, as generated above; intermediate CA issuance is
not supported.

`identity.Bundle().PEM()` exports the public chain, leaf first.
`identity.PrivatePEM()` explicitly exports the unencrypted private key. Your
provisioning operation owns secure persistence and distribution; the
[CLI provisioning commands](https://arkoren.dev/docs/guides/cli/certificates.md) offer exclusive file
creation when you do not need custom storage. The package provides no automatic
renewal or client-certificate generator.

## Load Identities

During application startup, load previously provisioned credentials instead of
generating replacements. With `certificate` imported, read the deployment's
server chain and private key:

```go
identity, err := certificate.LoadIdentity("server.pem", "server-key.pem")
if err != nil {
	return err
}
credentials, err := identity.TLS()
if err != nil {
	return err
}
```

These checks stop setup if files are missing or credentials cannot be converted.
With `crypto/tls` imported, prepare the server's native configuration:

```go
settings := &tls.Config{
	MinVersion:   tls.VersionTLS12,
	Certificates: []tls.Certificate{credentials},
}
```

Pass it to the [server's HTTPS setup](https://arkoren.dev/docs/guides/http-networking/server.md#serving-https) before startup.
The loader accepts RSA, ECDSA, and Ed25519 identities, not just generated P-256
keys. Chains are leaf-first and can omit the root. Private keys must be
unencrypted PKCS #1, SEC 1, or PKCS #8 PEM. Each input or file read is bounded
to 1 MiB; malformed blocks, extra keys, headers, and non-whitespace junk fail
parsing. File errors preserve their OS causes.

Parsing checks encoding, supplied chain order and signatures, and the key match.
It does not establish trust, current validity, revocation, hostname coverage,
or suitability for a particular TLS purpose. Peer verification during a native
TLS handshake applies its policy; use `x509.Certificate.Verify` with explicit
roots and options if you need a separate verification step.

## Choose Client Trust

For a client that must trust only your private CA, obtain its public root through
a trusted distribution path. Parsing a downloaded certificate does not establish
its provenance. Load that public file, not the CA's private key:

```go
roots, err := certificate.LoadBundle("development-ca.pem")
if err != nil {
	return err
}
pool, err := roots.Pool()
if err != nil {
	return err
}
```

`Pool` creates an independent private-only trust pool. It adds no system roots,
so this pool does not also accept ordinary public issuers. With `net/http`,
`time`, `crypto/tls`, and `arkoren.dev/core/http/client` imported, give the pool
to a transport:

```go
transport, err := client.NewTransport(&tls.Config{
	MinVersion: tls.VersionTLS12,
	RootCAs:    pool,
})
if err != nil {
	return err
}
outgoing := &http.Client{Transport: transport, Timeout: 8 * time.Second}
```

Use this client with the [request example](https://arkoren.dev/docs/guides/http-networking/http-client.md#read-responses).
For the generated localhost identity, request `https://localhost:8443` from a
server configured to listen there. The URL must match a SAN; trusting the CA
does not bypass name checks. `RootCAs` verifies the server and does not present
a client identity.

`NewTransport` clones the current default native transport, preserving proxy,
pooling, and HTTP/2 settings. A default that is not a nonnil `*http.Transport`
returns an error; for a base you already own, use
[custom transport ownership](https://arkoren.dev/docs/guides/http-networking/http-client.md#own-transports). When retiring the
owned pool, call `outgoing.CloseIdleConnections()`; active requests continue.

### Add Public Roots

If this client intentionally calls both private services and public services,
replace `roots.Pool()` with an additive system pool:

```go
pool, err := roots.SystemPool()
if err != nil {
	return err
}
```

Use this pool in the same `RootCAs` field. It retains system roots and adds the
bundle's certificates to an independent pool; it does not alter the OS store.
A system-root loading error is returned with its cause, not silently replaced
by private-only trust. A zero bundle fails before system roots are loaded.
Add only issuers you intend this client to trust.

## Require Client Identity

Mutual TLS (mTLS) adds client-certificate verification to server verification.
Obtain a client-purpose identity from your issuer; `GenerateServer` deliberately
does not issue client identities. Load the supplied client chain and key with
the same checked conversion:

```go
identity, err := certificate.LoadIdentity("client.pem", "client-key.pem")
if err != nil {
	return err
}
credentials, err := identity.TLS()
if err != nil {
	return err
}
```

On the client, use `serverRoots`, a pool of CAs trusted to issue server
certificates, and present these credentials:

```go
transport, err := client.NewTransport(&tls.Config{
	MinVersion:   tls.VersionTLS12,
	RootCAs:      serverRoots,
	Certificates: []tls.Certificate{credentials},
})
if err != nil {
	return err
}
outgoing := &http.Client{Transport: transport, Timeout: 8 * time.Second}
```

Sending a client identity does not replace verification of the server. Keep
that separate trust policy and retire the owned client pool as above.

On the server, use `settings` containing its own server identity from
[Load Identities](#load-identities). Before starting it, require verified client
certificates with `clientRoots`, a pool made from the public client-issuing CA
bundle:

```go
settings.ClientCAs = clientRoots
settings.ClientAuth = tls.RequireAndVerifyClientCert
```

`ClientCAs` and the outgoing client's `RootCAs` have different roles and need
not contain the same authorities. A missing, untrusted, expired, or wrong-purpose
client certificate fails the handshake. After authentication, your application
still decides which operations that identity may perform; trusting its CA does
not grant unrestricted access.

## Share Safely

`Identity` and `Bundle` are immutable. Native certificates, pools, PEM buffers,
and TLS exports are detached, including the private key returned by `TLS()`.
Changing an export does not change the original value. Their zero values are
invalid; handle conversion errors or `Err()` before use. Do not log secret
exports even though ordinary identity formatting is redacted.

Once you place exported values into a native `tls.Config`, Go's ownership rules
apply. `tls.Config.Clone` is shallow: referenced pools, certificates, keys, and
callback state remain shared. Finish configuration before sharing it with a
running server or client, and do not mutate it afterward. Native certificate
callbacks can select prepared identities; they do not provide automatic renewal.

The retrying client's default policy treats TLS errors as transport errors for
eligible requests. Retrying does not repair bad trust or names. This differs
from the CLI's [condition-wait policy](https://arkoren.dev/docs/guides/cli/reference.md#retry-rules), which
treats certificate-verification failures as terminal.

For parsing formats, generation constraints, and every export, see the
[`certificate` API reference](https://pkg.go.dev/arkoren.dev/foundation/certificate).

---

Source: https://arkoren.dev/docs/guides/browser-security/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/browser-security/overview)

# Overview

A visitor can stay signed in to your application while browsing an untrusted
site. That site might frame your account page, try to read your API, or submit
a request with cookies the browser attaches automatically. Browser security
policies address these different interactions; no single policy covers them all.

These defenses do not identify the caller or grant access to a resource. Keep
[authentication](https://arkoren.dev/docs/guides/authentication/overview.md) and
[authorization](https://arkoren.dev/docs/guides/authorization/overview.md) on protected operations, including
requests from trusted browser origins and non-browser clients.

## Choose Defenses

An **origin** is a scheme, host, and optional port. A page at
`https://portal.example.com` has a different origin from an API at
`https://api.example.com`, even though both belong to your organization.
Choose each defense for the interaction it controls:

| Interaction | Defense | Boundary |
| --- | --- | --- |
| Another site embeds your page, or a browser guesses its content type | [Secure headers](https://arkoren.dev/docs/guides/browser-security/secure-headers.md) | Response headers instruct the browser how to handle your content. |
| Your separate browser front end needs to read API responses | [CORS](https://arkoren.dev/docs/guides/browser-security/cors.md) | Permission headers tell browser code which cross-origin responses it may access. |
| Another site submits an action using the visitor's cookies | [CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md) | Request-header checks reject unsafe cross-origin browser requests before application work. |

Start with secure headers. Their defaults forbid framing, disable content
sniffing, and restrict referrer disclosure. The default Content Security Policy
only forbids framing: HTML applications need a resource-loading policy suited to
their pages. Enable HSTS only after establishing a lasting HTTPS deployment.

Add CSRF protection wherever browser cookies authorize actions. Keep `GET`,
`HEAD`, and `OPTIONS` free of state changes because the checker always allows
them. Configure CORS only when a different browser origin needs response access;
same-origin requests do not need CORS permission.

CORS is not a write barrier. A disallowed origin can still reach a handler
without permission headers, and a forged form submission can cause damage
without reading its response. Conversely, a trusted CSRF origin does not gain
permission to read responses. A separate cookie-backed front end can need both
policies, plus authentication and resource authorization.

## Place Policies

Apply shared response headers at router scope so they also cover unmatched
requests. With a fresh `routes` router and
`arkoren.dev/core/http/middleware/secureheaders` imported:

```go
routes.Use(secureheaders.Middleware())
```

For account settings authorized by cookies, apply CSRF protection before the
group's authentication and handlers. Import `arkoren.dev/core/http/router` and
`arkoren.dev/core/http/middleware/csrf`; `requireAccount` is your authentication
middleware and `changeEmail` is your existing authorized handler:

```go
routes.Group("/account", func(account *router.Router) {
	account.Use(csrf.Middleware(), requireAccount)
	account.Post("/email", changeEmail)
})
```

The group inherits the router headers; the CSRF policy does not affect unrelated
routes. If `requireAccount` uses sessions, install the
[session middleware](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#enable-sessions) before
it. Register middleware before routes and choose
[scope](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#choose-scope) deliberately: signed
webhook receivers outside this group need their own request verification.

If you add CORS, place it before authentication and body checks so allowed
preflights can finish without credentials. Place secure headers outside CORS
if preflight responses also need those headers. See
[policy ordering](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#order-policies) for the
execution model.

## Verify Boundaries

Check headers on successful responses and errors, then test CORS with allowed
and disallowed origins. A successful server-side test alone does not prove that
a browser can read a response; browser permission headers must also match.

For CSRF tests, send cross-site browser headers and assert that rejection leaves
application state unchanged. Requests without `Sec-Fetch-Site` or `Origin` pass
the CSRF checker; that is compatibility behavior, not evidence of identity.
Use the focused [rejection test](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md#test-rejection) to exercise
the actual boundary.

Begin with [Secure Headers](https://arkoren.dev/docs/guides/browser-security/secure-headers.md) to establish the shared response
policy. For traffic volume and automated submissions, use
[abuse prevention](https://arkoren.dev/docs/guides/abuse-prevention/overview.md) as a separate defense.

---

Source: https://arkoren.dev/docs/guides/browser-security/secure-headers.md
[HTML documentation](https://arkoren.dev/docs/guides/browser-security/secure-headers)

# Secure Headers

An account page should not become an invisible control inside another site's
frame. Security response headers tell browsers whether they may embed your
pages, how to interpret content, and which resources a page may load. The
`secureheaders` middleware supplies a shared baseline; it does not authenticate
requests or reject forged actions.

## Set Defaults

On a fresh [router](https://arkoren.dev/docs/guides/http-fundamentals/routing.md) named `routes`, import
`arkoren.dev/core/http/middleware/secureheaders` and register the policy before
routes:

```go
routes.Use(secureheaders.Middleware())
```

At router scope, requests that reach this layer receive the configured headers,
including unmatched requests. A policy outside this layer can reject requests
before the headers are set; choose [middleware order](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#order-policies)
accordingly.

The baseline sets:

- `X-Frame-Options: DENY` and `Content-Security-Policy: frame-ancestors 'none'`,
  forbidding other pages, including your own, from framing the response;
- `X-Content-Type-Options: nosniff`, disabling content-type guessing rather than
  fixing an incorrect [response content type](https://arkoren.dev/docs/guides/http-fundamentals/responses.md);
- `Referrer-Policy: strict-origin-when-cross-origin`, retaining the full referrer
  for same-origin requests, sending only the origin across origins, and omitting
  it on HTTPS-to-HTTP downgrades;
- `X-XSS-Protection: 0`, disabling the legacy browser XSS auditor.

HSTS and Permissions Policy are omitted. In particular, HTTPS alone does not
enable HSTS. The default Content Security Policy forbids framing but does not
restrict scripts, styles, or images.

## Restrict Resources

If your portal serves scripts, styles, and images from its own origin, restrict
resource loading explicitly. Replace the baseline registration with this
application-specific policy:

```go
routes.Use(secureheaders.Middleware(func(headers *secureheaders.Builder) {
	headers.ContentSecurityPolicy("default-src 'self'; object-src 'none'; base-uri 'self'; frame-ancestors 'none'")
	headers.PermissionsPolicy("camera=(), microphone=(), geolocation=()")
}))
```

The browser limits resources to the page's origin, disables object embeds, and
restricts base URLs. Inline scripts/styles and third-party resources need their
own deliberate policy; test your actual pages before deploying this example.
Permissions Policy separately disables the named browser capabilities.

`ContentSecurityPolicy` replaces the whole default string. Keep
`frame-ancestors 'none'` if framing must remain forbidden. If your own pages
must frame one another, use `frame-ancestors 'self'` and set
`headers.FrameOptions("SAMEORIGIN")` in the same construction callback so the
two framing policies agree. A resource policy reduces exposure; it does not
replace safe HTML rendering or output escaping.

## Enable HSTS

HTTP Strict Transport Security (**HSTS**) tells a browser to remember that a host
must use HTTPS for subsequent requests. Before enabling it, establish
[HTTPS service](https://arkoren.dev/docs/guides/http-networking/server.md) and ensure the host can remain
available over HTTPS for the entire selected duration. HSTS neither configures
TLS nor redirects an initial HTTP visit.

For a host ready to commit to one day, use this alternative registration. If you
also need the resource policy above, keep those settings in the same callback:

```go
routes.Use(secureheaders.Middleware(func(headers *secureheaders.Builder) {
	headers.StrictTransportSecurity("max-age=86400")
}))
```

The middleware sends this value when `request.TLS` is nonnil or
`X-Forwarded-Proto` is exactly `https`; otherwise it omits HSTS. Forwarded
protocol detection has no proxy allowlist. A TLS-terminating proxy must overwrite
client-supplied forwarding headers with the real external protocol. Browsers
ignore HSTS received over plain HTTP; the forwarding header does not turn an
HTTP connection into HTTPS.

> [!WARNING]
> Add `includeSubDomains` only if every subdomain can serve HTTPS for the policy
> lifetime, or remembered policy can make those subdomains unreachable. Add
> `preload` only after reviewing the [preload requirements and removal process](https://hstspreload.org/);
> it is a separate deployment commitment.

If you relied on the previous automatic two-year policy, preserve it explicitly
with `headers.StrictTransportSecurity("max-age=63072000")`. The current default
omits it. Omitting HSTS, or configuring an empty string, does not clear a policy
already remembered by browsers.

## Delegate Headers

If your ingress owns a particular response header, omit that header from this
middleware rather than maintaining two policies. For an ingress-owned referrer
policy, use this alternative configuration:

```go
routes.Use(secureheaders.Middleware(func(headers *secureheaders.Builder) {
	headers.ReferrerPolicy("")
}))
```

An explicit empty setting stays omitted; other settings retain their defaults.
Omission does not remove a header another handler or proxy sets. Review the
final response at the browser-facing edge, not only the application response.

For all header settings, see the [`secureheaders` API reference](https://pkg.go.dev/arkoren.dev/core/http/middleware/secureheaders).

---

Source: https://arkoren.dev/docs/guides/browser-security/cors.md
[HTML documentation](https://arkoren.dev/docs/guides/browser-security/cors)

# CORS

A portal at `https://portal.example.com` cannot normally read responses from
`https://api.example.com`. Cross-Origin Resource Sharing (**CORS**) lets the API
grant that browser access with response headers. An origin consists of a scheme,
host, and optional port; changing any of them changes the origin.

CORS is a browser response-access policy, not authentication or a write barrier.
Disallowed origins can still reach your handlers without CORS permission
headers. Keep authentication, resource authorization, and
[CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md) where cookies authorize actions.

## Allow a Front End

For a portal that reads account details and sends JSON changes with a bearer
token, allow its exact origin, methods, and request headers. Import `net/http`
and `arkoren.dev/core/http/middleware/cors`. On a fresh API router `routes`,
register this before authentication, body checks, and routes:

```go
routes.Use(cors.Middleware(func(policy *cors.Builder) {
	policy.AllowedOrigins("https://portal.example.com")
	policy.AllowedMethods(http.MethodGet, http.MethodPatch)
	policy.AllowedHeaders("Content-Type", "Authorization")
}))
```

Requests with that `Origin` receive permission headers. `Authorization` must be
listed when browser code supplies a bearer token; listing it does not validate
the token. The policy applies to every route on this router, including unmatched
requests, so the portal can read eligible error responses too. Choose narrower
[group scope](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#choose-scope) if only part of
the API shares the policy.

Use bare ASCII origins, without a trailing slash. Construction panics for
missing schemes or hostnames, userinfo, paths (including `/`), query or fragment
delimiters (including empty `?` or `#`), wildcard host patterns, and invalid or
empty ports. Decimal ports must be in `0`–`65535`; international hostnames need
ASCII punycode. Only the complete `*` wildcard is supported.

Configured origins are lowercased, not fully canonicalized. For example,
`https://portal.example.com:443` does not match a browser's
`https://portal.example.com`. Configure the spelling the browser sends in
`Origin`; this validation is not a general URL sanitizer.

## Handle Preflights

Before a cross-origin JSON `PATCH`, the browser asks whether the API permits
the method and headers. This **preflight** is an `OPTIONS` request with
`Origin` and `Access-Control-Request-Method`:

```http
OPTIONS /account/preferences HTTP/1.1
Host: api.example.com
Origin: https://portal.example.com
Access-Control-Request-Method: PATCH
Access-Control-Request-Headers: authorization, content-type
```

For an allowed origin, the middleware answers `204 No Content` without calling
the next layer. It advertises your configured method and header lists; the
browser checks those lists before sending the actual request. A `204` alone
does not mean the requested method or headers were approved.

Requests without `Origin`, requests from disallowed origins, and ordinary
`OPTIONS` requests without `Access-Control-Request-Method` continue to the next
layer. Allowed non-preflight requests also continue. CORS does not reject an
ordinary request because its method is absent from `AllowedMethods`.

Preflights arrive without credentials, so CORS must precede authentication.
Otherwise an authentication failure can stop the browser before it sends its
authenticated request. See [policy ordering](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#order-policies).
Non-browser clients do not enforce these permission headers.

Preflight answers default to `Access-Control-Max-Age: 300`, allowing five minutes
of caching, subject to browser limits. To disable this advertised cache lifetime,
add `policy.MaxAge(0)` to the construction callback. Positive durations are
rounded down to whole seconds: `1500*time.Millisecond` sends `1`. Nonpositive
durations and positive durations below one second omit the header; import `time`
when using duration expressions.

## Send Credentials

If the portal uses session cookies instead of bearer tokens, explicitly allow
credentials for that trusted origin. Replace the previous registration with
this cookie-backed variant:

```go
routes.Use(cors.Middleware(func(policy *cors.Builder) {
	policy.AllowedOrigins("https://portal.example.com")
	policy.AllowedMethods(http.MethodGet, http.MethodPatch)
	policy.AllowedHeaders("Content-Type")
	policy.AllowCredentials(true)
}))
```

Credentials are off by default. Combining `AllowCredentials(true)` with any `*`
origin entry panics at construction. Never replace a trusted origin list with
arbitrary reflection of the request's origin.

Browser code must also opt in to credentials, for example with
`fetch(url, { credentials: "include" })`. Cookie `SameSite`, `Secure`, and browser
privacy rules still apply; CORS does not override them. Configure
[session cookies](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#enable-sessions) appropriately
and retain CSRF protection, with a [trusted origin](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md#trust-browser-origins)
for intentional cross-origin writes. Trusting an origin for response access neither
authenticates its visitors nor authorizes their account changes.

## Expose Metadata

If the portal displays an API request ID for support, allow script access to
that response header. This separate construction example assumes your server
already sets `X-Request-ID`:

```go
routes.Use(cors.Middleware(func(policy *cors.Builder) {
	policy.AllowedOrigins("https://portal.example.com")
	policy.ExposedHeaders("X-Request-ID")
}))
```

`ExposedHeaders` names response headers beyond the browser's basic readable
set; it does not create their values. Include the method, request-header, and
credential settings from your chosen policy when combining these examples.
With an explicit origin list, responses vary on `Origin`, including requests
without that header. Recognized allowed preflights also vary on requested
method and headers, so shared caches can distinguish permission answers.

## Refine Defaults

Use `cors.Middleware()` without configuration only when you intend public
cross-origin response access without credentials. It allows `*`, advertises
`GET`, `POST`, and `HEAD`, and allows request headers `Accept` and `Content-Type`.
No extra response headers are exposed by default.

List setters replace earlier lists. After all callbacks run, omitted or nil
origin, method, and request-header lists receive defaults; nonnil empty lists
remain empty. To grant no origins explicitly:

```go
routes.Use(cors.Middleware(func(policy *cors.Builder) {
	policy.AllowedOrigins([]string{}...)
}))
```

That omits origin permission but still passes requests onward. In contrast,
`policy.AllowedOrigins()` supplies nil and restores the wildcard default. The
same distinction applies to `AllowedMethods` and `AllowedHeaders`: explicit
empty lists omit their preflight headers. Configuration lists are copied during
construction, and validation uses the final settings after all callbacks.

For every setting, see the [`cors` API reference](https://pkg.go.dev/arkoren.dev/core/http/middleware/cors).

---

Source: https://arkoren.dev/docs/guides/browser-security/csrf-protection.md
[HTML documentation](https://arkoren.dev/docs/guides/browser-security/csrf-protection)

# CSRF Protection

A hostile page can submit a request to your application without knowing the
visitor's session cookie. If cookie scope and `SameSite` rules allow it, the
browser attaches that cookie automatically. Accepting the request solely because
it has a valid session can let the hostile page change the visitor's settings.
This is **cross-site request forgery** (CSRF).

Arkoren's `csrf` middleware uses Go's `http.CrossOriginProtection` to check
browser-supplied headers before application work. It needs no form tokens and
stores nothing in the session. It does not replace authentication or the
resource authorization inside your handlers.

## Protect Cookie Actions

Put cookie-authorized actions behind the middleware, before authentication and
handlers. With a fresh router `routes`, import `arkoren.dev/core/http/router`
and `arkoren.dev/core/http/middleware/csrf`. Here `requireAccount` is your
authentication middleware and `savePreferences` is your existing authorized
handler:

```go
routes.Group("/account", func(account *router.Router) {
	account.Use(csrf.Middleware(), requireAccount)
	account.Patch("/preferences", savePreferences)
})
```

Later group policy examples replace this `Use` call; configure the chosen
policy before registering the group's routes.

The policy rejects unsafe cross-origin browser requests before either inner
layer runs. Protect login and other cookie-backed actions too, not only routes
that already require a signed-in user. If authentication uses sessions, install
[session middleware](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#enable-sessions) before
it; the CSRF checker itself does not need a session. Session changes still follow
the normal [save behavior](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#save-changes).

## Understand Checks

`GET`, `HEAD`, and `OPTIONS` always pass. Never use these methods for logout,
deletion, or other state changes: those actions would bypass this protection.
For other methods, the checker uses this order:

1. If `Sec-Fetch-Site` is nonempty, `same-origin` and `none` pass. Other values,
   including `same-site`, fail unless an explicit exception applies. Sharing a
   parent domain is not enough to trust another origin.
2. Otherwise, a nonempty `Origin` passes if its parsed host, including any port,
   equals the request's `Host`. This fallback does not compare schemes; HTTPS and
   a deliberate [HSTS policy](https://arkoren.dev/docs/guides/browser-security/secure-headers.md#enable-hsts) help address the
   older-browser boundary.
3. If both headers are missing or empty, the request passes as a possible
   same-origin or non-browser request.

Browser pages cannot freely set these headers, but server clients can supply or
omit them. Passing the checker is never proof of client identity. Modern
browsers provide the fetch-context header; do not assume old clients implement
it or enforce the session cookie's default `SameSite=Lax` behavior.

The Host fallback uses the request as it reaches the application, not
`X-Forwarded-Host`. Configure reverse proxies to preserve the intended public
host and browser security headers rather than replacing them with internal or
client-selected values.

A blocked request returns `csrf.ErrBlocked` without calling the next layer.
With the default router renderer and an acceptable JSON representation, the
client receives a `403` problem with this identity:

```json
{
  "type": "tag:arkoren.dev,2026-09:core:http/middleware/csrf:blocked",
  "title": "Cross-Origin Request Blocked",
  "status": 403,
  "detail": "The request was rejected because its origin or fetch context did not meet security requirements."
}
```

The router's [error negotiation](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md#negotiate-error-output)
still applies, including `406` if the client excludes every supported format.

## Trust Browser Origins

If your portal intentionally sends account changes to a separate API origin,
trust its exact browser origin rather than every subdomain. Replace the default
CSRF registration in the account group with:

```go
account.Use(csrf.Middleware(func(protection *csrf.Builder) {
	protection.TrustedOrigins("https://portal.example.com")
}), requireAccount)
```

An exact matching `Origin` passes the CSRF check even for cross-site fetch
context. Use `scheme://host[:port]` without a path; malformed origins panic
during middleware construction. This is permission to pass this checker, not
permission to change an account. The authenticated visitor still needs resource
authorization, and the portal needs a separate [CORS policy](https://arkoren.dev/docs/guides/browser-security/cors.md#send-credentials)
to read credentialed API responses. Cookies must also permit the request.

## Isolate Exceptions

Some routes intentionally accept cross-site posts. Keep a signature-verifying
webhook receiver outside the protected browser group rather than weakening the
group. On a fresh router without global CSRF middleware, `receivePayment` is
your configured, signature-verifying handler:

```go
routes.Post("/callbacks/payment", receivePayment)
routes.Group("/account", func(account *router.Router) {
	account.Use(csrf.Middleware(), requireAccount)
	account.Patch("/preferences", savePreferences)
})
```

The callback must perform its own [webhook verification](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md).
A browser login callback accepting cross-site `POST` likewise needs its login
flow's callback checks. Being outside the group provides no verification.

If scoping cannot isolate a verified callback, supply a native checker with one
narrow bypass. Import `net/http` and use this alternative global policy:

```go
engine := http.NewCrossOriginProtection()
engine.AddInsecureBypassPattern("POST /callbacks/payment")

routes.Use(csrf.Middleware(func(protection *csrf.Builder) {
	protection.Protection(engine)
}))
```

The bypass disables CSRF checks for matching requests; it does not verify them.
Patterns follow `http.ServeMux` syntax and precedence, and malformed or
conflicting patterns panic. Broad path patterns can silently expose unrelated
actions, so keep the method and callback path explicit.

The provided checker remains shared. Configuring `TrustedOrigins` adds origins
to that same instance; later native changes affect future requests. Do not
treat it as an isolated copy or modify it casually after publication.

## Check JSON Inputs

JSON syntax alone is not a browser boundary: a hostile page can send JSON-looking
bytes labeled `text/plain` without a CORS preflight. If your account group accepts
only JSON, add `arkoren.dev/core/http/middleware/contenttype` and use this policy
before body decoding:

```go
account.Use(csrf.Middleware(), contenttype.Middleware(), requireAccount)
```

The default media-type policy accepts `application/json` and `+json` types,
rejecting disallowed form and plain-text declarations. Missing headers on empty
requests can pass; see [empty-request checks](https://arkoren.dev/docs/guides/http-fundamentals/content-type.md#check-empty-requests).
The media-type check neither validates the JSON contents nor replaces CSRF or
resource authorization. Keep CSRF protection when allowing browser form formats.

An API authenticated exclusively by an explicit bearer token in `Authorization`
does not have the automatic-cookie credential risk: browsers do not attach that
header on their own. You can omit CSRF middleware for that token-only boundary.
If the same API also accepts session cookies, retain it. Non-browser token clients
without browser headers pass; browser token clients are still subject to the
header checks and any configured trusted origins.

## Observe Rejections

To receive rejection notifications, pass your existing `*event.Bus` named `bus`
in a construction callback, keeping authentication after the checker:

```go
account.Use(csrf.Middleware(func(protection *csrf.Builder) {
	protection.Events(bus)
}), requireAccount)
```

Each blocked request emits `csrf.Blocked` with its method, path, `Origin`, and
native error. With no bus, nothing is published. Listeners run synchronously and
cannot change the rejection. Paths can contain personal identifiers, and origin
and error values are diagnostic, potentially untrusted data; review them before
logging or exporting them. A rejection can indicate either an attack or a
misconfigured browser policy, not proof of either.

If clients need different failure wording, `Builder.ErrorResponse` accepts a
`problem.Details` value. Derive from `csrf.ErrBlocked` to preserve the `403` status
and stable problem identity, and keep internal causes out of client-facing text.

## Test Rejection

An ordinary `httptest.NewRequest` lacks browser headers and passes the checker.
Test a cross-site write explicitly and make any downstream execution fail the
test. With `net/http`, `net/http/httptest`, and `t *testing.T`:

```go
handler := csrf.Middleware()(func(http.ResponseWriter, *http.Request) error {
	t.Fatal("rejected request reached application work")

	return nil
})

incoming := httptest.NewRequest(http.MethodPatch, "/account/preferences", nil)
incoming.Header.Set("Sec-Fetch-Site", "cross-site")
incoming.Header.Set("Origin", "https://untrusted.example")
response := handler.Record(incoming)
```

Register cleanup for the recorded response body and assert the rejection:

```go
t.Cleanup(func() {
	if err := response.Body.Close(); err != nil {
		t.Fatal(err)
	}
})
if response.StatusCode != http.StatusForbidden {
	t.Fatalf("status = %d, want 403", response.StatusCode)
}
```

For your configured routes, also test that blocked requests leave stored data
unchanged, trusted origins can reach the intended policy layers, and authentication
and authorization still reject visitors without permission.

For configuration and events, see the [`csrf` API reference](https://pkg.go.dev/arkoren.dev/core/http/middleware/csrf).

---

Source: https://arkoren.dev/docs/guides/abuse-prevention/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/abuse-prevention/overview)

# Abuse Prevention Overview

A public contact form can receive valid input faster than your application can
process it. Abuse prevention controls how much work a caller can trigger and
whether a browser submission needs additional verification. Choose these controls
for the operation you are protecting, rather than applying one policy to every request.

## Choose Controls

Use [rate limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) to give callers a request budget over a time
window. The cache-backed middleware rejects requests over that budget before
calling the protected handler. It counts admitted attempts, not successful
business outcomes: an attempt can consume budget even if the handler later fails.

Add [CAPTCHA verification](https://arkoren.dev/docs/guides/abuse-prevention/captcha.md) when a browser submission needs another
abuse signal. Cloudflare Turnstile verifies a token against your hostname and
action policy. Google reCAPTCHA returns a risk score as well; your application
chooses whether to accept that score, deny the submission, or require another factor.

Neither control establishes [identity](https://arkoren.dev/docs/guides/authentication/overview.md) or grants
[permission](https://arkoren.dev/docs/guides/authorization/overview.md). An authenticated user can exceed an
abuse budget, and an accepted CAPTCHA token does not authorize access to another
user's data. Browser session endpoints still need the appropriate
[CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md).

## Place Checks

For a JSON contact submission, reject an unsupported
[content type](https://arkoren.dev/docs/guides/http-fundamentals/content-type.md) and bound the
[body size](https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md) before decoding. Apply a rate
budget before expensive work, including external CAPTCHA calls. Decode and
validate the contact input, verify its token once, then perform the contact effect.
Follow [middleware ordering](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#order-policies)
and [scope selection](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#choose-scope) when placing
those checks on your routes.

The budget's key determines who shares it. An IP key can group unrelated people
behind one network; an account key requires authentication before the limiter.
Behind a reverse proxy, trust only the proxy networks you control, not arbitrary
forwarded headers. Separate unrelated budgets with distinct cache names, and use
a shared backend when replicas must consult the same counters. Backend failures
stop the protected request; they do not disable the limiter.

For CAPTCHA, choose handler verification when the token is part of JSON. Use the
optional header middleware instead when the token arrives separately. Do not use
both for one token: verification consumes a single-use credential. If verification
fails or its outcome is unknown, stop the effect and obtain a fresh token for a
later attempt, rather than retrying the same verification.

A fresh token does not make a repeated submission safe. Keep duplicate-effect
handling in the business operation; neither rate limiting nor CAPTCHA promises
exactly-once execution. Likewise, [signed links](https://arkoren.dev/docs/guides/http-capabilities/secure-links.md)
protect URL integrity and expiry, not request volume or general permission.

## Check Rejections

Test that an over-budget request never reaches the handler, and that CAPTCHA
rejection or provider unavailability leaves the protected effect untouched. For
Google, include a valid token whose score fails your policy. Exercise different
client keys and the backend failure path, not only successful requests.

Use [request logging](https://arkoren.dev/docs/guides/observability/request-logging.md) for request outcomes
and [cache monitoring](https://arkoren.dev/docs/guides/cache/monitoring.md) for backend failures. Keep browser
tokens and credentials out of logs. Client keys and concrete URL paths can also
identify people or resources; restrict and redact them before export.

Start with a [route budget](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md), then add
[browser verification](https://arkoren.dev/docs/guides/abuse-prevention/captcha.md) where that operation needs it.

---

Source: https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md
[HTML documentation](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting)

# Rate Limiting

A password-reset endpoint can trigger hashing, mail, or external calls even when
its caller never completes the flow. A rate budget bounds admitted attempts before
that work runs. The `ratelimit` middleware counts requests by a client key in a
cache and returns `429 Too Many Requests` when its estimate exceeds your limit.

## Route Budgets

Use your configured cache facade `store`; [Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md)
owns backend construction and lifecycle. A memory backend keeps independent
budgets in each process. If replicas must consult the same counters, give them
the same shared backend and key policy. This does not make the limiter an exact
distributed request log.

For an existing router `routes` and password-reset handler `reset`, import
`time`, `arkoren.dev/core/service/cache`, and
`arkoren.dev/core/http/middleware/ratelimit`. Protect only that route:

```go
routes.With(ratelimit.Middleware(store, func(limiter *ratelimit.Builder) {
	limiter.Limit(4)
	limiter.Window(10 * time.Minute)
	limiter.Namespace(cache.NewNamespace[int64]("app:password-reset"))
})).Post("/password-reset", reset)
```

This policy admits up to its sliding-window estimate of four requests per client
over ten minutes. Place it before expensive verification or handler work. Choose
the appropriate [middleware scope](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#choose-scope)
and [execution order](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#order-policies); a
route-specific limiter does not protect other routes.

The registration snippets on this page are alternatives. Combine the settings
you need in one construction rather than registering the same route repeatedly.

If you intentionally want the default policy across a router's subsequently
registered routes, register it before those routes:

```go
routes.Use(ratelimit.Middleware(store))
```

The default is 60 requests per minute per client IP. Omitting `Limit` or `Window`,
or setting either to zero, restores that setting's default; zero does not disable
limiting. Positive `int` limits are supported. A nonzero window must be positive
and at most `time.Duration(math.MaxInt64/2)` so retention fits two windows.

The default namespace is `arkoren:ratelimit`, with `":"` separators. The first
example instead stores counters named
`app:password-reset:<client key>:<window index>`. Distinct policies need distinct
physical names; a namespace is naming, not an access-control boundary. Read
[Cache Keys](https://arkoren.dev/docs/guides/cache/keys.md) before sharing or changing prefixes and separators.
Changing a deployed name starts separate counters and can reset an active budget.
For a namespace previously using dots, explicitly keep `namespace.Separator(".")`
in its `cache.NewNamespace` callback; new namespaces default to colons.

Construction callbacks run once in order, then final settings are validated. A
nil cache, negative limit, invalid window, or zero/invalid namespace panics. A
later callback can replace an earlier invalid setting before validation; a nil
callback itself panics.

## Client Keys

By default, the limiter uses `RemoteAddr` without its port, unmaps IPv4-mapped
IPv6 addresses, and groups IPv6 addresses by `/64`. It does not trust forwarded
headers. People behind one public address share a budget, so choose limits with
that grouping in mind.

Behind a reverse proxy, configure only networks belonging to trusted proxies.
With `net/netip` imported, this alternative policy uses the example proxy subnet;
replace it with your actual deployment's trust boundary:

```go
routes.With(ratelimit.Middleware(store, func(limiter *ratelimit.Builder) {
	limiter.KeyFunc(ratelimit.TrustedProxyKey(
		netip.MustParsePrefix("192.0.2.0/24"),
	))
})).Post("/password-reset", reset)
```

`TrustedProxyKey` ignores `X-Forwarded-For` unless the immediate peer is trusted.
It walks trusted chains right to left and uses the first untrusted address.
Malformed entries or an entirely trusted chain fall back to the nearest trusted
hop that reported them, rather than turning attacker-chosen text into a key.
The resulting addresses use the same IPv6 grouping as the default.

For an account budget, supply `KeyFunc` that reads your application's verified
identity. Run authentication before that limiter and return a stable, nonempty
key for each admitted user. Do not let a request body or arbitrary account header
choose whose budget is charged. A nil `KeyFunc` restores IP-based keys.

Client keys appear in cache names and optional denial events. Prefer an opaque
application identifier over an email, credential, or token. Account budgets
control volume; your application must still check permission for the operation.

## Counting Behavior

The limiter divides time into fixed windows aligned to the Unix epoch. Each
request atomically increments the current window counter. If the counter has no
expiry, the operation gives it twice-window retention; an existing expiry is preserved.
It then reads the previous counter and adds a decreasing
share of that count to the current count. That weighted sum estimates usage over
the last window length.

The estimate assumes requests were evenly distributed within the previous
window, so it is not an exact rolling log. Replicas also need reasonably
synchronized clocks. Integer-ratio calculations preserve large-limit precision
and round recovery upward at nanosecond boundaries, but do not remove these
assumptions. No cache lock serializes the complete increment/read/refund sequence.

An over-budget request is refunded with a decrement and never reaches the
handler. Repeated denied requests therefore do not extend their own block when
refunds succeed. Concurrent requests can briefly observe an increment before
its refund and be denied conservatively. An admitted request remains counted
even if its handler later returns an error.

The backend must implement atomic `IncrementWithTTL`; unsupported drivers return
`cache.ErrUnsupported`. Increment, previous-counter read, or refund errors stop
the request without calling the handler. A nonpositive counter after increment
also fails the request; a missing or nonpositive previous counter counts as zero.
Counter retention overrides the namespace's write TTL.

These failures are not rate-limit denials or permission to bypass protection.
The middleware returns backend errors, wrapping refund failures with context;
it does not sanitize native backend diagnostics. The default HTTP renderer hides
ordinary unclassified backend errors behind a generic `500`, but your logging or
custom error handling can still expose them. Use [cache monitoring](https://arkoren.dev/docs/guides/cache/monitoring.md) and a
[shared telemetry handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle) for backend
health, with an explicit privacy policy for error data.

## Response Timing

After successful counter operations, responses carry these headers:

| Header | Meaning |
| --- | --- |
| `RateLimit-Limit` | Configured request count. |
| `RateLimit-Remaining` | Estimated available requests, rounded down; zero on denial. |
| `RateLimit-Reset` | On admission, seconds until the full budget recovers if traffic stops. On denial, seconds until one request fits. |
| `Retry-After` | On denial only, the same recovery interval as `RateLimit-Reset`. |

Recovery intervals round up to whole seconds and are at least one second. They
assume no intervening traffic: another caller sharing the key can spend the
recovered budget. These are estimate-based delays, not the age of the oldest
individual request. Counter failures return before these headers are set.

To change denial text without accidentally changing its `429` status or problem
identity, derive from `ratelimit.ErrRateLimited`. Import
`arkoren.dev/foundation/problem` for the callback:

```go
failure := ratelimit.ErrRateLimited.Derive(func(details *problem.Builder) {
	details.Detail("The reset budget is exhausted. Try again later.")
})
```

Use that value when constructing the route's limiter:

```go
routes.With(ratelimit.Middleware(store, func(limiter *ratelimit.Builder) {
	limiter.ErrorResponse(failure)
})).Post("/password-reset", reset)
```

Only a completely zero `problem.Details{}` selects the default denial. A nonzero
problem with no status is preserved; the HTTP renderer fills missing standard
members, including status `500`. Its identity, detail, headers, extensions, and
internal causes are not discarded. Denial still skips the handler and sets
`Retry-After` regardless of your custom problem's status. Causes are internal
diagnostics, not response text.

## Denial Events

Events are off by default. To inspect denials, configure your existing
[`*event.Bus`](https://arkoren.dev/docs/guides/events/overview.md) when constructing the route's limiter:

```go
routes.With(ratelimit.Middleware(store, func(limiter *ratelimit.Builder) {
	limiter.Events(bus)
})).Post("/password-reset", reset)
```

Normally the first denial for each key in each fixed window publishes
`ratelimit.Exceeded` (`http.ratelimit.exceeded`). Its payload contains `Key`,
`Method`, `Path`, `Limit`, and `Window`. Publication does not depend on a custom
denial problem retaining status `429`.

With events enabled, each denial attempts a cache `Add` for a marker retained
for twice the window. An existing marker suppresses publication; a marker-write
failure still publishes and can produce duplicates. This best-effort mechanism
does not weaken the denial, but it is neither exactly-once delivery nor a durable
audit log. Consumers must tolerate duplicate events.

`Key` is the client portion, without the namespace or window index. `Path` is the
actual URL path, not a route template. Both can identify people or resources:
restrict access, redact before export, and choose retention deliberately. Never
send the event payload to a client as an explanation of its denial.

For all options, see the [`ratelimit` API reference](https://pkg.go.dev/arkoren.dev/core/http/middleware/ratelimit).

---

Source: https://arkoren.dev/docs/guides/abuse-prevention/captcha.md
[HTML documentation](https://arkoren.dev/docs/guides/abuse-prevention/captcha)

# CAPTCHA

A feedback form can receive automated submissions even when every field passes
validation. CAPTCHA adds a provider-issued browser token that your server verifies before
accepting the feedback. An accepted token is an abuse signal, not proof of identity
or permission; it does not replace [CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md).

Arkoren ships vendor-local verifiers and optional route middleware. There is no
core CAPTCHA contract, and the two providers' tokens are not interchangeable.

## Provider Choice

Choose Cloudflare Turnstile for an accepted-or-rejected token check. Choose Google
reCAPTCHA when your application needs a risk assessment and owns the decision to
accept, deny, or require more verification. A valid Google token can have a low
score: direct `Verify` returns that assessment successfully. Only the optional
middleware applies a configured minimum score automatically.

### Compatibility Matrix

This comparison covers Arkoren's server-side adapters, not every vendor feature
or a guarantee that a visitor is human.

| Provider | Hostname Allowlist | Expected Action | Risk Score | Route Middleware | Middleware Score Policy |
| --- | --- | --- | --- | --- | --- |
| [Cloudflare Turnstile](#cloudflare-turnstile) | Yes | Yes | No | Yes | No |
| [Google reCAPTCHA](#google-recaptcha) | Yes | Yes | Yes | Yes | Yes — Requires an explicit finite minimum in [0, 1]. |

Both adapters require an allowlist from trusted deployment configuration. Supply
bare hostnames, not URLs, wildcards, or a list derived from the incoming `Host`
header. Neither implicitly trusts subdomains. Turnstile compares case-sensitively;
Google compares case-insensitively.

Choose an action literal on the server, such as `feedback`, and use it in the
browser integration too. Empty `ExpectedAction` or middleware `Action` skips the
local action comparison. Do not copy the expected value from the submitted body.

Create one verifier during application setup, handle construction errors before
mounting routes, and reuse it concurrently. Neither verifier has lifecycle or
readiness hooks. Keep secrets and API credentials on the server; browser site
keys are public.

## Cloudflare Turnstile

[Create a Turnstile widget](https://developers.cloudflare.com/turnstile/get-started/)
for your deployment's hostnames. Store its secret in server configuration and
use its public site key in the browser. From your application module, install:

```sh
go get arkoren.dev/driver/cloudflare
```

### Server Setup

Import `arkoren.dev/driver/cloudflare/cloudflarecaptcha`. With `secret` loaded
from server configuration, construct the verifier:

```go
verifier, err := cloudflarecaptcha.New(secret, []string{"app.example.com"})
```

Construction validates local settings without network I/O. A blank secret, empty
allowlist, or invalid hostname returns `ErrConfiguration`; handle `err` before
using `verifier`. Verification uses Turnstile's HTTPS Siteverify endpoint.

The default verification deadline is ten seconds, including response reading.
If feedback needs a shorter budget, import `time` and use this alternative
construction:

```go
verifier, err := cloudflarecaptcha.New(secret, []string{"app.example.com"},
	func(connection *cloudflarecaptcha.Builder) {
		connection.Timeout(2 * time.Second)
	})
```

An earlier caller deadline still wins. Zero restores ten seconds; negative
timeouts fail construction. `Builder.HTTPClient` can borrow an existing
`*http.Client`: construction snapshots its fields, disables redirects in the
snapshot, and retains its transport and cookie jar without modifying your client.
A shorter native timeout still applies. Configure shared objects before
concurrent use; the verifier closes neither client nor transport.

Verification makes one attempt and does not send Siteverify's optional
idempotency key. Do not add a retrying transport: a timeout can mean the token
was consumed. Empty tokens or tokens over 2,048 bytes reject before I/O. Nil
error means provider success, a nonzero challenge timestamp, an allowed hostname,
and a matching action when requested. Returned `CData` remains untrusted,
potentially sensitive client-originated metadata, not a permission claim or risk score.

Formatting a verifier value or pointer with `%v`, `%+v`, or `%#v` emits
`[REDACTED Turnstile verifier]` without traversing its borrowed objects. This is
not a general sanitization boundary for reflection, serialization, invalid format
verbs, or native objects, and does not apply to the Google verifier. There is no
explicit secret-zeroization operation.

### Browser Token

Define the token callback before loading the widget script. Replace
`TURNSTILE_SITE_KEY` with your public key:

```html
<script>
  function feedbackToken(token) {
    // Attach token as captcha_token to the pending feedback submission.
  }
</script>
<script src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer></script>
<div class="cf-turnstile" data-sitekey="TURNSTILE_SITE_KEY"
     data-action="feedback" data-callback="feedbackToken"></div>
```

The callback receives a credential, not verification success. Attach it to your
submission and wait for the server's result. Clear stored tokens on widget error
or expiry and reset the widget before another attempt. Traditional forms instead
receive a `cf-turnstile-response` field; decode that representation explicitly
rather than treating it as JSON.

[Turnstile tokens](https://developers.cloudflare.com/turnstile/get-started/server-side-validation/)
expire after five minutes and are single-use. Continue with
[JSON verification](#json-verification), or choose [header middleware](#header-middleware)
instead; do not verify the same token in both places.

For constructor and evidence details, see the
[Cloudflare CAPTCHA API reference](https://pkg.go.dev/arkoren.dev/driver/cloudflare/cloudflarecaptcha).

## Google reCAPTCHA

Create a Google Cloud project, enable the reCAPTCHA Enterprise API, and
[create a score-based website key](https://docs.cloud.google.com/recaptcha/docs/create-key-website)
for your hostnames. Use that project and site key on the server. Install:

```sh
go get arkoren.dev/driver/gcp
```

### Server Setup

Configure Application Default Credentials (ADC) for assessment creation, with
`roles/recaptchaenterprise.agent` on the project. Import
`arkoren.dev/driver/gcp/gcpcaptcha`. With the setup context `ctx`, trusted
`projectID`, and public `siteKey`, construct the verifier:

```go
verifier, err := gcpcaptcha.New(ctx, projectID, siteKey, []string{"app.example.com"})
```

Construction validates settings and loads credentials; it does not create an
assessment or perform readiness checks. ADC discovery can do credential-loading
I/O, and its synchronous discovery call cannot be interrupted by `ctx`. The
constructor checks cancellation around discovery and uses `ctx` for quota-project
lookup, but does not retain it. Later native token refresh uses each verification
request's bounded context; asynchronous background refresh is disabled.

If you use a restricted server API key instead of ADC, pass it in this alternative
construction. `apiKey` is not the public browser site key:

```go
verifier, err := gcpcaptcha.New(ctx, projectID, siteKey, []string{"app.example.com"},
	func(access *gcpcaptcha.Builder) {
		access.APIKey(apiKey)
	})
```

This bypasses ADC and sends the key as `X-Goog-Api-Key`, never in the URL. Keep
the key and credential-bearing construction draft out of logs.

### Borrowed Clients

If your application already owns an authenticated `*http.Client` named `client`,
select it instead of ADC:

```go
verifier, err := gcpcaptcha.New(ctx, projectID, siteKey, []string{"app.example.com"},
	func(access *gcpcaptcha.Builder) {
		access.HTTPClient(client)
	})
```

A nonnil client bypasses ADC completely. Without `APIKey`, its transport must
authenticate requests. If you intentionally configure both, the adapter adds
the API-key header without removing transport authentication. A nil client
restores normal ADC/API-key selection.

Construction snapshots the client, discards its jar, and disables redirects.
Its transport remains borrowed and is never closed. Positive client timeouts
shorter than ten seconds are retained; zero, negative, or longer timeouts become
ten seconds. Each verification, including ADC token refresh, also has a ten-second
context limit and honors an earlier caller deadline. Custom transports must be
concurrency-safe, honor cancellation, and never retry tokens. The builder has no
timeout or score-threshold setting; use a shorter caller context and choose score
policy separately.

This adapter uses `projects.assessments.create` (`CreateAssessment`), not legacy
Siteverify or a shared-secret flow. It makes one attempt. Empty tokens and tokens
over 64 KiB reject locally; that size is an application guard, not a documented
Google maximum. Successful assessments expose `Score`, `Name`, and reason/evidence
fields. They do not authenticate a visitor or expose other Fraud Defense APIs.

### Browser Token

Load the Enterprise script with your public score-based site key:

```html
<script src="https://www.google.com/recaptcha/enterprise.js?render=RECAPTCHA_SITE_KEY"></script>
```

In your existing feedback-submit callback, execute the matching action. Replace
the key here with the same value:

```js
grecaptcha.enterprise.ready(async () => {
  const token = await grecaptcha.enterprise.execute("RECAPTCHA_SITE_KEY", {
    action: "feedback",
  });
  // Attach token as captcha_token to this feedback submission.
});
```

Handle script-load and execution errors in your UI; do not submit as though
verification succeeded. Generate the token at the interaction, not when the page
first opens. [Google tokens](https://docs.cloud.google.com/recaptcha/docs/create-assessment-website)
expire after two minutes and can be assessed only once. Obtain a fresh token after
expiry or a failed submission. Continue with [JSON verification](#json-verification)
or the alternative [header middleware](#header-middleware).

For authentication and assessment fields, see the
[Google CAPTCHA API reference](https://pkg.go.dev/arkoren.dev/driver/gcp/gcpcaptcha).

## JSON Verification

For a token inside JSON, let the handler decode the body once and verify the token
before its effect. Apply [body limits](https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md),
[content-type checks](https://arkoren.dev/docs/guides/http-fundamentals/content-type.md), and
[rate limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) before external verification;
`request.JSON` does not enforce `Content-Type` itself. See
[middleware ordering](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md#order-policies) when
placing those checks.

Use one provider branch below, not both. Neither adapter retries automatically,
and custom HTTP transports must not replay a verification. If a response is lost,
the token might already be spent: stop the effect and ask for a fresh token on a
later attempt. A fresh token does not deduplicate feedback or any other business
operation; application-owned idempotency remains separate.

### Map Failures

Direct `Verify` returns plain classified errors, not HTTP problems. Before
registering a Turnstile handler, map them with
`arkoren.dev/core/http/middleware/errormap`, `arkoren.dev/foundation/problem`, and
`net/http`. Here `routes` is your existing router:

```go
routes.Use(errormap.Middleware(
	errormap.Map(cloudflarecaptcha.ErrRejected, problem.Forbidden()),
	errormap.Map(cloudflarecaptcha.ErrUnavailable, problem.New(func(details *problem.Builder) {
		details.Status(http.StatusServiceUnavailable)
	})),
	errormap.Map(cloudflarecaptcha.ErrConfiguration, problem.InternalServerError()),
))
```

For Google, replace the three sentinel qualifiers with `gcpcaptcha`. Rejection
then returns `403`, provider unavailability `503`, and configuration failure
`500`, without publishing provider diagnostics. Without mapping, a plain rejection
gets the router's generic `500`, not `403`. Cancellation and deadline causes remain
identifiable with `errors.Is`; they are operational failures, not bad tokens.

Both adapters keep native diagnostics out of their `Error()` text. Original
causes remain inspectable with `errors.Is` and `errors.As`, however, and can contain
confidential transport or provider data. Do not dump unwrapped causes or assume
arbitrary formatting and serialization of wrappers sanitizes them.

### Decode Input

Define the feedback request separately from provider input. JSON tags describe
the browser's representation, not verification policy:

```go
type FeedbackInput struct {
	Message      string `json:"message"`
	CAPTCHAToken string `json:"captcha_token"`
}
```

With `request` imported from `arkoren.dev/core/http/request` and `incoming` the
handler's `*http.Request`, decode once:

```go
input, err := request.JSON[FeedbackInput](incoming)
if err != nil {
	return err
}
```

Validate the message using your application's input rules before verification.
Keep `CAPTCHAToken` out of request-body logging, traces, and validation diagnostics.

### Verify Turnstile

Using the Turnstile `verifier` from setup, require the server-owned feedback
action and stop on every verification error:

```go
_, err = verifier.Verify(incoming.Context(), cloudflarecaptcha.Input{
	Token:          input.CAPTCHAToken,
	ExpectedAction: "feedback",
})
if err != nil {
	return err
}
// Accept the validated feedback here.
```

Only nil error permits the feedback operation to proceed. The hostname check
always applies; the action check applies because you supplied a nonempty action.

### Evaluate Scores

For Google, choose a finite `feedbackMinScore` in `[0, 1]` from your site's
observations and validate it when loading server configuration. There is no
universal safe threshold. Higher scores indicate lower assessed risk. Replace
the Turnstile block with:

```go
assessment, err := verifier.Verify(incoming.Context(), gcpcaptcha.Input{
	Token:          input.CAPTCHAToken,
	ExpectedAction: "feedback",
})
if err != nil {
	return err
}
if assessment.Score < feedbackMinScore {
	return problem.Forbidden()
}
// Accept the validated feedback here.
```

The operation runs only after token verification and score policy succeed.
Explicit zero accepts every valid score, but still requires valid token evidence,
an allowed hostname, the requested action, and a present risk-analysis message.
An omitted or null score inside a present message means zero under ProtoJSON;
a positive threshold denies it as policy, not a verification outage. A missing
risk-analysis message instead fails verification.

For login, direct verification can support a step-up decision instead of a flat
denial. After checking credentials and verifying once with `ExpectedAction: "login"`,
a low score can start an application-owned pending factor challenge.
That branch must not create an authenticated session. Verify the extra factor
on a separate endpoint before signing in. A high score never replaces factors
your authentication policy always requires.

Optional `Input.RemoteIP` is a `netip.Addr`; supply it only from your trusted
client-address resolver, never directly from arbitrary forwarded headers. Google
also accepts optional `Input.UserAgent`. Sending these fields discloses visitor
data to the provider; omit them unless your privacy policy permits and needs
that disclosure. The examples omit both.

## Header Middleware

If the token travels separately in a header, use middleware instead of handler
verification. It extracts and verifies once, then calls the handler only on
success. Neither adapter automatically parses JSON or forms, stores evidence in
request context, or infers client IP; Google middleware also infers no user agent.
Use direct verification when you need an assessment for a step-up decision.

Keep body limits, content-type checks, and rate budgets outside CAPTCHA middleware.
For cross-origin browser requests, [CORS](https://arkoren.dev/docs/guides/browser-security/cors.md) must permit
the token header. With `net/http` imported, define a body-preserving extractor:

```go
extract := func(incoming *http.Request) (string, error) {
	return incoming.Header.Get("X-CAPTCHA-Token"), nil
}
```

For an existing Turnstile verifier and application handler `feedback`, mount:

```go
routes.With(cloudflarecaptcha.Middleware(verifier, extract,
	func(policy *cloudflarecaptcha.MiddlewareBuilder) {
		policy.Action("feedback")
	})).Post("/feedback", feedback)
```

For Google, use this alternative and explicitly configure the minimum score:

```go
routes.With(gcpcaptcha.Middleware(verifier, extract,
	func(policy *gcpcaptcha.MiddlewareBuilder) {
		policy.Action("feedback")
		policy.MinScore(feedbackMinScore)
	})).Post("/feedback", feedback)
```

The handler must not verify that token again. Google requires a finite minimum
in `[0, 1]`; omission is a configuration error, not a default threshold.

Both middleware adapters return safe HTTP problems: rejection is `403`, including
a Google score below the minimum; provider unavailability is `503`; configuration
failure is `500`. Ordinary extractor errors reject with `403`. Extractor errors
wrapping cancellation, deadline, or unavailability return `503` without adding a
token-rejection identity. Configuration takes precedence over other classes.
Original causes remain inspectable, but their text is not rendered to the client.

Turnstile panics during wiring for a nil verifier, extractor, next handler, or
construction callback. A zero verifier instead fails requests with configuration
errors. Google reports a nil verifier/extractor or missing/invalid score policy
as `500` on requests without extracting or verifying. Nil construction callbacks
panic in both adapters. These are configuration faults, not permission to bypass
the check.

## Rejection Tests

Inject a controlled `*http.Client` through each provider's `Builder.HTTPClient`
for deterministic responses. Google tests with a supplied client bypass ADC;
they do not verify live credentials or IAM permissions. Neither adapter has a
memory driver or local emulator. A stub proves application decisions, not a
genuine browser challenge.

Test accepted and missing tokens, rejection, hostname/action mismatches,
unavailability, and cancellation. For Google, include the exact score boundary,
a valid score below it, ProtoJSON's zero score, and missing middleware `MinScore`.
Assert that every failure leaves the feedback effect untouched. If login uses a
step-up branch, assert that it creates only a pending challenge, never a session.
Keep real-browser and live-provider checks separate from deterministic tests.

For complete contracts, see the
[Cloudflare CAPTCHA API reference](https://pkg.go.dev/arkoren.dev/driver/cloudflare/cloudflarecaptcha)
and [Google CAPTCHA API reference](https://pkg.go.dev/arkoren.dev/driver/gcp/gcpcaptcha).

---

Source: https://arkoren.dev/docs/guides/authentication/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/authentication/overview)

# Authentication Overview

Authentication establishes who sent a request. Arkoren resolves a session or
bearer credential into a Go value that your application owns; it supplies no
user table, registration flow, or login screen. [Authorization](https://arkoren.dev/docs/guides/authorization/overview.md)
then decides what that identity may do.

Use [Session Login](https://arkoren.dev/docs/guides/authentication/session-login.md) for passwords and browser sessions,
[API Tokens](https://arkoren.dev/docs/guides/authentication/api-tokens.md) for application-issued keys, [OAuth Login](https://arkoren.dev/docs/guides/authentication/oauth.md)
for external browser sign-in, or [OIDC Tokens](https://arkoren.dev/docs/guides/authentication/oidc-bearer.md) for provider-issued
JWTs. All four use the same request identity model.

Focused snippets keep error results visible but omit routine propagation.
Checks remain where failure changes authentication or access. Handle all errors
in your application.

## Define Identity

Choose one identity type for handlers and policies. An invoice application might
use this local account value:

```go
type User struct {
	ID       int
	Name     string
	Disabled bool
}
```

No field or interface is required by Arkoren. Your verifier or loader must
reject unknown and disabled accounts before returning success. Returning
`User{}, nil` still authenticates a request; `auth.Require` checks presence,
not whether fields are nonzero. The same applies to a typed nil pointer returned
without an error.

Identity lookup uses the exact Go type. `User`, `*User`, and a separately defined
type with identical fields are different identities; Arkoren does not convert
between them.

## Resolve Credentials

An **authenticator** is a `func(*http.Request) (User, error)` that verifies
credentials and returns your identity. Import `auth` from
`arkoren.dev/core/http/auth`. Its result controls whether another authenticator
can try:

| Result | Middleware behavior |
| --- | --- |
| User and nil error | Store that user and stop trying authenticators. |
| `auth.ErrNoCredentials`, including wrapped errors | Try the next authenticator. |
| Any other error | Stop the request without calling its handler. |

Return `ErrNoCredentials` only when the request has no credentials that this
authenticator handles. A presented invalid bearer token must return a different
error. Do not trust a client-supplied user ID or header as proof of identity.

For routes accepting both browser sessions and API keys, use your existing
`loadUser` (`func(context.Context, int) (User, error)`) and `verifyToken`
(`func(context.Context, string) (User, error)`):

```go
identity := auth.NewSessionIdentity(loadUser)
r.Use(auth.Middleware(
	identity.Authenticator(),
	auth.Bearer(verifyToken),
))
```

Here `r` is your existing router. This browser group already needs
[session and CSRF middleware](https://arkoren.dev/docs/guides/authentication/session-login.md#enable-login) before authentication.
The first success wins: a valid session in this order prevents inspection of a
simultaneously supplied bearer header. An error stops the chain; it does not
trigger fallback. Prefer separate browser and API groups when they need different
credential policies.

## Choose Route Access

Authentication middleware permits anonymous requests when every authenticator
returns `ErrNoCredentials`. This lets public pages optionally personalize their
response. To require a user for private routes, add `auth.Require` after the
authentication middleware:

```go
private := r.With(auth.Require[User]())
private.Get("/invoices", listInvoices)
```

`listInvoices` is your existing handler. An anonymous request receives a `401`
Problem Details response with `WWW-Authenticate: Bearer`; an authenticated
request reaches the handler. This proves identity presence, not permission to
read every invoice. Apply your [authorization rules](https://arkoren.dev/docs/guides/authorization/overview.md)
separately.

For browser pages, a `401` does not display a login form. If you want a login
redirect, write middleware that checks `auth.FromContext[User]` and redirects to
a fixed local login path. Validate any return target rather than redirecting to
an arbitrary request parameter. See [middleware scope](https://arkoren.dev/docs/guides/http-fundamentals/middleware.md)
for applying different behavior to different route groups.

## Read Identity

Handlers retrieve the exact type stored by authentication. With `request` and
`response` from their `arkoren.dev/core/http/` packages and standard-library
`net/http`, read the local user before returning a profile:

```go
user, err := request.User[User](r)
if err != nil {
	return err
}

return response.JSON(w, http.StatusOK, user)
```

`w` and `r` are the handler's response writer and request. Missing identities
return `auth.ErrUnauthenticated`, including when a different identity type is
present. In context-only code, `auth.FromContext[User](ctx)` returns the value
and a presence boolean instead.

Tests can use `auth.NewContext(ctx, User{ID: 42})` without performing a login.
That helper trusts the supplied value; keep it in trusted code, not in a path
that copies unverified request input. [Auth Tests](https://arkoren.dev/docs/guides/testing/auth-tests.md)
also covers testing the actual authenticators.

## Handle Rejections

To change the anonymous response, configure `auth.Builder.ErrorResponse` in
`auth.Require[User](...)`. It replaces the whole problem and its headers, so
include the appropriate authentication challenge if you keep status `401`.
A zero response restores the default. This setting does not change the error
returned by `request.User` or failures returned by an authenticator.

If you need anonymous-rejection events, supply your existing event bus:

```go
private := r.With(auth.Require[User](func(required *auth.Builder) {
	required.Events(app.Events())
}))
```

Every rejection publishes `http.auth.rejected`, even with a custom status.
Its payload contains `Method`, not a request path, identity, query, or cause.
HTTP methods and event context values can still be untrusted or sensitive;
normal Problem Details rendering can include the request path as `instance`.
Keep listeners cheap and apply [rate limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md)
before the required-user middleware when needed. No event bus means no event.

For all functions and options, see the
[`auth` API reference](https://pkg.go.dev/arkoren.dev/core/http/auth).

---

Source: https://arkoren.dev/docs/guides/authentication/session-login.md
[HTML documentation](https://arkoren.dev/docs/guides/authentication/session-login)

# Session Login

A browser login checks credentials once, then stores an account identifier in
the session. On later requests, `auth.SessionIdentity` loads the current account
from that identifier. You own the accounts, password checks, and account-status
policy; [Sessions](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#enable-sessions) owns cookie
configuration, storage, and response-time persistence.

The examples assume an existing router `web`, session store `store`, configured
[`hash.Hasher`](https://arkoren.dev/docs/guides/hashing/overview.md), and application repository `accounts`.
Import `auth`, `session`, and `response` from their `arkoren.dev/core/http/`
packages, `csrf` from `arkoren.dev/core/http/middleware/csrf`, and standard-library
`context`, `errors`, and `net/http` where used.

## Load Accounts

Keep the session small: store an account ID, not a snapshot of permissions.
The loader can then reject a deleted or disabled account on the next request.
This application separates the handler identity from its password record:

```go
type User struct {
	ID   int
	Name string
}

type Account struct {
	User
	PasswordHash   []byte
	Disabled       bool
	SessionVersion int
}
```

Here `accounts.ByID(ctx, id)` returns `Account`; the application sentinel
`ErrAccountNotFound` means absence, while other errors retain storage failures.
Bind the loader to the identifier type `int`:

```go
loadUser := func(ctx context.Context, id int) (User, error) {
	account, err := accounts.ByID(ctx, id)
	if errors.Is(err, ErrAccountNotFound) {
		return User{}, auth.ErrNoCredentials
	}
	if err != nil {
		return User{}, err
	}
	if account.Disabled {
		return User{}, auth.ErrNoCredentials
	}

	return account.User, nil
}
```

The loader runs on every request with a usable stored identifier. Absence and
disabled status become anonymous requests; a storage outage instead stops the
request. Returning a zero user with nil error would authenticate it, so do not
use that result for absence. If you cache accounts, your invalidation policy
determines how quickly account changes take effect.

## Enable Login

Construct one identity definition and share it between middleware and handlers:

```go
identity := auth.NewSessionIdentity(loadUser)
web.Use(session.Middleware(store))
web.Use(csrf.Middleware())
web.Use(auth.Middleware(identity.Authenticator()))
```

Session middleware must run first. Keep [CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md)
on login, logout, and other state-changing browser routes: browsers send session
cookies automatically, including on requests another site can initiate. Add
`auth.Require[User]()` only to routes requiring a signed-in account, not to the
login route itself.

Construction does not call the loader. Check `identity.Err()` during setup
when configuration is assembled dynamically; a zero definition or nil loader
returns `auth.ErrInvalidSessionIdentity`, not an anonymous result. Missing
sessions, missing identity fields, and incompatible stored types are anonymous.
Persisted JSON that remains compatible can still decode after an identifier
type change; a type change is not a revocation mechanism.

## Check Passwords

Start with [validated request input](https://arkoren.dev/docs/guides/http-fundamentals/requests.md), not an
arbitrary email or account ID from a header. A login request has only the fields
needed to check credentials:

```go
type LoginInput struct {
	Email    string `json:"email"`
	Password string `json:"password"`
}
```

Use the same rejection for an unknown email, a wrong password, and a disabled
account. Otherwise login becomes an account-enumeration endpoint. The following
fragment runs in your login handler with validated `in LoginInput` and a
configured `hasher *hash.Hasher`. `accounts.ByEmail` returns the same application
absence sentinel as `ByID`:

```go
account, err := accounts.ByEmail(r.Context(), in.Email)
if err != nil && !errors.Is(err, ErrAccountNotFound) {
	return err
}

found := err == nil
storedHash := dummyHash
if found {
	storedHash = account.PasswordHash
}
```

`dummyHash` is a hash of a non-secret placeholder, computed once at startup
with the same hasher and cost as real passwords. Check it on the missing-account
path so that path does not skip the deliberately expensive password check:

```go
matches, err := hasher.Check(in.Password, storedHash)
if err != nil {
	return err
}
if !found || !matches || account.Disabled {
	return auth.ErrUnauthenticated
}
```

The explicit `!found` check prevents a password matching the dummy hash from
creating a login. This reduces a timing difference; it does not guarantee equal
response times. Preserve storage and hash-check errors rather than classifying
them as bad passwords, and keep passwords and hashes out of error messages.
Apply [login rate limits](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) before credential
checks to limit guessing and hashing work.

Create password hashes at registration using the [hashing guide](https://arkoren.dev/docs/guides/hashing/overview.md).
Keep the dummy hash aligned with [cost changes](https://arkoren.dev/docs/guides/hashing/cost.md). For old
hashes, follow [hash upgrades](https://arkoren.dev/docs/guides/hashing/upgrading.md), including its conditional
write that avoids overwriting a concurrent password reset.

## Start Sessions

After the credential checks succeed, resolve the identity's selected session and
store the verified account ID. Do both before writing response headers:

```go
current, err := identity.Session(r.Context())
if err != nil {
	return err
}
if err := identity.Login(current, account.ID); err != nil {
	return err
}

return response.Status(w, http.StatusNoContent)
```

`Login` writes the identifier and then regenerates the session ID. Rotation
protects against **session fixation**, where an attacker knows an ID planted
before the victim signs in. If the field write fails, no rotation occurs and
you must not report success. Login preserves other session values, creation
time, and expiry; configure idle and absolute limits in
[session lifetimes](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#set-lifetimes).

This call changes the request's session, not persistent storage. On a non-5xx
response, session middleware saves changes and retires the old ID before
committing the response. Failed persistence or retirement can leave an old ID
usable; configure and monitor the middleware's error handling rather than
treating `Login` success as proof of durable revocation. See
[Save Changes](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#save-changes) for the commit and
concurrency boundaries.

## End Sessions

Mount logout as a CSRF-protected `POST`, not a `GET`. Resolve the same identity's
session and remove its login before sending a response:

```go
current, err := identity.Session(r.Context())
if err != nil {
	return err
}
if err := identity.Logout(current); err != nil {
	return err
}

return response.Status(w, http.StatusNoContent)
```

Logout removes even a stale, wrongly typed identifier and rotates the ID after
successful deletion. Other values, such as a language preference, remain.
Use `current.Destroy()` instead when the whole session must end. Both paths
still depend on response-time persistence; a failed retirement can leave the
old ID valid.

## Revoke Account Sessions

To end sessions across devices after a password change or compromise, store an
account version with the ID. This is an alternative identity definition, not
an additional identity entry:

```go
type SessionUser struct {
	ID      int
	Version int
}
```

Change your loader to `func(context.Context, SessionUser) (User, error)`.
After loading the account and handling absence or storage errors as above,
reject a stale version:

```go
if account.Disabled || account.SessionVersion != saved.Version {
	return User{}, auth.ErrNoCredentials
}
return account.User, nil
```

Here `saved` is the loader's `SessionUser` argument. Reconstruct the definition
with that loader and supply both values after successful credential checks:

```go
if err := identity.Login(current, SessionUser{
	ID:      account.ID,
	Version: account.SessionVersion,
}); err != nil {
	return err
}
```

Increment the persisted `SessionVersion` to invalidate older versions on each
device's next account lookup. No session enumeration is required. Existing
in-flight requests are not retroactively canceled, and account caching can
delay the effect.

## Select Sessions

If you configured a nondefault session context slot, share one handle between
the middleware and the identity:

```go
handle := session.NewHandle()
identity := auth.NewSessionIdentity(loadUser, func(binding *auth.SessionBuilder) {
	binding.Handle(handle)
})
web.Use(session.Middleware(store, func(cookies *session.Builder) {
	cookies.Handle(handle)
}))
```

Register CSRF and authentication after this session middleware as before.
`identity.Session(ctx)` selects only this slot and returns `session.ErrNoSession`
for an absent or nil session; it never falls back to the default slot. A zero
handle selects the default. Explicit `Login` and `Logout` arguments are not
checked against the handle, so use `identity.Session` to select them consistently.

A handle is not cookie or storage isolation. Definitions writing to the same
physical session share `arkoren:auth:user`: the last login replaces the identity,
and logout removes it. Independent browser sessions also need distinct cookie
and storage configuration in [Sessions](https://arkoren.dev/docs/guides/http-capabilities/sessions.md).
For authentication tests, combine [direct identity tests](https://arkoren.dev/docs/guides/testing/auth-tests.md)
with a real cookie-based login/logout test that exercises persistence.

For all functions and options, see the
[`auth` API reference](https://pkg.go.dev/arkoren.dev/core/http/auth).

---

Source: https://arkoren.dev/docs/guides/authentication/api-tokens.md
[HTML documentation](https://arkoren.dev/docs/guides/authentication/api-tokens)

# API Tokens

A program authenticates by sending a secret token with each request, rather
than signing in through a browser form. `auth.Bearer` extracts that token and
delegates verification to your application. It does not issue tokens, maintain
a token store, or implement a JWT verification policy.

If you already have a verifier, connect it directly. If your service issues its
own credentials, a random **API key** backed by a stored hash gives you a
per-key revocation point. For identity-provider JWTs, use
[OIDC Tokens](https://arkoren.dev/docs/guides/authentication/oidc-bearer.md) instead of building another verifier.

## Accept Bearer Credentials

Serve credential-bearing requests over HTTPS. Send the token in the header,
never in a URL, and keep it out of logs:

```http
GET /api/invoices HTTP/1.1
Authorization: Bearer ak_example_token
```

The value above illustrates the header, not a usable key. With your existing
router group `api` and `verifyKey func(context.Context, string) (User, error)`,
import `auth` from `arkoren.dev/core/http/auth` and register:

```go
api.Use(auth.Middleware(auth.Bearer(verifyKey)))
api.Use(auth.Require[User]())
```

`auth.Middleware` alone allows missing credentials; `Require` closes that path
for private routes. Your verifier must return the same exact `User` type used
by handlers and policies. A zero-valued user returned without an error still
counts as authenticated, so reject unknown or disabled accounts explicitly.

The bearer authenticator accepts a single `Authorization` header with a
case-insensitive `Bearer` scheme and a nonempty token containing no spaces.
No header returns `auth.ErrNoCredentials`. A malformed header is rejected before
verification; a verifier error normally produces `401` with
`WWW-Authenticate: Bearer error="invalid_token"`.

> [!WARNING]
> Never return `auth.ErrNoCredentials` for a presented invalid, expired, or revoked
> token. That error permits fallback or anonymous handling. Return a different
> error, such as `auth.ErrUnauthenticated`.

HTTP responses do not expose verifier error messages, but retain the error as
a cause. Database failures are also rendered as invalid-token responses through
this adapter; inspect controlled diagnostics to distinguish them from bad keys.
Do not put raw tokens or sensitive account details in those causes.

## Issue Keys

An API key is a bearer secret: anyone holding it can use its authority. Issue
it only after authenticating the owner and authorizing key creation. The owner
ID must come from that trusted identity, not from a submitted request field.
The application types below separate user input from the stored record:

```go
type User struct {
	ID int
}

type IssueKeyInput struct {
	Name string `json:"name"`
}

type APIKey struct {
	UserID  int
	Hash    []byte
	Name    string
	Revoked bool
}
```

With validated `in IssueKeyInput`, authenticated `owner User`, an operation
context `ctx`, and your application repository `keys`, generate a random key.
Import standard-library `crypto/rand` and `crypto/sha256`:

```go
token := "ak_" + rand.Text()
digest := sha256.Sum256([]byte(token))
err := keys.Save(ctx, APIKey{
	UserID: owner.ID,
	Hash:   digest[:],
	Name:   in.Name,
})
```

`rand.Text` provides at least 128 bits of randomness; the prefix only identifies
your key format. Store the hash, not the original token. Return the token once
only after `Save` succeeds, using an authenticated response that is not cached
or logged. It cannot be recovered from the stored record; lost keys need replacement.

A fast SHA-256 hash is suitable for a high-entropy random secret. It is not
appropriate for human-chosen passwords, which need
[password hashing](https://arkoren.dev/docs/guides/hashing/overview.md). Hashed storage reduces exposure if
the key table leaks; it does not protect keys already stolen from clients or
exposed during issuance.

## Verify Stored Keys

Keep a unique lookup index on the stored hash. Your `keys.ByHash` application
method returns an `APIKey` or an error; absence must not be reported as
`auth.ErrNoCredentials`. Hash the entire presented token before looking it up:

```go
verifyKey := func(ctx context.Context, token string) (User, error) {
	digest := sha256.Sum256([]byte(token))
	stored, err := keys.ByHash(ctx, digest[:])
	if err != nil {
		return User{}, err
	}
	if stored.Revoked {
		return User{}, auth.ErrUnauthenticated
	}

	return loadActiveUser(ctx, stored.UserID)
}
```

`loadActiveUser` is your `func(context.Context, int) (User, error)` account lookup.
It returns an error for missing or disabled accounts, never a zero user with
nil error or `ErrNoCredentials`. If keys have expiry or granted permissions,
check expiry here and carry the permitted authority in a request-local identity;
[authorization](https://arkoren.dev/docs/guides/authorization/overview.md) still decides which actions and
resources it allows.

`auth.Bearer` checks the header shape, not your key format. Very short tokens can
reach the verifier. If your format extracts a public lookup identifier or prefix,
validate lengths before slicing. When comparing raw secrets rather than looking
up a full digest, use `crypto/subtle.ConstantTimeCompare`, not ordinary equality.

## Revoke Keys

Delete a key record or mark it revoked, and ensure every verification checks
that state. The next uncached lookup rejects the key. If you cache verification
results, invalidate those entries too; otherwise cache lifetime delays revocation.
Already authenticated requests are not canceled by changing the record.

Let users name keys and revoke individual credentials without changing every
client. For replacement, issue a new key, move the client to it, then revoke the
old one. Scope key-management operations to the authenticated owner and apply
authorization separately; possession of an arbitrary key does not automatically
grant permission to manage other credentials.

## Choose Token Semantics

A signed JWT can carry identity claims without a per-request database lookup.
That saves shared-state access but changes the revocation boundary: a purely
offline verifier cannot observe an account deletion, permission change, or key
revocation until its verification policy makes the token unacceptable, commonly
at expiry. Short lifetimes reduce this window; a server-side denylist or account
lookup adds state back to verification.

If you choose JWTs, use a maintained verifier with an explicit trusted issuer,
intended audience, allowed algorithms and keys, expiry policy, and provider-specific
token-purpose checks. Decoding claims or checking a signature alone is not
authentication. Arkoren's core bearer adapter does not implement these checks or
provide a token endpoint, refresh flow, or authorization server.

## Separate Browser Routes

Prefer bearer-only API groups when scripts should not inherit browser session
authority. Such routes do not need CSRF protection for bearer authentication
because the credential is explicitly supplied, not automatically sent as a cookie.
Do not apply that exception to routes that also accept sessions.

If the same routes deliberately accept both credentials, configure the
[session and CSRF middleware](https://arkoren.dev/docs/guides/authentication/session-login.md#enable-login) and choose
[authenticator order](https://arkoren.dev/docs/guides/authentication/overview.md#resolve-credentials). Both authenticators must
return the same identity type; the first successful one wins even if the request
also carries another credential. For MCP, reuse the verifier and follow
[endpoint security](https://arkoren.dev/docs/guides/mcp/securing-mcp.md) for protocol and tool-access rules.

For all functions and options, see the
[`auth` API reference](https://pkg.go.dev/arkoren.dev/core/http/auth).

---

Source: https://arkoren.dev/docs/guides/authentication/oauth.md
[HTML documentation](https://arkoren.dev/docs/guides/authentication/oauth)

# OAuth Login

Let a provider check a person's credentials, then create a local browser login
for the matching account. OpenID Connect (OIDC) adds verified ID tokens to the
OAuth 2 authorization flow; plain OAuth2 providers instead supply identity
through an authenticated user API. `oauthauth` supports both paths.

You still own local accounts and external-account linkage. The adapter completes
the provider flow, calls your mapper, and stores its returned account identifier
with [Session Login](https://arkoren.dev/docs/guides/authentication/session-login.md). It is not an authorization server and
does not acquire API tokens for arbitrary clients or refresh tokens for your
service. To accept an existing JWT at an API, use [OIDC Tokens](https://arkoren.dev/docs/guides/authentication/oidc-bearer.md).

## Choose Providers

Use a preset when it matches your provider; it includes identity checks that a
generic adapter does not infer. The matrix describes shipped adapter behavior,
not every feature offered by the provider. PKCE is the authorization-code
challenge sent by default, not a claim about provider-side enforcement.

### Compatibility Matrix

| Provider | Browser login | OIDC discovery | PKCE default | Verified-email filter | JWT verification |
| --- | --- | --- | --- | --- | --- |
| [Google](#google) | Yes | Yes | Yes | Yes | Yes — Requires explicit API audiences. |
| [Microsoft](#microsoft) | Yes — Enforces the selected tenant policy. | Yes | Yes | Yes — Requires boolean xms_edov. | Yes — Enforces the selected tenant policy. |
| [GitHub](#github) | Yes | No | Yes | Yes — Selects the primary verified address. | No |
| [Apple](#apple) | Yes — Requires a POST callback and key.Signing. | Yes | No | No | Yes — Requires explicit API audiences. |
| [Generic OIDC](#generic-oidc) | Yes | Yes | Yes | No | Yes — Requires explicit API audiences. |
| [Generic OAuth2](#generic-oauth2) | Yes | No | Yes | Partial — The application owns user-info validation. | No |

All clients require an encrypter. JWT verification requires OIDC, explicit API
audiences, and your provider's token-purpose policy; it is not interchangeable
with browser ID-token verification. A filtered email is still not an account key.

## Configure Clients

Register an application and exact callback URL with your provider. Obtain its
client ID and required credentials, then install the adapter in your module:

```sh
go get arkoren.dev/driver/oauth
```

Import it as `oauth`. The example uses an existing `cipher *crypto.Encrypter`
from [Encryption](https://arkoren.dev/docs/guides/encryption/overview.md), plus `clientID` and `clientSecret`
loaded from your configuration/secret provider:

```go
import oauth "arkoren.dev/driver/oauth/oauthauth"

client, err := oauth.New(oauth.Google(), cipher, clientID,
	func(login *oauth.ClientBuilder) {
		login.ClientSecret(clientSecret)
		login.RedirectURL("https://app.example.com/oauth/callback")
	},
)
```

Handle the constructor error before using `client`. Construction validates
inputs but performs no network I/O. Use a stable encryption key shared by all
instances that can receive callbacks; key provisioning belongs in
[Encryption Keys](https://arkoren.dev/docs/guides/encryption/keys.md), not in each request handler.

For OIDC, the positional `clientID` is the required browser ID-token audience,
its intended recipient. `ClientBuilder.Audiences` configures only
[API verification](https://arkoren.dev/docs/guides/authentication/oidc-bearer.md#prepare-verification), not this callback check.
PKCE starts enabled; disable it only for a provider that cannot support it.
Apple forces it off independently of this setting.

Use a distinct `ClientBuilder.Name` for each provider client. It prefixes state
cookies and defaults to `arkoren.oauth`; invalid cookie names fail construction.
Configuration callbacks run once in order, and nil callback entries panic.
The published client copies its settings and scope/audience slices; retained
drafts cannot change it, but services and captured application state remain shared.

## Start Discovery

OIDC clients must discover their endpoints before serving requests. In your
existing [application composer](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md),
register startup with `application` from `arkoren.dev/core/application`:

```go
compose := func(composer *application.Composer) {
	composer.Startup(client.Start)
}
```

Before successful startup, OIDC handlers return `ErrNotStarted` (`503`). A
subsequent `Start` refreshes discovered data. Plain OAuth2 clients prepare their
configured endpoints at construction; their `Start` is a no-op. Signing keys
can still be retrieved during OIDC token verification.

Keep the returned `*Client`; do not copy its atomic state. It has no `Stop`
method and does not stop the shared encrypter or event bus. Your application
owns those services and any supplied native HTTP client; keep encryption
available until handlers have finished.

## Connect Browser Routes

Use an existing session-enabled, CSRF-protected browser group `web`, as in
[Enable Login](https://arkoren.dev/docs/guides/authentication/session-login.md#enable-login). Import `auth` from
`arkoren.dev/core/http/auth`. Your `loadUser` loads a current local `User` by
integer ID; `mapIdentity` below returns that same ID type:

```go
identity := auth.NewSessionIdentity(loadUser)
web.Use(auth.Middleware(identity.Authenticator()))
web.Get("/login", client.Login())
web.Get("/oauth/callback", client.Callback(identity, mapIdentity))
web.Post("/logout", client.Logout(identity))
```

`Login` redirects to the provider. A successful callback verifies the provider
identity, maps it, rotates the selected local session ID, and redirects to the
saved local destination. Subsequent requests use the normal session loader.
`Logout` removes the local identity and redirects to `/`; it does not sign the
person out at the provider. Keep logout CSRF-protected.

Check `identity.Err()` during setup. An invalid definition or nil mapper is a
configuration error, not a rejected credential. Callback and logout select
exactly the identity's session slot. An absent session returns `401` before state
consumption or provider exchange, without default-slot fallback. If you use
[custom session selection](https://arkoren.dev/docs/guides/authentication/session-login.md#select-sessions), share its handle
with the session middleware. Middleware still owns
[response-time persistence](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#save-changes);
successful callback mutation does not prove storage or old-ID retirement succeeded.

## Link Local Accounts

Identify an external account by the pair `Issuer` and `Subject`, not by email
or display name. Require an existing link for ordinary sign-in; create accounts
only under your explicit registration policy. To link another provider to an
existing account, require proof of control of that local account as a separate,
authorized action.

With standard-library `context`, this application mapper looks up an existing
link using `accounts.ByIdentity(ctx, issuer, subject)`, which returns the
application's `Account` or an error:

```go
mapIdentity := func(ctx context.Context, external oauth.Identity) (int, error) {
	if external.Issuer == "" || external.Subject == "" {
		return 0, auth.ErrUnauthenticated
	}
	account, err := accounts.ByIdentity(ctx, external.Issuer, external.Subject)
	if err != nil {
		return 0, err
	}
	if account.Disabled {
		return 0, auth.ErrUnauthenticated
	}

	return account.ID, nil
}
```

The repository owns uniqueness of the issuer/subject link and rejection of
unknown accounts. A mapper returning zero and nil error still stores zero as
the login identifier; Arkoren cannot infer whether it denotes a real account.
Mapper errors matching `auth.ErrNoCredentials` become `ErrInvalidCallback`
(`400`); other application errors pass through unchanged.

> [!WARNING]
> Never match or merge accounts by email alone. Some providers pass through
> unverified email claims, and even verified addresses can change. An automatic
> email match can give an attacker another person's existing local account.

`Identity.Email` is optional profile data. Google filters using boolean
`email_verified`, Microsoft using boolean `xms_edov`, and GitHub using its primary
verified email record. Generic OIDC and Apple pass email through without that
filter; generic OAuth2 uses whatever your user-info function returns. If an
action needs a verified address, enforce that policy separately. Treat `Name`
as untrusted display text and escape it when rendering.

For browser callbacks, `Identity.Scopes` comes from the token response. If the
response omits scopes, the client supplies an independent copy of the requested
scopes. These are provider grants, not your local account's resource permissions.

## Protect Login State

To return a person to an invoice after login, start the flow with a local path:

```text
/login?return_to=/invoices/42
```

The client validates this target before redirecting to the provider, seals it
with the login state, and uses a safe redirect after the callback. Unsafe targets
return `ErrInvalidLoginRequest` (`400`). Do not bypass this path by redirecting
to another unvalidated request parameter.

Each login gets random state, an OIDC nonce when applicable, and a PKCE verifier
when enabled. State binds the callback to the browser login; nonce binds the
ID token to that attempt. These values live in an encrypted, HttpOnly cookie
for ten minutes, not in the application session. The client keeps room for at
most five live login cookies, allowing parallel tabs.

Callbacks reject missing, tampered, expired, or mismatched state cookies. For
OIDC they also check the ID token's signature, expiry, issuer, positional
client-ID audience, nonempty final subject, and saved nonce before mapping.
Plain OAuth2 relies on the authenticated user API instead of an ID token.
Cookie deletion is browser cleanup, not a server-side replay registry: a copied
cookie can survive until expiry. The provider must enforce one-time authorization
codes.

Cookies use `Secure` and the `__Host-` prefix by default. Use HTTPS for login
and registered callbacks. `ClientBuilder.AllowInsecure(true)` removes both
protections for plain-HTTP development only; never enable it in production.
Apple's cross-site cookie requires HTTPS even during development.

## Google

Choose Google for Google account sign-in. Register a web client and the callback
URL in Google, then use `oauth.Google()` in the shared constructor with its
client ID and secret. Keep the startup hook for discovery and the ordinary GET
callback.

If only one Workspace domain may sign in, replace the provider argument with:

```go
provider := oauth.Google(func(google *oauth.GoogleBuilder) {
	google.HostedDomain("example.com")
})
```

Pass `provider` to `oauth.New`. This checks the verified `hd` claim as well as
sending the account-chooser hint; the default accepts any domain. Default scopes
are `openid`, `email`, and `profile`. The preset normalizes the issuer to
`https://accounts.google.com` and leaves email empty without boolean
`email_verified: true`. Keep issuer/subject linkage even with this filter.

See the [`Google` API](https://pkg.go.dev/arkoren.dev/driver/oauth/oauthauth#Google).

## Microsoft

Choose Microsoft for Entra ID or Microsoft-account sign-in. Register the callback
URL and obtain the application's client ID and secret. Replace the shared
provider argument with a deliberately chosen tenant policy:

```go
provider := oauth.Microsoft("organizations")
```

Pass it to `oauth.New`, keep `Start` for discovery, and use the shared browser
routes and mapper. `organizations` rejects the reserved personal-account tenant;
`consumers` accepts only that tenant (`9188040d-6c67-4c5b-b112-36a304b66dad`).
`common` accepts both classes. For single-tenant use, pass the tenant ID, matched
case-insensitively, not a domain name such as `contoso.onmicrosoft.com`.

Both callback and bearer checks enforce the verified `tid` policy and require
`oid`. The issuer must be `https://login.microsoftonline.com/<tid>/v2.0`, and
the final subject is `oid`. The check uses tenant context, not the home `idp`:
a personal account invited into an organization can pass `organizations` in
that organization's context. Add a narrower tenant allowlist in your mapper
if the selected preset is broader than your policy.

Default scopes are `openid`, `profile`, and `email`. Email remains empty unless
the optional `xms_edov` claim is boolean true; configure that claim in the app
registration if needed. It is still not a local-account key.

See the [`Microsoft` API](https://pkg.go.dev/arkoren.dev/driver/oauth/oauthauth#Microsoft).

## GitHub

Choose GitHub for plain OAuth2 sign-in. Register a GitHub OAuth application and
the callback URL, then pass its ID/secret to the shared constructor with:

```go
provider := oauth.GitHub()
```

Use the shared GET callback and local mapper. `Start` is a no-op because this
adapter has configured endpoints, not discovery. Default scopes are `read:user`
and `user:email`; PKCE remains enabled.

After code exchange, the preset calls the user and email APIs. Its issuer is
`https://github.com`, and subject is the positive numeric user ID, not the mutable
login name. Missing, zero, or negative IDs and user/email API failures return
`ErrProviderUnavailable` (`502`) before mapping. Email is the primary verified
address or empty. This preset cannot verify OIDC bearer tokens.

See the [`GitHub` API](https://pkg.go.dev/arkoren.dev/driver/oauth/oauthauth#GitHub).

## Apple

Choose Apple for Sign in with Apple. Register an HTTPS callback and a Services
ID, obtain your team ID/key ID, and load the Apple-issued `.p8` signing key with
`key.LoadSigning` from `arkoren.dev/foundation/key`. Do not generate an unrelated
local replacement for a key registered with Apple.

```go
appleSigning, err := key.LoadSigning("AuthKey_ABC123DEFG.p8")
```

Handle that error, then construct the provider:

```go
provider := oauth.Apple(teamID, keyID, appleSigning)
```

Use the Services ID as `oauth.New`'s client ID, omit `ClientSecret`, and configure
`RedirectURL("https://app.example.com/oauth/apple/callback")`. The constructor
rejects an invalid key. The provider signs a short-lived client secret per code
exchange, requires discovery, requests `openid`, `name`, and `email`, and forces
PKCE off.

Apple sends a cross-site POST callback. Mount only that callback outside CSRF
protection, while retaining session middleware; keep CSRF on logout and all
other state-changing browser routes. With `r` your session-enabled router,
the callback is:

```go
r.Post("/oauth/apple/callback", client.Callback(identity, mapIdentity))
```

Use an Apple-configured `client`, not the Google baseline. Its encrypted state
cookie protects this callback and uses `SameSite=None`; HTTPS is required.
Apple sends the name only on first authorization in an unsigned form field,
so store it as optional display text, not an authorization claim. Email can be
a private relay address and is not filtered for verification by this preset.

See the [`Apple` API](https://pkg.go.dev/arkoren.dev/driver/oauth/oauthauth#Apple).

## Generic OIDC

Choose this adapter for another OIDC service. Register a browser client and
callback with the service, then configure its trusted issuer URL:

```go
provider := oauth.OIDC("https://id.example.com/realms/invoices")
```

Pass the provider and registered credentials to the shared constructor; keep
discovery startup and the shared GET callback. Default scopes contain only
`openid`. If supported and needed, use `ClientBuilder.Scopes("openid", "profile",
"email")`. PKCE is enabled by default. No provider-specific tenant or email
policy is inferred: implement those checks before linking the local account.

See the [`OIDC` API](https://pkg.go.dev/arkoren.dev/driver/oauth/oauthauth#OIDC).

## Generic OAuth2

Choose this adapter only when the provider has OAuth2 but no OIDC. Register
your application/callback and obtain its trusted authorization and token URLs.
Supply your own authenticated user-info function:

```go
provider := oauth.OAuth2(oauth.Endpoint{
	AuthURL:  "https://id.example.com/authorize",
	TokenURL: "https://id.example.com/token",
}, loadIdentity)
```

`loadIdentity` is `oauth.UserInfoFunc`: it receives `context.Context` and an
authenticated `*http.Client` and returns `oauth.Identity`. It must call the
provider's user API with that context, check status, close the response body,
and validate a nonempty stable issuer/subject and any email policy. The generic
adapter does not impose a provider-specific subject schema.

Pass the provider and registered credentials to the shared constructor and
use its GET callback. There are no default scopes; configure `Scopes` for the
provider's user API. Endpoints are prepared without network I/O, and `Start`
is a no-op. PKCE starts enabled; disable it only if unsupported. User-info
failures become `502`, and `Verify` is unsupported because there is no OIDC
verifier.

See the [`OAuth2` and `UserInfoFunc` API](https://pkg.go.dev/arkoren.dev/driver/oauth/oauthauth#OAuth2).

## Handle Failures

Invalid state, failed ID-token checks, provider callback errors, and a token
endpoint's native `invalid_grant` rejection return `ErrInvalidCallback` (`400`).
Start a fresh login rather than replaying an invalid, expired, or consumed code.
Other exchange failures return `ErrProviderUnavailable` (`502`). Returned errors
retain native causes for explicit inspection; provider response text and claims
can be sensitive, so do not dump those chains into logs.

Discovery errors retain native causes and context cancellation/deadline identity
when applicable. The bearer-only [verification path](https://arkoren.dev/docs/guides/authentication/oidc-bearer.md#handle-verification)
has additional context checks; do not assume every browser callback or local
policy denial behaves the same way.

## Events

If you need outcome events, configure `ClientBuilder.Events` with your existing
[event bus](https://arkoren.dev/docs/guides/events/overview.md). Nil publishes nothing. Successful callbacks
publish issuer and subject, which can identify a person. Logout has no payload
and reports local mutation, not provider logout or durable session persistence.

`LoginFailedEvent.Err` contains `ErrInvalidCallback`, `ErrProviderUnavailable`,
or an opaque local-login classification, without the original cause.
`ProviderError` is an untrusted, allowlisted RFC 6749 callback hint; unknown
values and arbitrary descriptions are omitted. Configuration, not-started, and
missing-session exits before cookie consumption do not publish callback outcomes.
`TokenRejectedEvent.Err` is likewise a fixed opaque classification; unsupported
providers and missing audiences return before publication. Do not classify opaque
errors by matching their text. Returned errors retain their separate diagnostic
chains, and event context values are not sanitized by these payload limits.

Ordinary `fmt` formatting redacts owned client/draft configuration; `Config`
formatting redacts a nonempty client secret. This is not a general serialization
or logging guarantee: arbitrary claims, native objects, callback captures, and
explicitly unwrapped errors remain outside that boundary.

For all functions and options, see the
[`oauthauth` API reference](https://pkg.go.dev/arkoren.dev/driver/oauth/oauthauth).

---

Source: https://arkoren.dev/docs/guides/authentication/oidc-bearer.md
[HTML documentation](https://arkoren.dev/docs/guides/authentication/oidc-bearer)

# OIDC Tokens

An API can accept an existing JWT issued by a trusted OpenID Connect provider
without browser routes or a session. `oauthauth.Client.Verify` validates the
token and returns an external identity; `auth.Bearer` connects that result to
request authentication.

The provider or a separate client flow acquires and refreshes tokens. This
adapter does not implement a token endpoint, authorization server, opaque-token
introspection, or provider revocation checks. Plain OAuth2 adapters, including
GitHub, cannot use `Verify`. For browser sign-in, use [OAuth Login](https://arkoren.dev/docs/guides/authentication/oauth.md).

## Prepare Verification

Register an audience intended for this API with your provider. An **audience**
names a token's intended recipient. Prefer separate API and browser-client
audiences; do not accept a browser client ID merely because it is convenient.
Use a trusted issuer from application configuration, never from unverified
request claims.

Install `arkoren.dev/driver/oauth` as shown in
[OAuth client setup](https://arkoren.dev/docs/guides/authentication/oauth.md#configure-clients). With an existing
`cipher *crypto.Encrypter` from [Encryption](https://arkoren.dev/docs/guides/encryption/overview.md) and your
registered `clientID`, configure the verifier:

```go
import oauth "arkoren.dev/driver/oauth/oauthauth"

client, err := oauth.New(
	oauth.OIDC("https://id.example.com/realms/invoices"),
	cipher,
	clientID,
	func(verification *oauth.ClientBuilder) {
		verification.Audiences("https://api.example.com/invoices")
	},
)
```

Handle the constructor error before use. The API-only path still requires a
non-nil encrypter and nonempty client ID because they are constructor inputs;
it needs no redirect URL or client secret. `Audiences` is required by `Verify`.
Without it, verification returns `ErrNoAudiences` rather than defaulting to the
browser client ID.

Construction performs no network I/O. Register `client.Start` in your existing
application composer, importing `arkoren.dev/core/application`:

```go
compose := func(composer *application.Composer) {
	composer.Startup(client.Start)
}
```

Startup discovers endpoints and prepares the verifier. Until it succeeds,
verification returns `ErrNotStarted`. Key retrieval can occur later during
verification, so successful discovery does not prove every request can reach
the signing-key endpoint. Keep the returned client pointer; it owns no stop
hook and does not stop its shared encrypter.

## Set Verification Boundaries

`Verify` checks the JWT signature, expiry, trusted issuer, and whether at least
one token audience matches a configured API audience. It requires a nonempty
final subject after any provider-specific mapping. Google and Microsoft apply
their [preset identity checks](https://arkoren.dev/docs/guides/authentication/oauth.md#choose-providers); generic OIDC does not
infer tenant restrictions or email-verification policy.

The discovered provider supplies signing keys through its JWKS endpoint, a
published set of verification keys. The underlying OIDC verifier caches keys
and can fetch them during signature verification. Accepted algorithms come
from supported discovery metadata, with an RS256 fallback when none are set;
the token header alone does not define the trust policy. Arkoren's builder does
not expose a custom algorithm allowlist or native-verifier configuration. Use a
different application verifier when your policy needs controls this adapter
does not expose.

Time checks use the service's wall clock. The current native verifier rejects
expired tokens and allows five minutes of tolerance for a future `nbf`
(not-before) claim; that is not a general five-minute expiry grace period.
There is no clock or skew setting on `ClientBuilder`. Keep service clocks
synchronized, and do not assume signature/expiry checks enforce a maximum token
age or every provider's additional time policy.

> [!WARNING]
> Audience and signature checks do not prove a JWT is an API access token.
> `Verify` rejects any payload `nonce` claim and a payload `typ` equal to `ID`
> case-insensitively, but an unmarked JWT with an accepted audience can pass.
> Enforce your provider's access-token purpose policy before granting API access.

These marker checks are heuristics, not a universal access-token profile; the
adapter does not require every provider to use `at+jwt`. If additional signed
claims must be checked, wrap `client.Verify` with your application checks or
use a provider-specific verifier. A successful verification still needs
[authorization](https://arkoren.dev/docs/guides/authorization/overview.md).

Local JWT verification is not an online check that the provider still accepts
the token or account. A token can remain acceptable until expiry even after
provider-side revocation. A current local-account lookup can enforce your own
disabled-account policy, but cannot substitute for provider introspection.

## Require Identity

When handlers work directly with the provider identity, use its exact type
throughout middleware and lookup. Import `auth` and `request` from their
`arkoren.dev/core/http/` packages. With your existing API router group `api`:

```go
api.Use(auth.Middleware(auth.Bearer(client.Verify)))
api.Use(auth.Require[oauth.Identity]())
```

This assumes the audience and purpose policy above is sufficient for your
provider. Otherwise pass your stricter application verifier instead of
`client.Verify`. Authentication middleware alone permits missing credentials;
`Require` rejects that anonymous path. Invalid bearer credentials produce `401`
with `WWW-Authenticate: Bearer error="invalid_token"`.

In a handler, read the same identity type:

```go
external, err := request.User[oauth.Identity](r)
if err != nil {
	return err
}
```

`external.Issuer` and `external.Subject` identify the external account.
`*oauth.Identity` and your local `User` are different context keys, not alternate
views of this value. `Claims`, `Email`, and `Name` remain provider-controlled
data; do not log arbitrary claims or use email to link local accounts.

## Load Local Accounts

If handlers need current local permissions, map the verified issuer/subject
pair to an existing account. Keep token scopes request-local instead of mutating
a shared cached account. For example:

```go
type User struct {
	ID     int
	Scopes []string
}
```

With standard-library `context` and `slices`, `accounts.ByIdentity` returning
your application's `Account`, and a `client` satisfying your provider's purpose
policy, use:

```go
verifyUser := func(ctx context.Context, token string) (User, error) {
	external, err := client.Verify(ctx, token)
	if err != nil {
		return User{}, err
	}
	account, err := accounts.ByIdentity(ctx, external.Issuer, external.Subject)
	if err != nil {
		return User{}, err
	}
	if account.Disabled {
		return User{}, auth.ErrUnauthenticated
	}

	return User{ID: account.ID, Scopes: slices.Clone(external.Scopes)}, nil
}
```

The repository must reject unknown links with an error other than
`auth.ErrNoCredentials`. That sentinel would permit credential fallback, not
reject the presented token. A zero user with nil error would count as a success.
Do not create or merge accounts automatically from the email claim; see
[local-account linkage](https://arkoren.dev/docs/guides/authentication/oauth.md#link-local-accounts).

Wire the mapped type, not the external type:

```go
api.Use(auth.Middleware(auth.Bearer(verifyUser)))
api.Use(auth.Require[User]())
```

Scope checks and resource permissions belong in
[Authorization Policies](https://arkoren.dev/docs/guides/authorization/policies.md). `oauth.HasScopes` works
with `oauth.Identity`, not this local `User`; use a User-typed rule for the mapped
identity. Token scopes limit the presented credential's authority rather than
replacing the account's resource permissions.

## Handle Verification

Direct `Verify` calls preserve their error identity and native causes. After
unsupported-provider and missing-audience checks, an ended context is rejected
with `context.Canceled` or `context.DeadlineExceeded`, including when cached or
local keys could otherwise succeed. A final context check also prevents identity
publication after cancellation during verification. These checks do not promise
that every underlying key-fetch operation stops with the request.

Through `auth.Bearer`, verifier errors become an invalid-token HTTP response,
without exposing the error message to the client. The attached cause can still
reach diagnostics. Native errors can contain provider-controlled claims, so
inspect deliberately instead of logging entire chains. Optional
[token-rejection events](https://arkoren.dev/docs/guides/authentication/oauth.md#events) contain a bounded opaque error instead;
they do not retain the native cause.

One started OIDC client can handle browser callbacks and APIs, but each uses
its own audience rule: callbacks require the positional client ID, while
`Verify` uses configured API audiences. For MCP, apply the same bearer and
required-user middleware outside the mount and follow
[MCP endpoint security](https://arkoren.dev/docs/guides/mcp/securing-mcp.md) for challenges, admission, and
tool/resource authorization. The CLI sends existing credentials; it does not
acquire or refresh them.

For all functions and options, see the
[`oauthauth` API reference](https://pkg.go.dev/arkoren.dev/driver/oauth/oauthauth).

---

Source: https://arkoren.dev/docs/guides/authorization/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/authorization/overview)

# Authorization Overview

A signed-in user is not necessarily allowed to export a report or change
another customer's records. Authentication establishes an identity;
**authorization** decides what that identity may do. Arkoren's `gate` package
runs your permission checks before the protected action proceeds.

A **rule** is a Go function that receives a context and your authenticated user
type, then returns `(bool, error)`. You supply the roles, ownership checks, and
other application decisions. The gate supplies ordered evaluation and the
missing-user and forbidden errors.

## Identity Boundaries

Configure [credential resolution](https://arkoren.dev/docs/guides/authentication/overview.md#resolve-credentials)
before using the gate. Rules must accept the exact type that authentication puts
in the context: `User` and `*User` are different types. A mismatch produces
`auth.ErrUnauthenticated` (`401`), even if credentials were valid for the other
type. The gate does not resolve credentials itself.

The examples use standard-library `context`, `auth` from
`arkoren.dev/core/http/auth`, and `gate` from
`arkoren.dev/core/http/auth/gate`. Assume your authenticator stores this
application-owned identity:

```go
type User struct {
	ID         int
	CustomerID int
	CanExport  bool
}
```

These fields come from your trusted identity loader, not request form fields.
Focused snippets omit routine error propagation, but retain checks that prevent
unauthorized work.

## Route Admission

Use a route rule for an ability that does not depend on a particular record.
For example, an export entitlement can admit users to the report-export route:

```go
func canExport(_ context.Context, user User) (bool, error) {
	return user.CanExport, nil
}
```

With your authenticated router `r` and application handler `exportReport`, apply
the rule before the handler runs:

```go
r.With(gate.Require(canExport)).Post("/reports/{id}/export", exportReport)
```

No user of type `User` produces `401`. A rule returning `false, nil` produces
`gate.ErrForbidden` (`403`); a non-nil error is returned unchanged, even if the
boolean is `true`. An infrastructure failure therefore does not become either
permission or an ordinary denial. [Handling Denials](https://arkoren.dev/docs/guides/authorization/denials.md) explains how
these errors render.

If you pass several rules, every rule must allow. They run in argument order,
stopping at the first denial or error; put inexpensive checks before external
lookups. Express alternatives inside a rule with ordinary Go conditions.
`gate.Require` panics with no rules. For sign-in alone, use
`auth.Require[User]()` instead.

## Resource Checks

Passing `canExport` does not authorize every report. Restrict which records the
caller can load through a [scoped store lookup](https://arkoren.dev/docs/guides/authorization/row-access.md), then check any
additional action-specific permission against the stored record. Never build an
ownership decision from a request-supplied `CustomerID` or `OwnerID`.

For reports whose existence is already visible to the caller, an owner-only
export rule can close over the report returned by your store:

```go
type Report struct {
	ID         int
	CustomerID int
	OwnerID    int
}

func canExportReport(report Report) func(context.Context, User) (bool, error) {
	return func(_ context.Context, user User) (bool, error) {
		return report.CustomerID == user.CustomerID && report.OwnerID == user.ID, nil
	}
}
```

In the handler, with `report` already loaded through that scoped lookup, stop
before exporting if the resource check fails:

```go
if err := gate.Authorize(r.Context(), canExport, canExportReport(report)); err != nil {
	return err
}
```

Keep checks in the shared application operation when HTTP, tools, and jobs can
perform the same action. A transport-only gate cannot protect a direct call;
[MCP security](https://arkoren.dev/docs/guides/mcp/securing-mcp.md) also separates tool availability from
resource permission. A check followed by a database write can race with another
request; use [conditional writes and locking](https://arkoren.dev/docs/guides/authorization/row-access.md#conditional-writes)
when permission or state can change between them.

## Deferred Work

A queued export might run after membership or permissions change. Load the
current user and the report through your authorized store again when the job
runs, rather than trusting the queued permission snapshot. With those trusted
`user` and `report` values, attach the actor to the job context and reuse the
rules:

```go
ctx = auth.NewContext(ctx, user)
if err := gate.Authorize(ctx, canExport, canExportReport(report)); err != nil {
	return err
}
```

`auth.NewContext` attaches a value; it does not authenticate that value. The job
owns resolving the queued actor reference to a current identity. Do not attach
an arbitrary user decoded from the job payload and treat it as trusted.

As related checks grow, [Writing Policies](https://arkoren.dev/docs/guides/authorization/policies.md) keeps them together
without introducing a separate policy language. For all functions and options,
see the [`gate` API reference](https://pkg.go.dev/arkoren.dev/core/http/auth/gate).

---

Source: https://arkoren.dev/docs/guides/authorization/policies.md
[HTML documentation](https://arkoren.dev/docs/guides/authorization/policies)

# Writing Policies

When several handlers need the same billing decision, keep that decision in a
**policy**: an application-owned struct with rule methods and explicit
dependencies. A policy is not an Arkoren registry or base type. Its methods
follow the [gate rule shape](https://arkoren.dev/docs/guides/authorization/overview.md), so callers use the same checks from
HTTP handlers, tools, or jobs.

## Feature Policies

For a team billing feature, keep the authenticated actor separate from the team
being managed. These application types distinguish global permissions from the
role held in one team:

```go
type User struct {
	ID          int
	Permissions map[string]bool
	Scopes      []string
}

type Team struct {
	ID int `db:"id"`
}

type Membership struct {
	Role string `db:"role"`
}
```

Authentication must store exactly `User`, not `*User`. Load `Permissions` from
your trusted application data. Populate `Scopes` only when verified token
credentials supply them; the [token checks](#token-limits) below use that field
for a separate decision.

Give the policy its configured database rather than reaching for a global.
Import `database` from `arkoren.dev/core/service/database`:

```go
type BillingPolicy struct {
	db *database.Database
}

policy := &BillingPolicy{db: db}
```

Construct it during application setup and share it with the operations that
need billing checks. Keep shared fields and captured inputs immutable during
requests; the gate does not synchronize your policy's state.

## Team Roles

A **role** groups responsibilities. If a user can be an owner in one team and a
viewer in another, look up the role for both the actor and the target team; a
global `User.Role` cannot express that boundary.

Assume `memberships` has a unique `(team_id, user_id)` pair and a `role` column.
Import standard-library `context` and `errors`, and `query` from
`arkoren.dev/foundation/query`. A billing rule allows owners and billing members:

```go
func (policy *BillingPolicy) CanManage(team Team) func(context.Context, User) (bool, error) {
	return func(ctx context.Context, user User) (bool, error) {
		member, err := policy.db.Find[Membership](ctx, query.Table("memberships", func(statement *query.Builder) {
			statement.Select("role")
			statement.Where("team_id", team.ID)
			statement.Where("user_id", user.ID)
		}))
		if errors.Is(err, database.ErrNoRows) {
			return false, nil
		}
		if err != nil {
			return false, err
		}

		return member.Role == "owner" || member.Role == "billing", nil
	}
}
```

No membership or an unrecognized role denies access. A lookup failure returns
an error instead of disguising an unavailable database as a missing membership.
Name the rule after the action, so changing the qualifying roles does not change
what callers ask.

With `team` returned by your [authorized store lookup](https://arkoren.dev/docs/guides/authorization/row-access.md), apply the
rule before changing billing details. Import `gate` from
`arkoren.dev/core/http/auth/gate`:

```go
if err := gate.Authorize(ctx, policy.CanManage(team)); err != nil {
	return err
}
```

The store lookup owns whether the caller may discover the team; this rule owns
whether the caller may manage its billing. Neither decision authorizes a
different team ID supplied later in the request. Keep writes tied to the same
target, and use [atomic write boundaries](https://arkoren.dev/docs/guides/authorization/row-access.md#shared-invariants) when
membership changes and writes must be serialized.

## Explicit Permissions

Use **permissions**, named abilities, when roles are too coarse. For example,
allow selected staff to prepare billing exports without making them owners of
every team. A missing map entry denies by default:

```go
func (policy *BillingPolicy) CanExport(_ context.Context, user User) (bool, error) {
	return user.Permissions["billing:export"], nil
}
```

This is an action-level permission, not access to all billing records. Combine
it with the target's membership rule and scoped query when exporting one team's
data. You can resolve role grants and individual grants into `Permissions`
during identity loading; Arkoren does not compute that union for you.

Permissions loaded once remain a snapshot until you reload them. Decide whether
that lifetime meets your revocation needs. Long-running or queued operations
should [reload the actor before work](https://arkoren.dev/docs/guides/authorization/overview.md#deferred-work), and sensitive
writes might need a transactional membership check rather than a request-start
snapshot.

## Token Limits

A **scope** limits a client's delegated authority. It must restrict, not replace,
the user's application permissions and resource access. For a bearer-token API
whose authenticator stores the `User` above, require the verified token scope
as another rule of that same type. Here `slices` is the standard-library package:

```go
func hasBillingScope(_ context.Context, user User) (bool, error) {
	return slices.Contains(user.Scopes, "billing:export"), nil
}

if err := gate.Authorize(ctx, hasBillingScope, policy.CanExport, policy.CanManage(team)); err != nil {
	return err
}
```

This token-specific path denies when the scope is absent. Do not fill `Scopes`
from a header or unverified claim, or copy the user's full permissions into it:
either action can expand what a restricted credential may do.

If authentication instead stores `oauth.Identity` directly, the OAuth adapter
provides a scope rule. Import
`oauth "arkoren.dev/driver/oauth/oauthauth"`; with the bearer-authenticated
router `api`, apply it as admission:

```go
api.With(gate.Require(oauth.HasScopes("billing:export"))).Post("/billing/exports", exportBilling)
```

`HasScopes` requires every named scope, and an empty requirement allows any
authenticated `oauth.Identity`. It neither authenticates nor checks local team
membership. Do not combine it with `User` rules in the same gate call: the types
differ. Map verified issuer-and-subject identity to your local actor before local
resource decisions, following [OIDC bearer verification](https://arkoren.dev/docs/guides/authentication/oidc-bearer.md).
[API keys](https://arkoren.dev/docs/guides/authentication/api-tokens.md#issue-keys) also need their credential
permissions intersected with the actor's authority.

## Policy Tests

Call rules directly to test decisions independently of HTTP. With `policy`
backed by a [test database](https://arkoren.dev/docs/guides/testing/integration-tests.md), seed a team and a
membership whose role is `viewer`. For that authenticated `viewer`, denial is
the expected result:

```go
allowed, err := policy.CanManage(team)(t.Context(), viewer)
if err != nil || allowed {
	t.Fatalf("viewer billing access = %v, %v; want false, nil", allowed, err)
}
```

Also cover owner and billing roles, absent membership, a member of a different
team, unknown roles, and database failure. Test missing scopes separately from
missing application permissions, then test protected routes for allowed,
forbidden, and anonymous callers. Direct rule tests cannot prove that a handler
actually calls the rule.

For all functions and options, see the
[`gate` API reference](https://pkg.go.dev/arkoren.dev/core/http/auth/gate).

---

Source: https://arkoren.dev/docs/guides/authorization/row-access.md
[HTML documentation](https://arkoren.dev/docs/guides/authorization/row-access)

# Row-Level Access

An invoice-list permission does not tell you which invoices a caller may read.
Put that record boundary in the SQL query, rather than loading all invoices and
discarding forbidden ones afterward. The database then filters the rows before
pagination, counting, or returning private data to the application.

A **scope** is a reusable query predicate. This is application-side SQL
filtering, not a database server's row-level security configuration. Keep scopes
inside your store methods so callers cannot accidentally omit them.

## Ownership Scopes

Assume your authenticator supplies `User` from trusted account data, and the
`invoices` table has `id`, `customer_id`, and `reference` columns. These are
application-owned types; database tags name SQL columns, not JSON input fields:

```go
type User struct {
	ID         int
	CustomerID int
}

type Invoice struct {
	ID         int    `db:"id"`
	CustomerID int    `db:"customer_id"`
	Reference  string `db:"reference"`
}
```

Import `query` from `arkoren.dev/foundation/query`. The customer's identity,
not a customer ID submitted in the request, determines the invoice boundary:

```go
func visibleInvoices(user User) query.Scope {
	return func(predicate *query.PredicateBuilder) {
		predicate.Where("customer_id", user.CustomerID)
	}
}
```

`Builder.Scope` ANDs each scope with the complete ordinary condition chain,
grouping both so a later `OrWhere` or an OR in `WhereRaw` cannot bypass it.
Scopes also apply to updates, deletes, and derived counts. An empty scope adds
no restriction, so every authorization scope must actually express its boundary.
Raw SQL still needs trusted SQL text and separately bound values; grouping does
not make arbitrary input safe SQL.

## Scoped Stores

Make the actor an explicit store argument. With standard-library `context` and
`database` from `arkoren.dev/core/service/database`, the single-invoice lookup
owns both its requested ID and its customer restriction:

```go
type Invoices struct {
	db *database.Database
}

func (invoices *Invoices) Find(ctx context.Context, user User, id int) (Invoice, error) {
	return invoices.db.Find[Invoice](ctx, query.Table("invoices", func(statement *query.Builder) {
		statement.Where("id", id)
		statement.Scope(visibleInvoices(user))
	}))
}
```

Apply the same scope in list methods. For an existing `db`, authenticated
`user`, and `page` of type `pagination.PageParams` from
`arkoren.dev/core/pagination`, the page and its total both use scoped rows:

```go
invoices := query.Table("invoices", func(statement *query.Builder) {
	statement.Scope(visibleInvoices(user))
	statement.OrderBy("id")
})
result, err := db.Paginate[Invoice](ctx, invoices, page)
```

The unique `id` provides a stable order. Filtering after fetching would leave
gaps and could expose unscoped totals; [pagination](https://arkoren.dev/docs/guides/queries/pagination.md)
belongs after the authorization predicate. Give genuinely unrestricted billing
jobs a separately named store operation with its own admission boundary, rather
than making a missing scope the way to request all records.

For membership-based visibility, use a correlated `WhereExists` subquery that
matches both the record's team and the authenticated user's ID. That checks
membership within SQL rather than running a separate rule for each returned
row. [Query Builder](https://arkoren.dev/docs/guides/queries/query-builder.md) explains subqueries; indexes
and uniqueness constraints should match the membership keys you query.

## Private Absence

A requested invoice outside the caller's scope and a nonexistent invoice both
produce `database.ErrNoRows`. In the handler, translate that absence to `404`;
do not turn other database failures into missing records. Here `store` is your
`Invoices` store, `id` is the parsed request ID, `errors` is standard-library,
and `problem` is `arkoren.dev/foundation/problem`:

```go
invoice, err := store.Find(r.Context(), user, id)
if errors.Is(err, database.ErrNoRows) {
	return problem.NotFound()
}
if err != nil {
	return err
}
```

The request ID selects a candidate; it is not evidence of ownership. This
response does not disclose whether an out-of-scope invoice exists. It does not
promise identical timing or hide information exposed by other endpoints. If the
caller may know the resource exists but lacks an action permission, use a
[forbidden response](https://arkoren.dev/docs/guides/authorization/denials.md#response-statuses) instead.

## Conditional Writes

Reading an authorized invoice and later updating it by ID alone leaves a gap:
the data used to authorize it might change between statements. Repeat the
boundary in the write so the database evaluates the predicate as part of the
mutation.

For a reference change, assume decoding and validation already produced this
input. Do not accept customer ownership fields in the writable shape:

```go
type ChangeReference struct {
	Reference string `json:"reference" db:"reference"`
}
```

Inside the scoped store operation, with `input` of that type and the requested
`id`, convert the allowlisted struct with `query.Values`:

```go
change := query.Table("invoices", func(statement *query.Builder) {
	statement.Where("id", id)
	statement.Scope(visibleInvoices(user))
	statement.Update(query.Values(input))
})
```

Execute the conditional write and distinguish an execution error from a
zero-row result:

```go
affected, err := db.Exec(ctx, change)
if err != nil {
	return err
}
if affected == 0 {
	return problem.NotFound()
}
```

No invoice outside the scope is changed. Affected-row reporting depends on the
driver: a no-op update can also report zero. Returning the same `404` preserves
the privacy boundary but does not prove why no change was reported. If your API
must distinguish a visible no-op from absence, perform the scoped lookup and
update under an appropriate transaction/lock protocol instead.

Apply the boundary to deletes too. Scopes do not apply to inserts: assign new
rows' customer/team ownership from authorized application values, and validate
every referenced record against that same boundary. A source invoice check
does not authorize attaching it to an arbitrary destination account.

## Shared Invariants

Removing a team member has two separate requirements: the actor must be allowed
to remove that member, and the team must retain at least one owner. A transaction
alone does not stop two owners from concurrently passing the last-owner check
and removing each other.

Serialize these operations by locking the parent team row before checking
current membership. Assume `Team` maps the team's stored ID, `teamID` is the
requested candidate, and `targetUserID` identifies the member to remove—not the
authenticated actor. Your application policy `members.CanRemove(team)` checks
the actor's current owner membership for this exact team, using this same `db`.

Import `gate` from `arkoren.dev/core/http/auth/gate`. In the shared application
operation, the transaction first locks the team, then checks the actor:

```go
err := db.WithTransaction(ctx, func(ctx context.Context) error {
	team, err := db.Find[Team](ctx, query.Table("teams", func(statement *query.Builder) {
		statement.Select("id")
		statement.Where("id", teamID)
		statement.ForUpdate()
	}))
	if err != nil {
		return err
	}
	if err := gate.Authorize(ctx, members.CanRemove(team)); err != nil {
		return err
	}

	return removeLockedMember(ctx, team, targetUserID)
})
```

`removeLockedMember` is your application helper, not an Arkoren function. It
must look up the target membership by both `team.ID` and `targetUserID`, count
owners in that same team, and reject removing the last owner with
`problem.Conflict()` (`409`). Otherwise, it deletes using both keys. The count,
target lookup, and delete all use the callback's `ctx` and the same database so
they join its transaction. Keeping one owner is not a substitute for the actor
check shown above.

After those checks, the helper's deletion keeps the target inside the locked
team. Do not delete by `targetUserID` alone, since that could remove the same
user's memberships in other teams:

```go
remove := query.Table("memberships", func(statement *query.Builder) {
	statement.Where("team_id", team.ID)
	statement.Where("user_id", targetUserID)
	statement.Delete()
})
affected, err := db.Exec(ctx, remove)
```

Handle execution failure and a missing target before reporting success. This
fragment runs inside `removeLockedMember` with the transaction context, not
after `WithTransaction` has returned.

This example assumes the caller may know the team exists. For private teams,
use a scoped lookup before exposing any team-specific response and map
out-of-scope absence to `404`; recheck current action permission under the lock.
Do not translate an arbitrary infrastructure error to `404` or `403`.

Every operation that can alter this invariant or its permission basis must
follow the same parent-lock protocol, including owner demotion and removal of
the actor's membership. Use an isolation level whose reads observe the preceding
lock holder's committed changes; verify that behavior for your database.
Keep the protocol in the shared operation, not only an HTTP handler, so tools
and jobs cannot bypass it. Row locks require supported database behavior;
[Locking](https://arkoren.dev/docs/guides/database/locking.md#row-locks) covers PostgreSQL/MySQL support and
the alternatives for SQLite.

Return an error from the transaction callback to roll back. Never replace its
context with the outer context, use it concurrently, or retain it after the
callback returns. [Transactions](https://arkoren.dev/docs/guides/database/transactions.md) explains that
lifetime and commit-failure handling.

## Boundary Tests

Seed two customers and exercise every read, count, update, and delete using the
other customer's record ID. Assert no private rows are returned or changed,
including when ordinary predicates contain OR conditions. Test lists' totals
as well as their items.

For membership changes, test an unauthorized actor, a target in another team,
last-owner rejection, and concurrent owner removals using your real database
driver. Query-generation tests alone cannot prove the database's lock and
isolation behavior. Also verify that every route and tool reaches the shared
store operation rather than an unscoped alternative.

For all query operations, see the
[`query` API reference](https://pkg.go.dev/arkoren.dev/foundation/query); for
execution, see the
[`database` API reference](https://pkg.go.dev/arkoren.dev/core/service/database).

---

Source: https://arkoren.dev/docs/guides/authorization/denials.md
[HTML documentation](https://arkoren.dev/docs/guides/authorization/denials)

# Handling Denials

A caller who lacks permission needs a different response from a caller whose
permission could not be checked. Preserve that distinction when handling gate
errors: a denied rule means the application reached a decision; a failed lookup
means it did not. Both stop the protected action, but they require different
client behavior and operational investigation.

## Response Statuses

Choose the response from the authentication state, the permission decision, and
what the caller may learn about the resource:

| Situation | Response | Responsibility |
| --- | --- | --- |
| No identity of the rule's exact type is present | `401 Unauthorized` | The gate returns `auth.ErrUnauthenticated`. |
| A known actor lacks permission and may know the resource exists | `403 Forbidden` | A rule returning `false, nil` produces `gate.ErrForbidden`. |
| The record is absent or outside the caller's visibility boundary | `404 Not Found` | Your handler maps scoped `database.ErrNoRows` to `problem.NotFound()`. |
| An authorized action violates current state, such as removing the last team owner | `409 Conflict` | Your application returns a conflict problem. |
| A database or service fails while checking permission | Normal error handling | The gate returns the rule's non-nil error unchanged. |

A public report can return `403` when a reader cannot edit it. A private invoice
outside the caller's [query scope](https://arkoren.dev/docs/guides/authorization/row-access.md#private-absence) should return
the same `404` as a missing invoice. The gate does not automatically turn
`ErrNoRows` into `404`; that translation belongs at your resource boundary.

Do not convert every authorization error to `403`, or treat every failed
database query as missing membership. An ordinary error uses the router's normal
error rendering, typically `500`; a rule returning a problem retains that
problem's status. See [HTTP errors](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md) for
the rendering boundary. Invalid credentials belong to authentication, while an
unexpected `401` with valid credentials can indicate a `User`/`*User` mismatch
between authentication and the rule.

## Forbidden Details

The shared `gate.ErrForbidden` problem uses this stable identity and default
message:

```json
{
  "type": "tag:arkoren.dev,2026-09:core:http/auth/gate:forbidden",
  "title": "Forbidden",
  "status": 403,
  "detail": "You do not have permission to perform this action."
}
```

Clients can use the `type` rather than parsing prose. The default response does
not disclose which rule denied access. HTTP rendering can add `instance` from
the request path; this response is not a sanitized diagnostic record.

If the caller may know why an action is unavailable, derive a more useful detail
without changing the problem identity. Assume `team` was loaded through your
authorized store and `policy.CanInvite(team)` is your application rule. Import
standard-library `errors`, `gate` from `arkoren.dev/core/http/auth/gate`, and
`problem` from `arkoren.dev/foundation/problem`:

```go
err := gate.Authorize(r.Context(), policy.CanInvite(team))
if errors.Is(err, gate.ErrForbidden) {
	return gate.ErrForbidden.Derive(func(details *problem.Builder) {
		details.Detail("Only team owners can invite members.")
	})
}
if err != nil {
	return err
}
```

`Derive` creates a new value; it does not mutate the shared problem. The derived
value still matches `gate.ErrForbidden` with `errors.Is`. Missing identity and
infrastructure errors pass through unchanged instead of acquiring a misleading
owner-only message.

Keep details about permitted actions, not private membership records, hidden
resource IDs, token contents, or internal service errors. For a resource the
caller may not discover, use the scoped `404` path instead of explaining a
resource-specific denial.

## Denial Events

The free functions `gate.Require` and `gate.Authorize` publish no events. If
you need denial observation, construct a `gate.Gate` with your existing
application event bus and use its methods consistently:

```go
access := gate.New(func(access *gate.Builder) {
	access.Events(app.Events())
})
```

With an authenticated router `r`, an application rule `canExport`, and handler
`exportReport`, protect the route using that instance:

```go
r.With(access.Require(canExport)).Post("/reports/{id}/export", exportReport)
```

The gate publishes `gate.Denied` (`http.gate.denied`) only when a rule returns
`false` without an error. It publishes nothing for allowed calls, missing
identities, or rule failures. Direct checks that need observation must call
`access.Authorize`, not the free `gate.Authorize`. Without an event bus, a gate
instance publishes nothing too.

The `gate.DeniedEvent` payload contains `Rule`, the zero-based position of the
first denying rule, and `Method`, the HTTP method for `Require`. `Authorize`
has no request, so its `Method` is empty. Later rules do not run. Rule positions
depend on the order at each call site; they are not globally unique policy IDs.

There is no resource path, identity, query string, or failure cause in the
payload. `DeniedEvent.Path` is not available; remove old path reads rather than
replacing them with raw request URLs. The event context still retains request
values, including the authenticated identity, and custom HTTP methods are
client-controlled. Select attributes deliberately; neither event contexts nor
problem `instance` values are safe to dump into diagnostics.

## Safe Observation

Register a listener on the same bus. This example uses standard-library
`context` and `log/slog`; it intentionally logs neither context values nor the
raw method:

```go
unsubscribe, err := app.Events().Listen(gate.Denied, func(_ context.Context, denied gate.DeniedEvent) error {
	slog.Warn("authorization denied", "rule", denied.Rule)

	return nil
})
```

Handle the registration error before relying on observation. The subscription's
owner retains `unsubscribe` and calls it when the subscription is no longer
needed; deferring it in a short-lived setup function would remove the listener
too early. If you record methods, map them to a fixed allowlist of standard
methods plus an `other` value rather than logging arbitrary client strings.

Every denial emits: there is no deduplication, and a client controls the rate.
Keep listeners inexpensive, bound log volume, and place
[rate limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) before the gate. A denial
listener is not an audit of successful actions or failed permission lookups;
observe those separately under your application's privacy policy.
[Event auditing](https://arkoren.dev/docs/guides/events/auditing.md) explains durable audit design.

For all functions, options, and event fields, see the
[`gate` API reference](https://pkg.go.dev/arkoren.dev/core/http/auth/gate).

---

Source: https://arkoren.dev/docs/guides/hashing/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/hashing/overview)

# Hashing Overview

Password hashing lets you check a password without storing a readable copy.
Arkoren's `hash.Hasher` produces an encoded record containing the algorithm,
work settings, random salt, and digest. Store that entire record; there is no
decryption operation. A leaked record can still be attacked by guessing passwords.

Unlike fast general-purpose hashes such as SHA-256, password-hashing algorithms
deliberately spend time, and Argon2 also spends memory, on each guess.

The examples assume a configured `*hash.Hasher` named `hasher`; choose and wire
one on [Drivers](https://arkoren.dev/docs/guides/hashing/drivers.md). Argon2id is the recommended choice for new passwords;
bcrypt serves existing bcrypt databases and compatibility requirements.
Focused fragments leave routine errors visible rather than showing application
error propagation. Handle those errors before using their results.

## Password Policy

Your password policy decides which new passwords the application accepts. The
hash policy decides how much work storing and checking them costs. Hashing does
not enforce minimum length, reject common passwords, or authorize a password
change: even an empty value is valid input to the hashing contract.

Validate new passwords before hashing, and apply request-size and password-byte
limits before any expensive check. bcrypt requires a limit of **72 bytes** on
both creation and checking. Its native verifier can accept a longer attempt
sharing a valid 72-byte prefix; never truncate the attempt. Argon2 has no such
bcrypt-specific limit, but still needs [bounded input and work](https://arkoren.dev/docs/guides/hashing/cost.md#bound-request-work).

Do not apply a newly tightened minimum-length rule to existing login attempts;
an older valid password might no longer satisfy your registration policy.
Keep applicable resource limits and scheme-specific input limits on that path.

## Store Passwords

At registration, hash only the validated password field, not the whole request.
For example, your decoded registration input can have this shape:

```go
type RegistrationInput struct {
	Email    string `json:"email"`
	Password string `json:"password"`
}
```

With `input` already validated, create the stored record:

```go
stored, err := hasher.Hash(input.Password)
```

On success, persist `stored` unchanged in the account's password-hash column.
Each call generates a fresh salt, so hashing the same password twice produces
different records that both verify. You do not generate or store a separate salt.

Use a column large enough for the chosen format and future policy changes.
A text column avoids a fixed format-length assumption; if you choose a bounded
column, test that complete records fit after changing salt or digest lengths.
Never store or log the plaintext password alongside the record.

## Check Passwords

At login, load the hash from application-owned account storage. Do not let a
request supply the stored hash or its work settings: verification uses the
record's settings, not the current new-hash policy.

Assume a decoded `LoginInput` with a `Password string` field, an account named
`account`, and `auth` imported from `arkoren.dev/core/http/auth`. After enforcing
the input limits above, check the attempt:

```go
matched, err := hasher.Check(input.Password, account.PasswordHash)
if err != nil {
	return err
}
if !matched {
	return auth.ErrUnauthenticated
}
```

Only `true, nil` passes this credential check. A well-formed hash with a wrong
password returns `false, nil`. An error matching
`errors.Is(err, hash.ErrInvalidHash)` means malformed or unsupported stored data,
not an ordinary mismatch. Import `errors` from the standard library and `hash`
from `arkoren.dev/core/service/hash` when classifying it. Other encoding or
driver errors are failures too; none authorizes login.

Keep failures distinguishable internally without exposing storage details,
passwords, or hashes to the client. The final digest comparison does not make
the whole login request constant-time. Account lookup, unknown-account paths,
different stored costs, and errors can change duration. Follow the
[account-hiding login flow](https://arkoren.dev/docs/guides/authentication/session-login.md#check-passwords)
for dummy checks and uniform public rejection; it is not a guarantee of identical
request timing.

`Check` does not update the account or consume the password. After success,
[check for a hash upgrade](https://arkoren.dev/docs/guides/hashing/upgrading.md) before discarding the attempt from your
request handling.

## Input Encoding

Exact `string` and `[]byte` values use their raw bytes. Other values, including
defined string or slice types, are JSON-encoded before hashing or checking.
Use the same encoding on both paths; explicitly convert a defined password type
to `string` if raw password bytes are intended. `HashRaw` and `CheckRaw` accept
bytes directly.

The typed methods clear their own encoded copy after use, not your input.
The raw methods also leave the caller's input unchanged. This is not a promise
that all copies of a credential are erased from application memory.

Slow password hashing is for guessable, human-chosen secrets. For securely
generated, high-entropy [API keys](https://arkoren.dev/docs/guides/authentication/api-tokens.md#issue-keys),
a fast SHA-256 digest is appropriate instead. If you must recover a secret's
original value, such as an OAuth refresh token, use
[encryption](https://arkoren.dev/docs/guides/encryption/overview.md), not password hashing.

For all operations, see the [`hash` API reference](https://pkg.go.dev/arkoren.dev/core/service/hash).

---

Source: https://arkoren.dev/docs/guides/hashing/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/hashing/drivers)

# Hashing Drivers

Choose Argon2id for new password storage. Choose bcrypt when an existing user
database or another system requires bcrypt compatibility. Both run in your
process, generate a fresh random salt for every hash, and store work settings
inside the encoded record. Neither verifies the other's format; changing
algorithms requires a [verification-time migration](https://arkoren.dev/docs/guides/hashing/upgrading.md#change-algorithms).

## Compatibility Matrix

This comparison covers Arkoren's two hashing drivers, not every feature of their
underlying libraries. Verification reads stored work settings; your new-hash
configuration does not cap verification cost.

| Driver | Random Salt | Work Settings | Memory Settings | Parallelism Settings | Rehash Detection | Other Driver's Format |
| --- | --- | --- | --- | --- | --- | --- |
| [Argon2id](#argon2id) | Yes | Yes — Iteration count. | Yes — Kibibytes per operation. | Yes | Yes — Full policy, including salt length. | No |
| [bcrypt](#bcrypt) | Yes | Yes — Logarithmic cost. | No | No | Yes — Complete-record validation and stored cost. | No |

Both support concurrent callers after construction and require no lifecycle
hooks. Neither has credentials, a network connection, a native-client accessor,
or a companion adapter. Facade metrics are available with either driver.

## Argon2id

The Argon2 driver creates Argon2id records by default. It also supports Argon2i
when another system requires that variant. From your application's module
directory, install the driver:

```sh
go get arkoren.dev/driver/argon2
```

### Create Hasher

Import the driver and facade, then construct one shared hasher:

```go
import (
	"arkoren.dev/core/service/hash"
	"arkoren.dev/driver/argon2/argon2hash"
)

hasher := hash.New(argon2hash.New())
```

Construction performs no hashing or lifecycle I/O and returns no error. The
published settings are immutable. Use this `hasher` to
[store and check passwords](https://arkoren.dev/docs/guides/hashing/overview.md#store-passwords).

### Set Policy

Defaults come from the native library's `argon2.DefaultConfig`: Argon2id version
19, 64 MiB of memory, three iterations, four parallel lanes, a 16-byte salt, and
a 32-byte digest. Measure these defaults under your deployment's resource limits
before changing them, following [Choosing Cost](https://arkoren.dev/docs/guides/hashing/cost.md).

If measurements justify 128 MiB per operation while retaining the other defaults,
construct an alternative policy:

```go
hasher := hash.New(argon2hash.New(func(policy *argon2hash.Builder) {
	policy.MemoryCost(128 * 1024)
}))
```

`MemoryCost` takes kibibytes, not bytes. `TimeCost`, `Parallelism`, `SaltLength`,
and `HashLength` configure the other dimensions. Zero settings restore native
defaults after all callbacks. Nonzero settings are not validated at construction;
use application-controlled settings, not arbitrary request values.

For an Argon2i interoperability requirement, import native constants from
`github.com/matthewhartstonge/argon2` and construct this alternative:

```go
hasher := hash.New(argon2hash.New(func(policy *argon2hash.Builder) {
	policy.Mode(argon2.ModeArgon2i)
	policy.TimeCost(3)
}))
```

Keep at least three iterations for Argon2i. Retain Argon2id for new passwords
otherwise. Only version 19 (`argon2.Version13`) is supported. Setting version 16
or another nonzero unsupported version does not fail construction, but `Hash`
returns an error before computing a digest. Other native hashing failures are
returned to the caller.

### Stored Records

`Check` accepts supported version-19 Argon2id and Argon2i records regardless of
the configured new-hash variant. It uses the record's memory, iteration, and
parallelism values as stored. Accept only application-owned records and
[bound concurrent work](https://arkoren.dev/docs/guides/hashing/cost.md#budget-concurrent-work).

A mismatch returns `false, nil`. Malformed records and unsupported versions
return an error wrapping `hash.ErrInvalidHash`. Version 16 is rejected, not
silently treated as version 19.

`NeedsRehash` compares memory, iterations, parallelism, mode, version, salt
length, and digest length. Malformed or unsupported records also return `true`;
that does not authorize login or replacement without a successful check.
Neither hashing nor checking modifies the caller's input bytes.

For all settings, see the [`argon2hash` API reference](https://pkg.go.dev/arkoren.dev/driver/argon2/argon2hash).

## bcrypt

The bcrypt driver uses Go's extended crypto library. Choose it for bcrypt
compatibility, and test imported records with known fixture passwords before
depending on historical implementation behavior. Install it from your
application's module directory:

```sh
go get arkoren.dev/driver/bcrypt
```

### Create Hasher

Construct the driver and facade once:

```go
import (
	"arkoren.dev/core/service/hash"
	"arkoren.dev/driver/bcrypt/bcrypthash"
)

hasher := hash.New(bcrypthash.New())
```

Construction returns no error and performs no hashing or lifecycle I/O. Use the
shared [password operations](https://arkoren.dev/docs/guides/hashing/overview.md#store-passwords), enforcing bcrypt's
input limit before both hashing and checking.

### Set Policy

Arkoren's default cost is `bcrypthash.DefaultCost`, or 12. A cost increase of one
approximately doubles the computation. If measurements justify cost 13,
construct an alternative hasher:

```go
hasher := hash.New(bcrypthash.New(func(policy *bcrypthash.Builder) {
	policy.Cost(13)
}))
```

Zero selects 12; other values are clamped to the supported range of 4 through 31.
Clamping is not a resource-safety policy: the upper endpoint is extraordinarily
expensive. Keep the cost application-controlled. Low settings such as 4 are
useful for isolated behavior tests, not a production recommendation.

### Stored Records

`Hash` rejects passwords longer than 72 bytes with the native
`bcrypt.ErrPasswordTooLong` error. Characters can occupy multiple bytes.
Your application must also reject overlength attempts before `Check`, because
native verification can accept a longer value sharing a valid 72-byte prefix.
Never truncate an attempt or migrate its unverified suffix into Argon2id.

Before computing a password check, the driver validates the complete stored
record: exact length, separators, two decimal cost digits, and canonical unpadded
bcrypt base64 for the 22-character salt and 31-character digest. Truncation,
trailing data, invalid characters, line breaks, and nonzero unused encoding bits
produce an error wrapping `hash.ErrInvalidHash`. A canonical record with a wrong
password returns `false, nil` instead.

Complete legacy `$2$` records are supported. Minor labels, including `$2a$`,
`$2b$`, `$2x$`, `$2y$`, and unrecognized minor labels, follow the native verifier;
the driver adds no minor-label allowlist. This does not establish compatibility
with every historical bcrypt implementation.

`NeedsRehash` validates the same complete record without password computation,
then compares stored cost with configured cost. A supported minor label alone
does not request renewal. Checking uses the stored cost even after you change
the new-hash policy. Input bytes remain unchanged.

For all settings, see the [`bcrypthash` API reference](https://pkg.go.dev/arkoren.dev/driver/bcrypt/bcrypthash).

## Observe Operations

To measure outcomes through either facade, pass your existing telemetry service
as `metrics` when constructing the facade around an existing `driver`:

```go
hasher := hash.New(driver, func(policy *hash.Builder) {
	policy.Telemetry(metrics)
})
```

Omitting telemetry or setting nil follows `telemetry.Default()` per call.
Passing `telemetry.New(nil)` explicitly disables reporting; import `telemetry`
from `arkoren.dev/core/service/telemetry`. See
[Telemetry](https://arkoren.dev/docs/guides/observability/telemetry.md) for service configuration.

Facade counts and durations cover encoding and driver outcomes, not traces or
events. A password mismatch is not an error; a rehash-needed observation is not
a persisted password update. Metric labels contain no password, stored hash, or
native error text. Direct driver calls bypass these metrics.

For facade operations and metric names, see the [`hash` API reference](https://pkg.go.dev/arkoren.dev/core/service/hash).

---

Source: https://arkoren.dev/docs/guides/hashing/cost.md
[HTML documentation](https://arkoren.dev/docs/guides/hashing/cost)

# Choosing Cost

Password hashing makes guesses expensive, including legitimate login attempts.
Choose a hash policy that fits your latency, CPU, and memory budgets at expected
concurrency. A quick check on an idle laptop does not establish that a busy
production instance can sustain the same policy.

Start with the [driver defaults](https://arkoren.dev/docs/guides/hashing/drivers.md): Argon2id uses 64 MiB, three
iterations, and four lanes; bcrypt uses cost 12. These are starting policies,
not throughput or security guarantees for your deployment. This page assumes
you have selected a driver and can run a controlled test on representative
hardware under the same container resource limits as the service.

## Set Budgets

Decide how much of a login's latency budget can go to password verification,
and how much CPU and memory one instance can reserve for password work. Account
for normal traffic, bursts, account-hiding dummy checks, and verification-time
rehashes, which add a new hash after a successful check.

An attacker's offline guessing hardware can differ substantially from yours.
There is no elapsed-time result that proves a password is unguessable. Prefer a
measured policy your service can sustain over an arbitrarily high setting that
makes login unavailable.

## Measure Checks

Use a fixed, locally generated fixture, not a real password or imported record.
Add this bounded measurement test to your application's `_test.go` file. Import `testing`
and `time`, `hash` from `arkoren.dev/core/service/hash`, and `argon2hash` from
`arkoren.dev/driver/argon2/argon2hash`.

```go
func TestHashCost(t *testing.T) {
	if testing.Short() {
		t.Skip("measures password hashing cost")
	}
	hasher := hash.New(argon2hash.New())
	const password = "local measurement fixture"
	stored, err := hasher.Hash(password)
	if err != nil {
		t.Fatal(err)
	}

	for range 3 {
		if err := t.Context().Err(); err != nil {
			t.Fatal(err)
		}
		start := time.Now()
		matched, err := hasher.Check(password, stored)
		if err != nil || !matched {
			t.Fatalf("fixture verification failed: matched=%t, error=%v", matched, err)
		}
		t.Logf("check duration: %s", time.Since(start))
	}
}
```

This performs one setup hash and three sequential checks with known default
settings. The reported durations cover each facade check, not fixture creation
or account lookup. No password or encoded record is logged.

Run one uncached measurement explicitly:

```sh
go test -run '^TestHashCost$' -count=1 -parallel=1 -timeout=30s -v
```

The test timeout stops the test process; it is not a per-hash cancellation API.
Do not replace the fixed loop with an automatic search that repeatedly raises
cost, or measure arbitrary stored records supplied by users. Inspect variation
across the samples, then repeat a bounded run under representative service load.
Record the policy, hardware, limits, and observed durations so later comparisons
have meaning. Measure hash creation separately if registration or upgrades are a
significant part of your load.

## Tune Settings

For Argon2id, increasing `MemoryCost` reserves more memory per operation;
increasing `TimeCost` performs more iterations. Parallelism changes CPU scheduling
and does not multiply the configured memory budget by the lane count. Change
one setting at a time and repeat the bounded measurement before deployment.

For example, to test a measured candidate with four iterations and otherwise
default settings, replace the test's construction with:

```go
hasher := hash.New(argon2hash.New(func(policy *argon2hash.Builder) {
	policy.TimeCost(4)
}))
```

This is a candidate to measure, not a universal recommendation. For bcrypt,
use `bcrypthash` from `arkoren.dev/driver/bcrypt/bcrypthash` instead:

```go
hasher := hash.New(bcrypthash.New(func(policy *bcrypthash.Builder) {
	policy.Cost(13)
}))
```

A bcrypt cost increase of one approximately doubles computation; it is not
comparable numerically to an Argon2 iteration count. Keep settings controlled by
deployment configuration, not request fields. Constructor normalization does not
make extreme costs safe to run.

Changing the policy affects new hashes only. Existing checks use their stored
parameters; [upgrade them after verification](https://arkoren.dev/docs/guides/hashing/upgrading.md). `NeedsRehash`
reports any policy difference, including a lower target cost, rather than only
stronger settings. Do not automatically weaken stored hashes to handle a burst.
Use deliberately low work settings only for isolated behavior tests, with a
separate production construction path.

## Budget Concurrent Work

Argon2's memory cost applies to each in-flight operation. Four default-policy
operations require roughly 256 MiB of Argon2 working memory, before runtime and
application overhead. Reserve headroom rather than treating that arithmetic as
a complete process-memory estimate. bcrypt needs a CPU budget even though its
working-memory demand is smaller.

Use a shared admission limit across checks, registration hashes, password changes,
dummy checks, and rehashes on each instance. For a simple no-wait policy,
initialize a channel once at service construction:

```go
slots := make(chan struct{}, 4)
```

The capacity of four is illustrative; derive yours from the deployment budget.
Every password operation must use the same instance-wide gate. In the operation
handler, acquire a slot before doing the expensive work; `errors` is the
standard-library package:

```go
select {
case slots <- struct{}{}:
	defer func() {
		<-slots
	}()
default:
	return errors.New("password processing capacity exhausted")
}
```

The slot stays held until that handler returns. Map the capacity error to your
application's overload response, not a wrong-password result. Unlike a waiting
queue, this rejects excess work immediately and does not accumulate waiting
requests. A nil channel admits no work in this form; initialize it explicitly.

Neither hashing driver's operations accept a context. Once computation begins,
request cancellation cannot interrupt it. Check cancellation before admission
where useful, but do not start a goroutine per request and assume abandoning its
result stops the computation. A per-instance gate also does not coordinate
capacity across replicas.

## Bound Request Work

Apply [rate limits](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) before password work on
login, registration, password changes, and recovery endpoints that create hashes.
Concurrency limits cap in-flight work; rate limits control how frequently clients
can claim that capacity. Neither replaces request-body or password-length limits.

Bound request bodies before decoding, then enforce an application-chosen maximum
password byte length before hashing. bcrypt requires at most 72 bytes on both
creation and verification; never silently truncate. Argon2 still needs an input
limit even without that scheme-specific bound. Keep stored hashes in trusted
application storage: a valid format can encode work far above your current
policy, so format validation alone is not a resource limit.

For the settings and operations, see the [`hash` API reference](https://pkg.go.dev/arkoren.dev/core/service/hash),
[`argon2hash` API reference](https://pkg.go.dev/arkoren.dev/driver/argon2/argon2hash),
and [`bcrypthash` API reference](https://pkg.go.dev/arkoren.dev/driver/bcrypt/bcrypthash).

---

Source: https://arkoren.dev/docs/guides/hashing/upgrading.md
[HTML documentation](https://arkoren.dev/docs/guides/hashing/upgrading)

# Upgrading Hashes

A stored hash cannot be converted into a hash of the same password under new
settings without the password. A successful login supplies a verified attempt:
check it against the old record, then hash those same password bytes under the
new policy. A password reset is different: it establishes a new password through
your application's recovery flow.

This page assumes configured [hashing drivers](https://arkoren.dev/docs/guides/hashing/drivers.md), trusted account
storage, and the [ordinary password check](https://arkoren.dev/docs/guides/hashing/overview.md#check-passwords). Budget
the old check and new hash together under your [password-work limits](https://arkoren.dev/docs/guides/hashing/cost.md).
Do not retain plaintext passwords in a queue or log them for a later migration.

## Detect Renewal

Changing construction settings affects new hashes, not the parameters used to
check an existing record. `NeedsRehash` compares a stored record with the current
policy; it does not rewrite storage or prove a password match.

Argon2 compares memory, iterations, parallelism, variant, version, salt length,
and digest length. bcrypt validates the complete record and compares cost.
Differences in either direction request renewal, so lowering the configured
cost can also flag a stronger stored hash. Review the target policy before
deploying a bulk policy change.

Both drivers return `true` for malformed records; Argon2 also does so for
unsupported stored versions. These records cannot be upgraded through a login
check that fails. Never treat `NeedsRehash` alone as permission to replace them.

## Verify First

Assume a decoded `LoginInput` with a `Password string` field, an account named
`account`, and the current `*hash.Hasher` named `hasher`. Apply the appropriate
input limits first. For bcrypt, reject attempts over 72 bytes before comparison:
native checking can accept an extra suffix after a valid 72-byte prefix.

With `auth` imported from `arkoren.dev/core/http/auth`, establish the credential
decision separately from renewal:

```go
matched, err := hasher.Check(input.Password, account.PasswordHash)
if err != nil {
	return err
}
if !matched {
	return auth.ErrUnauthenticated
}
```

Only this success path may attempt renewal. Preserve malformed-storage and other
failures internally instead of counting them as ordinary wrong passwords. Map
public failures without leaking account existence or storage details, as in
[Session Login](https://arkoren.dev/docs/guides/authentication/session-login.md#check-passwords).

## Replace Safely

Use a conditional write so renewal cannot overwrite a password reset that
completed after your account read. In this example, `accounts.ReplacePasswordHash`
is an application-owned repository method, not an Arkoren API. It takes the
account ID, expected old hash, and replacement, and returns `(applied bool, err error)`.
It must atomically update only that account whose stored hash still equals the
expected value. No match is a concurrent change, not permission to retry an
unconditional write.

After the successful check above, create a replacement only if policy differs.
Import standard-library `log/slog` for the bounded failure reports:

```go
if hasher.NeedsRehash(account.PasswordHash) {
	fresh, err := hasher.Hash(input.Password)
	if err != nil {
		slog.WarnContext(ctx, "password hash renewal failed", "stage", "hash")
	} else {
		applied, err := accounts.ReplacePasswordHash(ctx, account.ID, account.PasswordHash, fresh)
		if err != nil {
			slog.WarnContext(ctx, "password hash renewal failed", "stage", "store")
		} else if !applied {
			slog.InfoContext(ctx, "password hash renewal skipped", "reason", "concurrent change")
		}
	}
}
```

A successful write installs a fresh salted record. A rejected conditional write
leaves the newer credential untouched. This example treats renewal failures as
best-effort maintenance after a valid check, rather than rejecting login because
renewal failed; enforce any separate account or concurrent-reset login policy in
your authentication flow. An unchanged old record can be retried on a later
successful login.

Keep the attempt only for this request's synchronous work. The failure reports
contain no credential, stored hash, or native error text; use similarly bounded
diagnostics if you need to distinguish failures in monitoring.

## Change Algorithms

To move bcrypt accounts to Argon2id, keep a legacy bcrypt hasher alongside your
target Argon2id hasher. New registrations and password changes use the target.
Existing logins route by a known stored format, then let the selected driver
validate the complete record. Do not try another driver after a mismatch or
parse error.

Here `hasher` is the target Argon2id facade, `bcryptHasher` is the legacy facade,
and `bytes` is the standard-library package. Select the checker from the
application-owned record:

```go
checker := hasher
switch {
case bytes.HasPrefix(account.PasswordHash, []byte("$2")):
	if len(input.Password) > 72 {
		return auth.ErrUnauthenticated
	}
	checker = bcryptHasher
case bytes.HasPrefix(account.PasswordHash, []byte("$argon2id$")),
	bytes.HasPrefix(account.PasswordHash, []byte("$argon2i$")):
	// The Argon2 driver validates the stored variant, version, and parameters.
default:
	return auth.ErrUnauthenticated
}
```

A recognized prefix is only routing information, not format validation. Record
unknown formats internally as unsupported storage without exposing the record
to clients. Now verify with the selected checker:

```go
matched, err := checker.Check(input.Password, account.PasswordHash)
if err != nil {
	return err
}
if !matched {
	return auth.ErrUnauthenticated
}
```

After success, use the conditional renewal block above with the target `hasher`.
Its `NeedsRehash` returns true for bcrypt records it cannot parse, causing the
verified attempt to be hashed with Argon2id. It also renews supported Argon2
records whose policy differs. The prior successful legacy check—not that policy
result—is what makes migration eligible.

The 72-byte guard prevents an unverified bcrypt suffix from becoming part of a
new Argon2 password. If an imported account relied on historical truncation,
send it through verified recovery instead of truncating and migrating an attempt.

## Import Records

Treat imports as a controlled migration, not an endpoint for client-submitted
hashes. Audit formats and encoded work factors before enabling checks: a
parseable record can still demand excessive CPU or memory. Use isolated,
resource-limited tests with known fixture passwords, never production passwords
in source or logs. Confirm both valid and invalid attempts with the matching
driver before importing the full set.

Argon2 accepts Argon2i and Argon2id version 19 records. Version 16 and other
versions fail with `hash.ErrInvalidHash`. Do not edit a version label to force
acceptance; use a correct, separately reviewed legacy verifier or a verified
recovery flow for unsupported data.

bcrypt requires complete canonical salt and digest encodings. Truncated records,
trailing data, and noncanonical encodings are storage failures, not mismatches.
Legacy `$2$` and native-accepted minor labels do not prove compatibility with
historical algorithm quirks; test representative fixtures and retain the
[bcrypt input limit](https://arkoren.dev/docs/guides/hashing/drivers.md#bcrypt).

Neither driver accepts arbitrary legacy schemes such as salted SHA-1. Without
a suitable reviewed verifier, require password recovery rather than guessing a
conversion or accepting a hash as a credential.

## Inactive Accounts

Accounts that never log in retain their old records, as do accounts whose renewal
has not succeeded. Count those records by known scheme or policy to assess what
remains; a policy count is not evidence that their passwords still verify.

Choose whether to retain acceptable legacy records or require recovery for weak
ones. Before disabling any old credential, ensure the account has a working,
verified recovery route and plan the user impact. Do not clear hashes merely to
make a migration count reach zero: that can lock people out permanently.

For verification and policy operations, see the [`hash` API reference](https://pkg.go.dev/arkoren.dev/core/service/hash).

---

Source: https://arkoren.dev/docs/guides/keys/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/keys/overview)

# Application Keys

Your application's encrypted records and signed values must survive restarts.
Provision a key once, protect it, and load the intended material wherever that
purpose runs. The `key` package turns stored bytes into immutable, validated key
values; it does not choose your configuration source or manage rotation.

## Choose Material

Choose the type required by the consumer, not the file extension. A shared
secret protects data using the same secret on both sides. A signing identity
instead separates the private key that signs from the public key that verifies.

| Type | Material | Use |
| --- | --- | --- |
| `key.Shared` | Exactly 32 secret bytes | Symmetric encryption or shared-secret signing, with no public counterpart. |
| `key.Signing` | ECDSA P-256 private identity | Signatures verified with the matching public key. |
| `key.Exchange` | X25519 private identity | Key agreement inside an existing authenticated protocol. |

Use independent material for each purpose. For example, give encrypted records,
[signed links](https://arkoren.dev/docs/guides/http-capabilities/secure-links.md), and
[pagination cursors](https://arkoren.dev/docs/guides/queries/pagination.md) separate `Shared` values.
AES-256-GCM and ChaCha20-Poly1305 consume `Shared` for encryption; link signers and
cursor codecs use it for HMAC-based checks, not public-key signatures.

`Signing` is not an encryption key, and `Exchange` is neither a signer nor an
encryption protocol. These types do not accept arbitrary algorithms: an RSA or
Ed25519 key is not a `Signing` identity, and a P-256 ECDH key is not an `Exchange`
identity. For Web Push, follow [VAPID Keys](https://arkoren.dev/docs/guides/push/vapid-keys.md). For Apple Sign
In, load the Apple-issued P-256 key; local generation cannot establish provider
provenance. Provider API tokens, webhook secrets, password-hash salts, session
identifiers, and [TLS certificates](https://arkoren.dev/docs/guides/http-networking/certificates.md) are
different inputs, not additional key types.

## Generate Keys

Use [Key Commands](https://arkoren.dev/docs/guides/cli/keys.md) to provision protected files, validate them,
and extract public components. That guide owns command syntax, output permissions,
and recovery from incomplete writes.

Add the package from your application's module directory:

```sh
go get arkoren.dev/foundation/key
```

Import `arkoren.dev/foundation/key`. If you write your own provisioning tool,
use the package's generators. This disposable in-memory example creates a shared
key without displaying or persisting its secret:

```go
material, err := key.GenerateShared()
```

Generation uses `crypto/rand` and does not store the result. `GenerateSigning`
creates an independent P-256 identity; `GenerateExchange` creates an independent
X25519 identity. Provision production material outside application startup and
store it in protected deployment storage with a protected backup. Never generate
a replacement to hide a missing startup key: existing data or identities still
depend on the original.

## Load Configuration

Your application selects configuration sources and resolves
[secret-store values](https://arkoren.dev/docs/guides/secrets/managing.md). The package does not read an
implicit `APP_KEY`, fetch secrets, or search parent directories. For example,
an application's configuration layer can supply these named inputs:

```go
type KeyConfig struct {
	EncryptionBase64 string
	SigningFile      string
}
```

With an existing `config KeyConfig`, parse the encryption secret already read
from protected storage:

```go
material, err := key.ParseShared(config.EncryptionBase64)
```

Handle `err` before constructing an encryption consumer. On success, `material`
is a `key.Shared`; continue with [Encryption Keys](https://arkoren.dev/docs/guides/encryption/keys.md) for
driver wiring and encrypted-data use. `ParseShared` requires padded standard
base64 decoding to exactly 32 bytes. It accepts surrounding whitespace but
rejects embedded line breaks, noncanonical padding bits, and malformed encoding.
Base64 is an encoding, not encryption. `NewShared(raw)` imports exactly 32 raw
bytes instead; neither operation proves entropy. Do not use a password or a
predictable 32-byte value as a key.

For a mounted signing file, restrict read access to the application and trusted
operators before loading it. Keep the file and its parent directories under
trusted control: loaders follow the supplied path, including symlinks, and do
not check deployment permissions or sandbox untrusted paths.

```go
identity, err := key.LoadSigning(config.SigningFile)
```

On success, `identity` is a `key.Signing`. Loading reads without modifying the
file. `LoadShared(path)` reads base64 text; `LoadExchange(path)` reads X25519
private PEM. If configuration already contains PEM bytes, use
`ParseSigning(contents)` or `ParseExchange(contents)` instead.

Both private PEM parsers require exactly one unencrypted PKCS #8 `PRIVATE KEY`
block with the matching algorithm. They accept surrounding whitespace, but
reject extra blocks, PEM headers, non-whitespace junk, trailing DER bytes, other
curves, and alternate forms such as SEC 1 `EC PRIVATE KEY`. Encoded inputs and
file reads are limited to 64 KiB, including surrounding whitespace.

For existing native keys, `NewSigning(private)` accepts an `*ecdsa.PrivateKey`
from `crypto/ecdsa`, validates P-256 and public/private consistency, and copies
the material. `NewExchange(private)` accepts an X25519 `*ecdh.PrivateKey` from
`crypto/ecdh`. Neither constructor retains the caller's mutable key object.

Malformed, unsupported, and oversized material returns `key.ErrInvalid`; check
it with `errors.Is` from the standard-library `errors` package. File operations
preserve OS error causes, so a missing file can match `os.ErrNotExist` instead.
Validation errors do not include key contents or input-derived parser diagnostics.
Do not log the source configuration when reporting a failure.

## Export Material

Before exporting a key passed in by another component, check its `Err()` result.
All three types have invalid zero values; `Err()` returns `key.ErrInvalid` for
them. An empty export is not a replacement for validation: zero shared values
export empty bytes or text, and zero asymmetric identities export nil native
keys or PEM bytes. A zero signing identity's `VAPIDPublicKey()` is empty text.

After successfully loading the signing `identity`, obtain only its public
component for a verifier:

```go
public := identity.PublicPEM()
```

`public` is SPKI `PUBLIC KEY` PEM without private material. `PublicKey()` returns
a native public key instead. Both asymmetric types support these exports.
`Signing.VAPIDPublicKey()` is a different representation: the uncompressed P-256
public point in unpadded base64url, not PEM or DER. Use it only where the
[Web Push setup](https://arkoren.dev/docs/guides/push/vapid-keys.md#publishing-public-keys) requires it.

> [!WARNING]
> `Shared.Bytes()` and `Shared.Base64()` expose the shared secret. `PEM()` on
> either asymmetric type exports unencrypted PKCS #8 private material, and
> `PrivateKey()` exposes a native private key. Never send these exports to logs,
> browser responses, or public configuration. Persist them only through your
> protected provisioning path; an encoding does not protect a secret.

Formatting key values with `fmt` and logging them through `slog` produces fixed
redaction markers. That is not a persistence format or a general serialization
guarantee. Do not serialize an identity to publish a public key; explicitly
export its public component. Strings or native objects obtained through private
exports do not inherit the key value's redaction behavior.

Inputs and mutable exports are detached from the immutable value. Modifying an
exported slice or native key does not change the original or later exports.
This ownership rule does not promise to erase secret copies from process memory.

## Using Exchange Keys

Use an `Exchange` identity only when integrating a protocol that already defines
peer authentication and key derivation. `PrivateKey()` and `PublicKey()` supply
native X25519 keys; `ParseExchangePublic(contents)` imports a peer's X25519
SPKI `PUBLIC KEY` PEM under the same strict envelope and size rules as private
PEM. Parsing checks representation, not who owns the key. It also does not reject
every low-order point; native ECDH rejects those inputs when agreement is attempted.

> [!WARNING]
> An attacker who substitutes a peer public key can establish a different shared
> secret with you. Authenticate peer keys through your protocol and apply its
> key-derivation function with distinct purpose contexts. Never use raw ECDH
> output directly as an encryption key.

The package supplies no handshake, transport, or key-derivation API. Keep these
responsibilities in the existing protocol rather than designing a new encryption
protocol from the key containers.

## Rotate Keys

The application owns persistence, deployment, and rotation. Loading a new file
does not update an already constructed consumer. Keep the intended material
consistent across replicas serving the same purpose, and coordinate replacement
with the consumer's compatibility rules.

Retain previous encryption keys while stored values or backups need them; follow
[encryption rotation](https://arkoren.dev/docs/guides/encryption/keys.md#rotating-keys) for the rollout and
re-encryption sequence. Replacing a link or cursor secret can invalidate issued
values; plan around their lifetime and the consumer's supported key handling.
VAPID replacement requires [subscription migration](https://arkoren.dev/docs/guides/push/vapid-keys.md#rotating-identity),
not merely a different private key on the server.

For constructors, formats, and exports, see the
[`key` API reference](https://pkg.go.dev/arkoren.dev/foundation/key).

---

Source: https://arkoren.dev/docs/guides/encryption/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/encryption/overview)

# Encryption Overview

A calendar integration needs to keep an access token and recover it when sending
requests. Encryption lets you store that token as **ciphertext**, bytes that
require the secret key to recover the original value. Arkoren's
`crypto.Encrypter` also checks integrity: changed ciphertext fails authentication
instead of becoming trusted application data.

## Choosing Encryption

Use encryption when your application needs the original value back. Use
[password hashing](https://arkoren.dev/docs/guides/hashing/overview.md) for login passwords, which you only
need to verify. A [secret store](https://arkoren.dev/docs/guides/secrets/overview.md) protects the encryption
key outside your application data; it does not replace encryption of that data.
If a value may remain readable but must resist changes, consider a signature,
as used by [secure links](https://arkoren.dev/docs/guides/http-capabilities/secure-links.md).

Ciphertext authentication does not authorize the caller, establish freshness,
or prevent replay. Apply [authorization](https://arkoren.dev/docs/guides/authorization/overview.md) before
exposing a decrypt path. A compromised application with access to its keys can
still read the values it protects.

## Encrypting Values

Assume `enc` is a configured `*crypto.Encrypter`, from
`arkoren.dev/core/service/crypto`. [Drivers](https://arkoren.dev/docs/guides/encryption/drivers.md) owns construction;
[Encryption Keys](https://arkoren.dev/docs/guides/encryption/keys.md) owns persistent keys and rotation. Focused snippets
keep error results visible and omit routine propagation; handle each error
before using its result.

After your calendar callback has validated the provider response, keep the
recoverable credentials in a named application type:

```go
type CalendarCredentials struct {
	AccessToken  string `json:"access_token"`
	RefreshToken string `json:"refresh_token"`
}
```

With `credentials` holding that value, encrypt it before handing it to storage:

```go
sealed, err := enc.Encrypt(credentials)
```

`Encrypt` JSON-encodes the value, then encrypts those bytes with the current
driver. Encoding can fail before encryption happens. On success, `sealed` is
binary ciphertext, not JSON or text. Each encryption uses a fresh random nonce,
so encrypting the same credentials again produces a different result.

For an already-encoded payload named `payload`, use `EncryptRaw(payload)`
instead. It skips JSON encoding. Neither raw operation changes its input, and
its returned bytes do not alias the input.

## Reading Values

After selecting a stored connection the caller may access, decode its
ciphertext into the same application shape:

```go
credentials, err := enc.Decrypt[CalendarCredentials](sealed)
if err != nil {
	return err
}
```

Only use `credentials` after the error check. Decryption authenticates the
ciphertext before JSON decoding. A successful result restores the credential
fields; it does not validate provider token expiry or permission to use them.

Match raw encryption with raw decryption:

```go
payload, err := enc.DecryptRaw(sealedPayload)
```

Here `sealedPayload` was produced by `EncryptRaw`. `DecryptRaw` returns the
authenticated bytes without interpreting their format. Keep the encoding and
application schema compatible for as long as stored values exist; see
[Storing Data](https://arkoren.dev/docs/guides/encryption/storing.md).

## Handling Failures

With either shipped driver, truncated, altered, wrong-key, or wrong-context
ciphertext produces an error wrapping `crypto.ErrInvalidCiphertext`. The error
does not distinguish tampering from a missing historical key. If previous
drivers are configured, the facade tries them before returning a failure.

Import standard-library `errors` to classify an invalid stored credential
without including its contents in a user-facing error:

```go
if errors.Is(err, crypto.ErrInvalidCiphertext) {
	return errors.New("calendar credentials are unavailable; reconnect the calendar")
}
```

Reject invalid client-supplied ciphertext. For stored application data, investigate
key availability, cipher choice, authenticated context, and corruption rather
than silently replacing it with empty credentials. Do not log keys, plaintext,
or whole ciphertext values while investigating; inspect native error causes
before exporting them to logs.

Authenticated bytes can still fail JSON decoding into the requested type.
That is a decoding error, not `ErrInvalidCiphertext`; keep the original stored
value for recovery. A stopped driver returns `crypto.ErrEncrypterClosed`, which
indicates a lifecycle error rather than invalid user data. A usable previous
driver can still decrypt even if the current driver has stopped; stopping one
driver does not disable the entire facade.

For all operations, see the [`crypto` API reference](https://pkg.go.dev/arkoren.dev/core/service/crypto).

---

Source: https://arkoren.dev/docs/guides/encryption/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/encryption/drivers)

# Encryption Drivers

Choose AES-GCM for a server with hardware AES support. Consider
ChaCha20-Poly1305 when AES acceleration is unavailable; benchmark your workload
if performance determines the choice. Both run in your process without a cloud
encryption service and require a persistent 32-byte `key.Shared` value.
[Encryption Keys](https://arkoren.dev/docs/guides/encryption/keys.md) explains loading and rotating that material.

## Compatibility matrix

This compares Arkoren's shipped drivers, not every cipher mode available from
their underlying libraries. Facade metrics apply only when operations go through
`crypto.Encrypter`.

| Driver | Authenticated encryption | Purpose AAD | 32-byte keys | 16- or 24-byte keys | 192-bit nonces | Facade metrics |
| --- | --- | --- | --- | --- | --- | --- |
| [AES-GCM](#aes-gcm) | Yes | Yes | Yes | No | No | Yes |
| [ChaCha20-Poly1305](#chacha20-poly1305) | Yes | Yes | Yes | No | No | Yes |

Both use a random 96-bit nonce and add 28 bytes to the raw plaintext length.
Rotate before 2^32 encryptions under one key across all replicas. Neither
driver exposes XChaCha20 or a caller-supplied nonce. Changing cipher requires
keeping the original driver available to decrypt old values; see
[Rotating Keys](https://arkoren.dev/docs/guides/encryption/keys.md#rotating-keys).

## AES-GCM

The AES driver uses AES-256 in Galois/Counter Mode. It needs no account,
credentials beyond the encryption key, network connection, or `Start` call.

### Installing AES

From your application's module directory, install the driver:

```sh
go get arkoren.dev/driver/aes
```

### Creating Encrypters

Import `crypto` from `arkoren.dev/core/service/crypto` and `aescrypto` from
`arkoren.dev/driver/aes/aescrypto`. With validated material named `material`,
construct the driver and then its facade:

```go
driver, err := aescrypto.New(material)
enc, err := crypto.New(driver)
```

These are focused fragments; handle each construction error before continuing.
The driver builds its cipher locally and rejects a zero `key.Shared` with
`key.ErrInvalid`. Key import rejects lengths other than 32 bytes, so this is
not an AES-128 or AES-192 interface. `crypto.New` rejects nil current or previous
drivers with `crypto.ErrInvalidDriver`.

Register `driver.Stop` with [application cleanup](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md)
immediately after successful acquisition. Keep the constructor's pointer rather
than copying the driver, which contains synchronization state. The facade
shares that driver and does not stop it for you. Use `enc` for the ordinary
[typed encryption task](https://arkoren.dev/docs/guides/encryption/overview.md#encrypting-values).

### Binding Context

**Additional authenticated data** (AAD) binds ciphertext to known context
without encrypting that context. For a calendar-only driver, use a stable
purpose string when constructing it instead of the unconfigured constructor:

```go
driver, err := aescrypto.New(material, func(encrypter *aescrypto.Builder) {
	encrypter.AdditionalData([]byte("calendar-credentials:v1"))
})
```

No AAD is bound by default. `AdditionalData` replaces and copies the bytes;
nil or empty clears them. Construction copies the completed setting again.
The exact bytes must match during decryption. They are not included in the
ciphertext and do not increase its size. The setting applies to every operation
on this driver, not to an individual call; [Storing Data](https://arkoren.dev/docs/guides/encryption/storing.md#binding-records)
explains the record-binding boundary. There are no network, retry, or timeout
settings.

### Handling Shutdown

Operations support concurrent use. `Stop` waits for in-flight operations, clears
the driver's owned raw key copy, and releases its cipher reference. Cancellation
of the cleanup context does not interrupt that wait or skip release. Repeated
stops are safe; later operations return `crypto.ErrEncrypterClosed`.

This does not clear the caller's immutable key, exported bytes, plaintext, or
all internal cipher state. Treat process memory as sensitive. The constructor
pointer formats as `[REDACTED AES driver]` in ordinary `fmt` diagnostics, but
arbitrary serialization, reflection, copied values, and explicit exports are
not a sanitization boundary.

Invalid ciphertext wraps `crypto.ErrInvalidCiphertext`; input shorter than the
12-byte nonce additionally wraps `aescrypto.ErrMismatchedAESNonceSize`.
No native client or companion encryption adapter is exposed.

For driver options, see the [`aescrypto` API reference](https://pkg.go.dev/arkoren.dev/driver/aes/aescrypto).

## ChaCha20-Poly1305

The ChaCha20 driver uses the 96-bit nonce variant of ChaCha20-Poly1305, not
XChaCha20. Choose it when software performance without AES acceleration suits
your deployment. It needs no external service or `Start` call.

### Installing ChaCha20

From your application's module directory, install the driver:

```sh
go get arkoren.dev/driver/chacha20
```

### Creating Encrypters

Import `crypto` from `arkoren.dev/core/service/crypto` and `chacha20crypto` from
`arkoren.dev/driver/chacha20/chacha20crypto`. With persistent `key.Shared`
material named `material`, construct the driver and facade:

```go
driver, err := chacha20crypto.New(material)
enc, err := crypto.New(driver)
```

Construction is local. Handle each error before proceeding: a zero key returns
`key.ErrInvalid`, and key import rejects any length other than 32 bytes.
Register `driver.Stop` with [application cleanup](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md)
after acquiring it. Retain the pointer, not a copy, and stop it only after its
consumers finish. The facade does not own cleanup. Use the shared
[typed encryption task](https://arkoren.dev/docs/guides/encryption/overview.md#encrypting-values) with `enc`.

### Binding Context

To restrict this driver to calendar credentials, use the alternative constructor:

```go
driver, err := chacha20crypto.New(material, func(encrypter *chacha20crypto.Builder) {
	encrypter.AdditionalData([]byte("calendar-credentials:v1"))
})
```

The exact AAD bytes must match on decryption. No AAD is the default; repeated
settings replace and copy the bytes, nil or empty clears them, and construction
detaches the completed setting. Context is fixed for the driver's lifetime and
is not stored in its output. There are no timeout, retry, or network settings.

### Handling Shutdown

Operations are safe for concurrent use. `Stop` waits for in-flight operations
even with a canceled context, clears its owned raw key copy, and releases the
cipher reference. It is repeatable; subsequent calls to encrypt or decrypt
return `crypto.ErrEncrypterClosed`. The caller's key and all internal cipher
copies are not erased.

The constructor pointer formats as `[REDACTED ChaCha20 driver]` in ordinary
`fmt` diagnostics. This does not sanitize arbitrary serialization, reflection,
copied values, explicit key exports, or application data.

Authentication failures wrap `crypto.ErrInvalidCiphertext`. Input shorter than
the 12-byte nonce also wraps `chacha20crypto.ErrMismatchedChaCha20NonceSize`.
No native client or companion encryption adapter is exposed.

For driver options, see the [`chacha20crypto` API reference](https://pkg.go.dev/arkoren.dev/driver/chacha20/chacha20crypto).

## Facade Metrics

To observe logical encryption and decryption calls, pass your configured
[telemetry service](https://arkoren.dev/docs/guides/observability/telemetry.md), named `metrics`, to the facade:

```go
enc, err := crypto.New(driver, func(encrypter *crypto.Builder) {
	encrypter.Telemetry(metrics)
})
```

Without an explicit service, nil follows `telemetry.Default()` per call.
`telemetry.New(nil)` explicitly disables reporting. Counts and durations include
JSON work and all fallback attempts as one final operation, with no traces or
events. A successful previous-key outcome does not mean replacement ciphertext
was persisted. Labels contain no keys, plaintext, ciphertext, or native error
text; application logging still needs its own exposure policy. Direct driver
calls bypass these metrics.

For the facade and metric definitions, see the [`crypto` API reference](https://pkg.go.dev/arkoren.dev/core/service/crypto).

---

Source: https://arkoren.dev/docs/guides/encryption/keys.md
[HTML documentation](https://arkoren.dev/docs/guides/encryption/keys)

# Encryption Keys

Encrypted data must remain readable after a restart. Provision a persistent key
once and load it at startup; generating a replacement whenever loading fails
makes old data unreadable. Anyone who obtains the key can decrypt values made
with it when the cipher and required authenticated context are known.

Use independent keys for different purposes. Separate keys limit the data
exposed by one leaked key; purpose AAD prevents cross-purpose substitution but
does not contain a key compromise. Generic key types and encodings belong in
[Keys Overview](https://arkoren.dev/docs/guides/keys/overview.md), and protected provisioning commands belong
in [CLI Keys](https://arkoren.dev/docs/guides/cli/keys.md).

## Loading Keys

Load the persistent key from a protected mounted file or your configured
[secret store](https://arkoren.dev/docs/guides/secrets/overview.md). Both encryption drivers take a
`key.Shared`, which validates exactly 32 bytes. With `key` imported from
`arkoren.dev/foundation/key`, a mounted file can supply the material:

```go
material, err := key.LoadShared("/run/secrets/calendar-key.key")
if err != nil {
	return err
}
```

Fail startup on a missing, unreadable, or malformed key; do not print the file
contents or generate substitute material. For secret-store text, use
`key.ParseShared(encoded)` after successfully retrieving the value through your
configured secret facade. No environment variable or secret name is implicit.
See [Drivers](https://arkoren.dev/docs/guides/encryption/drivers.md) for constructing the selected cipher and owning its
cleanup.

Keep key backups separate from ciphertext backups, with controlled access and
a tested recovery procedure. Retain any key and context needed by backups for
their entire retention period. A ciphertext backup without its key is not a
recoverable backup.

## Selecting Keys

An encrypter has one **current driver**, used for every encryption, and an
ordered list of **previous drivers**, used only as decryption fallbacks. This
is a configured driver ring, not a remotely managed keyring: Arkoren does not
fetch missing keys, watch secret changes, or switch the current key at runtime.

Import `aescrypto` from `arkoren.dev/driver/aes/aescrypto` and `crypto` from
`arkoren.dev/core/service/crypto`. Here `newKey` and `oldKey` are validated
`key.Shared` values loaded from separate protected entries. Build both drivers
with the context used for their ciphertext:

```go
purpose := func(encrypter *aescrypto.Builder) {
	encrypter.AdditionalData([]byte("calendar-credentials:v1"))
}
current, err := aescrypto.New(newKey, purpose)
previous, err := aescrypto.New(oldKey, purpose)
```

Handle each construction error and register cleanup immediately after each
successful acquisition. Then publish the facade with the old driver available
for reads:

```go
enc, err := crypto.New(current, func(encrypter *crypto.Builder) {
	encrypter.Previous(previous)
})
```

Decryption tries the current driver first, then previous drivers in configured
order; repeated `Previous` calls append to that order. New values always use
`current`. The facade copies the driver list, not the driver objects, and is
immutable after construction. Rebuild it to change the ring.

Stop every current and previous driver only after all consumers finish. The
facade does not own their lifecycle. Stopping the current driver alone does
not prevent fallback decryption through an active previous driver.

## Rotating Keys

Rotation changes the key used for future writes. It does not rewrite stored
values. During a rolling deployment, old instances must be able to read new
ciphertext before any writer switches keys:

1. Provision the new key under a separate protected name; retain the old key.
2. Deploy all readers with the old driver current and the new driver in their
   previous list. Wait until every reader can accept either key.
3. Deploy the new driver as current and the old driver as previous. Readers
   from the preceding deployment can already read new writes.
4. [Re-encrypt stored values](https://arkoren.dev/docs/guides/encryption/storing.md#re-encrypting-values) and verify that
   no live writer or stored value still depends on the old driver.
5. Remove the previous driver from active configuration. Retain its key
   securely for backups that still need it.

The previous list is a decryption role, so step 2 can contain a newly provisioned
key. Deployments must preserve the exact cipher and AAD for each driver. Changing
from AES-GCM to ChaCha20-Poly1305 also requires retaining the old cipher for old
reads, even if key material stays available.

Rotate before 2^32 encryptions per key across all replicas sharing that key.
A time-based schedule alone does not enforce this volume limit. If a key leaks,
coordinate rotation and incident response; re-encryption cannot protect
ciphertext copies already obtained with the leaked key.

## Missing Keys

`crypto.New` rejects a nil current or configured previous driver with
`crypto.ErrInvalidDriver`. It does not know which historical keys your data
requires: omitting an old driver can construct successfully and break old reads.
Check the required key inventory during deployment rather than treating facade
construction as proof that all data is readable.

Ciphertext contains no key identifier. Decryption tries each configured driver;
if every attempt fails, the facade returns the current driver's original error,
not the last fallback error. With active shipped drivers, wrong-key, wrong-AAD,
altered, and truncated input wrap `crypto.ErrInvalidCiphertext`. There is no
separate missing-key sentinel that identifies which key to fetch.

Preserve the stored bytes and restore the required historical key or context
through your recovery process. Never retry an invalid value as plaintext or
overwrite it with an empty value. Avoid logging keys, exported key bytes, or
credential payloads when reporting the failure.

## Ciphertext Versions

The shipped ciphertext format contains a nonce, ciphertext, and authentication
tag, but no cipher name, key ID, AAD, or application schema version. The facade
does not negotiate versions or return an unknown-version error. Retain the
configuration needed to interpret existing values.

If your application adds a format discriminator, reject unknown versions before
selecting a decoder. Do not interpret an unknown version as the current one.
Keep old decoders until their values and backups have migrated; a successful
key fallback is not a schema migration. [Storing Data](https://arkoren.dev/docs/guides/encryption/storing.md#keeping-compatibility)
explains which metadata needs authentication and how to preserve compatibility.

For fallback configuration, see the [`crypto` API reference](https://pkg.go.dev/arkoren.dev/core/service/crypto).

---

Source: https://arkoren.dev/docs/guides/encryption/storing.md
[HTML documentation](https://arkoren.dev/docs/guides/encryption/storing)

# Storing Data

Encrypting a credential protects its contents, but your application still owns
the row that contains it, the metadata used to interpret it, and the rules for
replacing it. Decide those boundaries before writing ciphertext so future
deployments and backups remain readable.

## Choosing Storage

Prefer a binary column for ciphertext: PostgreSQL `BYTEA`, MySQL `VARBINARY`
or `BLOB`, or SQLite `BLOB`. Scan it into `[]byte`. Both shipped drivers return
a 12-byte nonce followed by encrypted bytes and a 16-byte tag, adding 28 bytes
to the raw plaintext length. `Encrypt` adds JSON encoding first, so field names
and JSON escaping also affect size. Size a bounded column for the largest
encoded payload or choose an appropriate unbounded column.

Encrypt only fields that need recovery. Leaving identifiers and routing fields
outside ciphertext permits normal queries, but exposes those fields to anyone
who can read storage. Encrypting a whole payload hides more content while
removing ordinary filtering and indexing over its contents. Ciphertext still
reveals its length and does not replace database access controls or encrypted
transport.

## Writing Credentials

Assume `enc` is your configured `*crypto.Encrypter`, `db` is your configured
`*database.Database`, and `ctx` is the operation context. Imports are
`arkoren.dev/core/service/crypto`, `arkoren.dev/core/service/database`, and
`arkoren.dev/foundation/query`. The application owns a `calendar_connections`
table with `id`, `account_id`, and binary `credentials` columns. The connection
and account identifiers come from an authorized application operation, not
arbitrary writable request fields.

Use a named payload that keeps record identity with the credential. Here
`credentials` is a validated `CalendarCredentials` value from
[Overview](https://arkoren.dev/docs/guides/encryption/overview.md#encrypting-values), and `connectionID` and `accountID`
identify the authorized target:

```go
type StoredCalendar struct {
	Version      int                 `json:"version"`
	ConnectionID int64               `json:"connection_id"`
	AccountID    int64               `json:"account_id"`
	Credentials  CalendarCredentials `json:"credentials"`
}
```

Encrypt that payload before building the database update:

```go
sealed, err := enc.Encrypt(StoredCalendar{
	Version:      1,
	ConnectionID: connectionID,
	AccountID:    accountID,
	Credentials:  credentials,
})
if err != nil {
	return err
}
```

Do not write a result from a failed encryption. The identity fields and version
are encrypted and authenticated together with the credentials. For the database
write, expose only the intended ciphertext column through a separate typed input:

```go
type CredentialUpdate struct {
	Credentials []byte `db:"credentials"`
}

input := CredentialUpdate{Credentials: sealed}
statement := query.Table("calendar_connections", func(statement *query.Builder) {
	statement.Where("id", connectionID)
	statement.Where("account_id", accountID)
	statement.Update(query.Values(input))
})
affected, err := db.Exec(ctx, statement)
```

Check the write error and affected-row count before reporting success. No
matching row is not a successful credential replacement. The database `db` tags
control column mapping; the payload's `json` tags control the encrypted JSON.
Do not persist the plaintext alongside the ciphertext or include it in query logs.

## Binding Records

A purpose string such as `calendar-credentials:v1` in
[driver AAD](https://arkoren.dev/docs/guides/encryption/drivers.md#binding-context) prevents using that ciphertext under a
different purpose. It does not stop copying ciphertext between two calendar
connections that share the same driver context. AAD is fixed at driver
construction; the facade has no per-operation AAD argument.

To reject record substitution with the payload above, compare its authenticated
identity against the independently established account and requested connection.
With standard-library `errors` imported, check both decryption and the embedded
metadata before using a credential:

```go
payload, err := enc.Decrypt[StoredCalendar](stored)
if err != nil {
	return err
}
if payload.Version != 1 {
	return errors.New("unsupported calendar payload version")
}
if payload.ConnectionID != connectionID || payload.AccountID != accountID {
	return errors.New("calendar payload belongs to another connection")
}
```

Here `stored` is the selected row's ciphertext. Do not take the expected identity
from the decrypted payload itself or trust an attacker-supplied account ID.
Cleartext columns and external version headers are not automatically
authenticated by encryption. Put security-relevant metadata inside the encrypted
payload or bind it through the exact configured AAD, then verify its meaning.
Neither approach replaces access authorization or prevents replay of an older
valid payload for the same record.

## Encoding Text

Where a transport accepts only text, encode the complete ciphertext. With
standard-library `encoding/base64`, URL-safe unpadded encoding is suitable for
an application token:

```go
token := base64.RawURLEncoding.EncodeToString(sealed)
```

Encoding adds roughly one third to the length; it is not another encryption
layer. Keep cookie payloads small enough for browser limits and account for
headers and other cookie fields. Avoid putting credentials in URLs, where
history, referrers, and access logs can expose replayable values.

Decode using the matching encoding before decrypting:

```go
stored, err := base64.RawURLEncoding.DecodeString(token)
if err != nil {
	return err
}
```

A base64 error is an encoding failure, not `crypto.ErrInvalidCiphertext`.
Successful decoding says nothing about authenticity; decryption must still
succeed. Apply transport size limits before allocating or decoding untrusted
input. Never use base64 to justify logging a key, plaintext, or ciphertext token.

## Searching Values

Random nonces mean encrypting a search term again does not reproduce stored
ciphertext. Query by an ordinary identifier whenever possible. If equality
lookup over encrypted data is necessary, an application-owned indexed keyed
hash can accompany the ciphertext, but reveals which rows share a value.

For an application that deliberately treats contact addresses case-insensitively,
this helper uses standard-library `crypto/hmac`, `crypto/sha256`, and `strings`:

```go
func contactLookup(secret []byte, email string) []byte {
	normalized := strings.ToLower(strings.TrimSpace(email))
	mac := hmac.New(sha256.New, secret)
	mac.Write([]byte(normalized))

	return mac.Sum(nil)
}
```

Choose normalization according to your account policy and apply it identically
on writes and searches. Use an independent protected lookup key, not the
encryption key. Store the result in an indexed binary column such as
`contacts.email_lookup`, updating it consistently with the encrypted address.
A plain unkeyed hash permits guessing common addresses from a database copy.

With validated `email`, protected bytes `lookupKey`, and your application row
type `Contact`, restrict the lookup to the authorized account:

```go
contact, err := db.Find[Contact](ctx, query.Table("contacts", func(statement *query.Builder) {
	statement.Where("account_id", accountID)
	statement.Where("email_lookup", contactLookup(lookupKey, email))
}))
```

The hash supports equality, not arbitrary encrypted search. Add it only when
the lookup is needed, and plan a separate index migration if its key or
normalization changes. Arkoren does not generate or rotate this index for you.

## Re-encrypting Values

[Key rotation](https://arkoren.dev/docs/guides/encryption/keys.md#rotating-keys) only changes future encryption. To replace
old stored ciphertext, configure current and previous drivers and ask whether
a previous driver authenticated the value. For an authorized loaded row
`connection`, read its `Credentials []byte` field:

```go
payload, previous, err := enc.DecryptWithRotation[StoredCalendar](connection.Credentials)
if err != nil {
	return err
}
```

Check `err` first: `previous` can be true even if later JSON decoding fails.
Apply the version and identity checks from [Binding Records](#binding-records)
before using or rewriting `payload`. If `previous` is false, no key replacement
is needed. Otherwise encrypt the verified payload with the current driver:

```go
fresh, err := enc.Encrypt(payload)
if err != nil {
	return err
}
```

Do not overwrite a credential changed by another request. Use the original
ciphertext as a compare-and-swap condition alongside the authorized target.
The update reuses `CredentialUpdate` from the write example:

```go
input := CredentialUpdate{Credentials: fresh}
affected, err := db.Exec(ctx, query.Table("calendar_connections", func(statement *query.Builder) {
	statement.Where("id", connection.ID)
	statement.Where("account_id", accountID)
	statement.Where("credentials", connection.Credentials)
	statement.Update(query.Values(input))
}))
if err != nil {
	return err
}
if affected == 0 {
	return errors.New("calendar credentials changed; reload before rotating")
}
```

These replacement steps run only for a successfully decoded, identity-checked
payload with `previous == true`. A failed write leaves rotation incomplete.
Alternatively, lock the row within your [transaction](https://arkoren.dev/docs/guides/database/transactions.md)
before reading and replacing it.

Read-time replacement gradually migrates active rows. To finish, run a bounded
application-owned batch job over every relevant row, handle conflicts by
reloading, and record completion without logging payloads. Already-migrated
rows report `previous == false`, so the key-replacement check is repeatable.
Do not retire old keys until no live writer or stored value needs them, and
retain keys required by backups.

For opaque byte payloads, `DecryptRawWithRotation` and `EncryptRaw` preserve
the byte format without JSON decoding. Application format validation still
belongs before any replacement.

## Keeping Compatibility

Persist complete ciphertext bytes; do not split or reinterpret the nonce and
tag in application code. The raw format carries no cipher, key identifier, AAD,
or schema version. Keep that configuration recoverable for old values and
backups. Changing AAD requires the original context on the old driver and the
new context on the new driver; changing a purpose string without retaining
the old decoder makes existing values fail authentication.

The version inside `StoredCalendar` is an application field, not an Arkoren
version protocol. Reject unsupported versions and preserve the original bytes.
If incompatible payload shapes need separate decoders, define an explicit
application format and validate its discriminator rather than guessing from
decryption failures. Unknown external discriminators must not select arbitrary
keys or cause plaintext fallback.

A successful JSON decode does not prove that a schema change preserved meaning:
renamed or absent fields can still produce zero values. Validate required fields
and version semantics before use or re-encryption. Key replacement and schema
migration are separate decisions; verify both before rewriting durable data.

For typed and raw rotation operations, see the [`crypto` API reference](https://pkg.go.dev/arkoren.dev/core/service/crypto).

---

Source: https://arkoren.dev/docs/guides/secrets/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/secrets/overview)

# Secrets Overview

A service needs credentials such as a reporting API token without putting them
in source code or its container image. A **secret store** holds those private
values outside the application; `secret.Store` reads them through your chosen
[driver](https://arkoren.dev/docs/guides/secrets/drivers.md). Provider permissions, retention, and auditing remain part
of that provider's configuration.

Reading a secret produces an ordinary Go value. The facade does not cache it,
refresh a running client, or revoke the credential in the system that accepts it.
[Rotation](https://arkoren.dev/docs/guides/secrets/rotation.md) coordinates those separate changes.

## Reading Values

With a configured `*secret.Store` named `secrets` and an operation context `ctx`,
read a provisioned text credential:

```go
token, err := secrets.String(ctx, "reporting-token")
```

On success, `token` contains the current value. `String` converts bytes to a Go
string without validating UTF-8. For binary material, use `Raw` instead; its
returned byte slice belongs to you. Azure accepts only UTF-8 text and Vault
stores JSON objects, so choose the [payload format](https://arkoren.dev/docs/guides/secrets/drivers.md#compatibility-matrix)
before provisioning.

A missing name or requested version wraps `secret.ErrNotFound`. Permission,
authentication, and connectivity failures are not missing-secret signals.
Use `errors.Is` for portable error checks rather than matching provider text.
An already-canceled context fails with its context error without changing storage.

Focused examples keep error results visible but omit routine propagation.
Handle errors before using results; the startup example below retains the
check that prevents using a missing credential.

## Typed Credentials

When one credential contains related fields, store a JSON object and share a
definition that binds its provider name to a Go type. Import `secret` from
`arkoren.dev/core/service/secret`:

```go
type ReportingCredentials struct {
	Account string `json:"account"`
	Token   string `json:"token"`
}

var reportingSecret = secret.NewSecret[ReportingCredentials]("reporting-client")
```

The definition performs no I/O and creates no provider resource. After
[provisioning](https://arkoren.dev/docs/guides/secrets/managing.md#writing-values) that JSON secret, read it through
the facade:

```go
credentials, err := secrets.Get(ctx, reportingSecret)
```

On success, `credentials` has type `ReportingCredentials`. Validate required
fields before using them: a shared definition is not a provider schema, and
independent definitions can assign different types to the same name.

Definitions are immutable and safe to share. An empty name or zero definition
reports `secret.ErrInvalidSecret`; `Err()` checks this without a read. `Get`
rejects invalid definitions before provider I/O, with an already-canceled context
taking precedence. JSON decoding errors can include the name and underlying
details; neither decoded values nor returned errors are redacted.

## Shared Names

For several JSON credentials of the same type, a namespace composes their names.
Choose a separator valid for your provider rather than assuming slash paths work
everywhere:

```go
reports := secret.NewNamespace[ReportingCredentials]("reports", "-")
archiveSecret := reports.Secret("archive")
credentials, err := secrets.Get(ctx, archiveSecret)
```

This reads the literal name `reports-archive`; it does not create a namespace in
the provider or grant access. Azure needs flat names. Google accepts a full
resource path or an unqualified ID with a configured project; slashes in a full
resource path are not valid separators inside that ID. AWS and Vault permit
path-style conventions. See [Drivers](https://arkoren.dev/docs/guides/secrets/drivers.md) before choosing deployed names.

`Derive` publishes an independent variant. `Subnamespace` joins a child prefix
with the parent's separator before applying settings for future joins:

```go
regional := reports.Subnamespace("eu")
replicaSecret := regional.Secret("replica")
```

The resulting name is `reports-eu-replica`. No escaping or normalization occurs,
and an empty separator concatenates components. Validate or unambiguously encode
untrusted components to prevent collisions. Namespaces are not authorization or
isolation boundaries.

Empty prefixes, empty child names, and zero namespaces are invalid; descendants
retain the first declaration error. `Name`, `Separator`, and `Err` inspect a
namespace. Construction callbacks run once in order, including on invalid
declarations; nil callbacks panic, and retained drafts cannot change published
values.

## Startup Loading

If a client uses its credential only during construction, load and validate the
credential before starting that client. A required-secret failure should stop
startup rather than silently select an empty value or generate a replacement.
For a safe application-owned error, import the standard-library `errors` package:

```go
token, err := secrets.String(ctx, "reporting-token")
if err != nil {
	return errors.New("reporting credential could not be loaded")
}
if token == "" {
	return errors.New("reporting credential is empty")
}
```

Only the success path passes `token` to the reporting client. The generic error
avoids exposing the provider error or secret name through ordinary startup logs;
use a protected diagnostic path if you need the underlying cause. A successful
read checks storage access, not whether the reporting API accepts that token.

Use [environment settings](https://arkoren.dev/docs/guides/first-steps/configuration.md#environment-settings)
for nonsecret choices such as the deployment's project or vault address. Grant
each runtime identity access only to the secrets it needs. Keeping store reads
out of request handling removes those reads from the request path, but a store
outage can still prevent startup or [refresh](https://arkoren.dev/docs/guides/secrets/rotation.md#runtime-refresh).

## Private Values

Treat secret names as potentially private as well as their contents. Typed
definitions and namespaces redact identities under ordinary formatting and
structured logging, but `Name()` deliberately exposes them. Raw `Reference` and
`Value` records, strings, decoded structs, native objects, and returned errors do
not inherit that protection.

Do not log credentials or credential-bearing connection strings, include them
in public URLs, or return them in API responses. Limit which application objects
retain them. Driver diagnostic records mask selected fields only; consult the
[provider logging rules](https://arkoren.dev/docs/guides/secrets/drivers.md), particularly when supplying native clients.

Drivers copy input and returned bytes. Clearing your returned slice does not
clear storage, other copies, strings, or decoded fields; copying a `secret.Value`
record alone still shares its `Data` slice. Arkoren does not guarantee secret
zeroization. Deleting a secret does not erase values already held by consumers.

For all definitions and operations, see the
[`secret` API reference](https://pkg.go.dev/arkoren.dev/core/service/secret).

---

Source: https://arkoren.dev/docs/guides/secrets/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/secrets/drivers)

# Secrets Drivers

Use memory for fixture credentials in tests and local development. For a deployed
service, choose the cloud store your organization operates, or Vault when you
already manage its KV v2 engine and authentication. Configure the workload's
identity and least-privilege policy explicitly: a default credential chain can
find credentials, but it does not prove they are the intended identity or grant
secret access.

The examples assume your application module already depends on
`arkoren.dev/core`. `ctx` is a startup or operation context. Check construction
errors before wrapping a driver; `secret.New` neither validates nor starts it.
Use a successful [credential read](https://arkoren.dev/docs/guides/secrets/overview.md#startup-loading) to check access,
then separately verify the credential with the system that accepts it.

## Compatibility Matrix

This comparison covers Arkoren's secret contract, not the provider's whole
administration API. Versioned reads select a retained concrete version within
one logical secret; they do not reserve its name or retain history.

| Driver | Versioned Reads | Arbitrary Bytes | Shared Storage | Permanent Delete | Native Access | Observed Mutations |
| --- | --- | --- | --- | --- | --- | --- |
| [Memory](#memory) | Yes | Yes | No | Yes | No | Yes |
| [AWS Secrets Manager](#aws-secrets-manager) | Yes | Yes | Yes | No — Schedules deletion with a recovery window. | Yes | Yes |
| [Azure Key Vault](#azure-key-vault) | Yes | No — Requires valid UTF-8 text. | Yes | No — Follows soft-delete retention. | Yes | Yes |
| [Google Cloud Secret Manager](#google-cloud-secret-manager) | Yes | Yes | Yes | Yes | Yes | Yes |
| [HashiCorp Vault](#hashicorp-vault) | Yes — Explicit selectors must be positive decimal versions. | No — Requires JSON objects and rejects numbers changed by native float64 JSON encoding. | Yes | Yes — Deletes metadata and all version history. | Yes | Yes |

Shared storage means processes address the same provider resource and have
permission to access it; memory is private to one process. Permanent deletion
describes removal through the contract, not erasure of provider backups or
application copies. Never use deletion as a rotation step.

Observed mutations are the facade's payload-free telemetry and optional
[notifications](https://arkoren.dev/docs/guides/secrets/managing.md#mutation-outcomes). Native access and raw driver calls
bypass them. All drivers support the facade's typed JSON definitions and literal
namespaces, subject to their own naming and payload rules. A namespace does not
create provider resources or permissions.

## Memory

The bundled `memorysecret` driver keeps a version history in process memory.
Use synthetic values only: it is neither durable storage nor a protected vault.

### Connecting

Seed a reporting credential for an isolated test:

```go
import (
	"arkoren.dev/core/service/secret"
	"arkoren.dev/core/service/secret/memorysecret"
)

driver := memorysecret.New(func(fixtures *memorysecret.Builder) {
	fixtures.Values(map[string][]byte{
		"reporting-client": []byte(`{"account":"local","token":"fixture"}`),
	})
})
secrets := secret.New(driver)
```

Each seed starts at version `"1"`. `Values` replaces and copies the map and
byte slices; nil or empty clears the seeds, and omitting it starts empty. A
retained construction draft cannot change the published store. There are no
start/stop hooks, native clients, timeout settings, or retry settings.

Use the [typed credential read](https://arkoren.dev/docs/guides/secrets/overview.md#typed-credentials) with this fixture.
The returned fields are `local` and `fixture`; do not print real credentials in
an equivalent production read.

### Storage Limits

Operations are concurrency-safe and copy payload bytes at their boundaries.
Already-canceled contexts change nothing; missing names or versions wrap
`secret.ErrNotFound`. Process exit loses all history. Deletion removes a name's
history immediately, and recreating it starts again at version `"1"`; an old
name/version selector can therefore select different contents.

See the [`memorysecret` API reference](https://pkg.go.dev/arkoren.dev/core/service/secret/memorysecret).

## AWS Secrets Manager

Choose AWS Secrets Manager for workloads with an explicitly configured AWS
identity. Runtime reads need `secretsmanager:GetSecretValue` on the required
resources; customer-managed encryption keys can require KMS permissions too.
Keep create, write, and delete permissions on provisioning identities.

### Connecting

From your application module, install the driver:

```sh
go get arkoren.dev/driver/aws
```

The AWS SDK discovers credentials and region through its standard sources,
including environment, shared configuration, and workload roles. Override the
region when the deployment does not supply it:

```go
import (
	"arkoren.dev/core/service/secret"
	"arkoren.dev/driver/aws/awssecret"
)

driver, err := awssecret.New(ctx, func(provider *awssecret.Builder) {
	provider.Region("eu-west-1")
})
secrets := secret.New(driver)
```

Construction loads SDK configuration; it does not test secret access. No driver
lifecycle hooks are required. Use the [startup read](https://arkoren.dev/docs/guides/secrets/overview.md#startup-loading)
with a provisioned name such as `reporting-token`.

### Provider Settings

`Credentials` supplies a shared credential provider instead of the default
chain; `Endpoint` overrides the service URL. Bound calls with operation contexts.
For transport, retries, and timeouts beyond those settings, configure a native
SDK client. The deprecated `Name` builder method is a compatibility no-op; it
does not select a default secret or namespace.

The driver accepts AWS secret names or supported identifiers. Reads accept text
or binary values; writes through Arkoren use binary payloads. Explicit
`Reference.Version` is an AWS `VersionId`, not a movable staging label. Missing
and pending-deletion secrets map to `secret.ErrNotFound`.

`Delete` schedules deletion using AWS's recovery window and does not immediately
free the name. A later same-name resource can have a different ARN. Keep the
logical secret and requested version retained for [pinned reads](https://arkoren.dev/docs/guides/secrets/managing.md#version-lifetimes).

### Native Access

If your application already owns an `aws.Config` named `config`, wrap its client:

```go
import "github.com/aws/aws-sdk-go-v2/service/secretsmanager"

driver := awssecret.NewFrom(secretsmanager.NewFromConfig(config))
native := driver.Client()
```

`native` provides SDK operations such as describing secrets. The driver does not
copy or close it. `Client()` returns nil for a nonnative test double. Native
operations bypass contract error mapping and facade observations.

The reporting-only `awssecret.Config` masks credential-provider contents by
type without retrieving credentials or invoking their formatter. Its endpoint
and name text, native objects, and returned errors are not sanitized. Keep
credentials out of endpoint URLs and ordinary diagnostics.

### Local Checks

With Docker, run an isolated MiniStack instance bound only to loopback. These
credentials are synthetic and must never be used as production credentials:

```sh
docker run --rm --name arkoren-secrets-aws -p 127.0.0.1:4566:4566 -e AWS_ACCESS_KEY_ID=000000000000 -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 ministackorg/ministack:1.5.8
```

After readiness, use this alternative construction in your local application:

```go
import "github.com/aws/aws-sdk-go-v2/credentials"

driver, err := awssecret.New(ctx, func(provider *awssecret.Builder) {
	provider.Region("us-east-1")
	provider.Endpoint("http://localhost:4566")
	provider.Credentials(credentials.NewStaticCredentialsProvider("000000000000", "test", ""))
})
secrets := secret.New(driver)
```

Run the disposable [version check](https://arkoren.dev/docs/guides/secrets/managing.md#writing-values) with this facade.
It verifies a synthetic write and pinned read, not AWS IAM permissions. Stop the
container with `docker stop arkoren-secrets-aws`; `--rm` discards its data.

See the [`awssecret` API reference](https://pkg.go.dev/arkoren.dev/driver/aws/awssecret).

## Azure Key Vault

Choose Key Vault when you have a vault and an Azure identity with secret access,
such as the Key Vault Secrets User role for runtime reads. Grant provisioning
permissions separately; default credential discovery is not an access check.

### Connecting

Install the driver from your application module:

```sh
go get arkoren.dev/driver/azure
```

Supply the vault's absolute HTTPS URL. The driver uses Azure's default credential
chain, including supported developer and managed identities:

```go
import (
	"arkoren.dev/core/service/secret"
	"arkoren.dev/driver/azure/azuresecret"
)

driver, err := azuresecret.New(ctx, "https://my-vault.vault.azure.net/")
secrets := secret.New(driver)
```

Construction validates the URL and creates SDK objects without contacting the
vault. There are no builder settings or driver lifecycle hooks. Use a
[startup read](https://arkoren.dev/docs/guides/secrets/overview.md#startup-loading) to verify access to the provisioned
name `reporting-token`.

### Provider Limits

Names are flat and allow letters, digits, and hyphens, not slash paths. Values
must be valid UTF-8; invalid bytes fail locally before a native write. Encode
binary material, for example as base64, and decode it after reading. Missing
names and versions wrap `secret.ErrNotFound`.

`Update` first reads for existence and then writes, so it requires read and write
permission. It has no native version precondition and is not an atomic
compare-and-set operation. Already-canceled operations fail before local UTF-8
or SDK parameter validation without a provider request.

`Delete` follows the vault's soft-delete retention. Writing a soft-deleted name
wraps `azuresecret.ErrSoftDeleted`; recovery restores the retained resource.
Do not automatically purge it to make provisioning succeed: purge is irreversible
and can be blocked by purge protection. Each value write creates a new version;
metadata and availability can still change independently.

### Native Access

For custom credentials, transport, retry, or timeout policy, use an existing
`azcore.TokenCredential` named `credential` and your HTTPS `vaultURL`:

```go
import "github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets"

native, err := azsecrets.NewClient(vaultURL, credential, nil)
driver := azuresecret.NewFrom(native)
```

Replace nil with SDK client options when needed. `driver.Client()` returns the
shared client for recovery and other administration outside the contract; the
driver neither copies nor closes it. Keep SDK body logging and native error
details out of ordinary diagnostics. No offline emulator setup is provided by
this driver; use memory for fixture tests.

See the [`azuresecret` API reference](https://pkg.go.dev/arkoren.dev/driver/azure/azuresecret).

## Google Cloud Secret Manager

Choose Secret Manager when your project has its API enabled and the runtime
identity has Secret Manager Secret Accessor permission on the required secrets.
Creating resources, adding versions, and deleting require separate permissions.

### Connecting

Install the driver from your application module:

```sh
go get arkoren.dev/driver/gcp
```

The SDK uses Application Default Credentials. Configure the project for short
names such as `reporting-token`:

```go
import (
	"arkoren.dev/core/service/secret"
	"arkoren.dev/driver/gcp/gcpsecret"
)

driver, err := gcpsecret.New(ctx, func(provider *gcpsecret.Builder) {
	provider.Project("my-project")
})
secrets := secret.New(driver)
```

Register `driver.Stop` with [application cleanup](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md)
immediately after successful construction. It owns the native gRPC connection;
stop consumers before releasing it. The construction context controls SDK
construction and credential discovery, not future calls. Verify permissions with
the [startup read](https://arkoren.dev/docs/guides/secrets/overview.md#startup-loading).

`New` installs a discard logger for SDK request/response and credential logging,
even if `GOOGLE_SDK_GO_LOGGING_LEVEL=debug` is set. This does not disable the
facade's separate payload-free observations.

### Provider Settings

`Project` defaults to empty and is not inferred from credentials. Either set it
for unqualified secret IDs or use full names such as
`projects/my-project/secrets/reporting-token`. Fully qualified names keep their
explicit project. Slashes belong to that resource path, not inside a secret ID.
Use operation contexts to bound calls; native client settings own retry and
timeout policy.

`Put` creates an absent secret with automatic replication and then adds a
version. A failed version write can leave an empty resource. `Update` only adds
a version to an existing resource. Empty version selects `latest`; concrete
versions returned by the driver are strings such as `"1"`, not resource paths.
Missing resources or versions wrap `secret.ErrNotFound`.

`Delete` removes the resource and its versions. Recreation can reuse numeric
version selectors; a movable alias such as `latest` never pins contents.

### Native Access

If you configure SDK endpoint, credentials, retries, or transport yourself,
`NewFrom` preserves that client's logging policy.

> [!WARNING]
> SDK debug logging can include secret payloads, including when enabled by
> `GOOGLE_SDK_GO_LOGGING_LEVEL`. Configure a discard logger before supplying a
> native client, after other client options so that it takes precedence.

With existing `options` of type `[]option.ClientOption`, construct the client:

```go
import (
	"log/slog"

	secretmanager "cloud.google.com/go/secretmanager/apiv1"
	"google.golang.org/api/option"
)

clientOptions := append(options, option.WithLogger(slog.New(slog.DiscardHandler)))
native, err := secretmanager.NewClient(ctx, clientOptions...)
```

After checking the error, wrap it with the naming policy:

```go
driver := gcpsecret.NewFrom(native, func(provider *gcpsecret.Builder) {
	provider.Project("my-project")
})
```

`driver.Client()` exposes native version administration. `Stop` closes supplied
clients too: keep one cleanup owner and do not share the client with consumers
that outlive this driver. Native close has no context parameter, so an expired
Stop context still initiates release but cannot bound SDK connection/resolver
teardown. Use memory for offline fixtures; this driver supplies no emulator
setup.

See the [`gcpsecret` API reference](https://pkg.go.dev/arkoren.dev/driver/gcp/gcpsecret).

## HashiCorp Vault

Choose Vault when you operate a KV v2 mount and an authentication policy for your
application. Runtime readers need access to its data paths; provisioning tools
need appropriate write and metadata permissions too. This driver does not use
Vault's dynamic-secret engines.

### Connecting

Install the driver from your application module:

```sh
go get arkoren.dev/driver/vault
```

For managed login, supply an existing Vault SDK `api.AuthMethod` named
`loginMethod`. For example, the separate
[Kubernetes auth SDK](https://pkg.go.dev/github.com/hashicorp/vault/api/auth/kubernetes)
requires a Vault role bound to the workload's service account and namespace.
Configure that role and method before constructing the driver:

```go
import (
	"arkoren.dev/core/service/secret"
	"arkoren.dev/driver/vault/vaultsecret"
)

driver, err := vaultsecret.New(func(provider *vaultsecret.Builder) {
	provider.Address("https://vault.internal:8200")
	provider.Auth(loginMethod)
})
secrets := secret.New(driver)
```

Construction performs SDK validation but no login. Register `driver.Stop` as
cleanup and `driver.Start` as startup through the
[application lifecycle](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md). Register
credential loading after Start succeeds. With managed authentication, operations
before successful Start or after Stop return `vaultsecret.ErrNotStarted`.

### Authentication Lifetime

`Mount` defaults to `secret`; `Address` follows SDK defaults including
`VAULT_ADDR`. The SDK can load `VAULT_TOKEN`; a nonempty `Token` setting overrides
it. Static tokens are never renewed by the driver. With `Auth`, successful Start
replaces the token with the login result.

Renewable login tokens are renewed in the background. When renewal ends, the
driver logs in again, retrying with exponential waits from one second to one
minute. A nonrenewable login token is not automatically refreshed. Repeated
Start while ready is a no-op; malformed native renewability fails Start.
`Logger` defaults to `slog.Default()` and reports failed stages and retry delays,
not native diagnostics, tokens, or response bodies. Returned errors are not
sanitized.

Start and Stop serialize. Call Stop after Start returns; it cannot cancel a
blocked authentication callback. Stop signals managed renewal and waits for its
owner and native watcher within the caller's wait budget. An expired context
still signals cleanup, but an in-flight native renewal RPC depends on SDK
timeouts and transport cooperation. Ownership remains tracked after an early
return; restart is rejected until that worker drains, and a later Stop can wait
again. The native HTTP transport is not closed by the driver.

### JSON Credentials

Vault requires a non-null JSON object. With a provisioned `reporting-client`
object, use the [typed read](https://arkoren.dev/docs/guides/secrets/overview.md#typed-credentials). `String` would return
the entire JSON object as text, not its `token` field. Names are KV v2 paths
relative to the selected mount; do not add the mount's API envelope yourself.

Reads re-encode objects, so whitespace and member order can change. Writes
reject numbers whose value changes under native float64 JSON serialization,
including nested numbers. Store exact identifiers or decimals as JSON strings;
the driver does not convert them for you. For example, a provisioning tool can
encode this input with `encoding/json/v2` before writing it:

```go
type PartnerCredential struct {
	AccountID string `json:"account_id"`
	Token     string `json:"token"`
}

input := PartnerCredential{AccountID: "9007199254740993", Token: "fixture"}
```

Accepted numeric tokens remain numbers and are sent unchanged; equivalent
notations such as `1e3` and `1000` are accepted. Existing data is not migrated,
and reads cannot recover precision lost by an earlier writer or backend.

### Version Limits

Explicit selectors must be positive decimal versions. Empty selects current;
zero, negative, or unparseable selectors wrap `secret.ErrNotFound`. Missing,
deleted, or destroyed payloads also wrap that sentinel. Already-canceled calls
fail before version/JSON validation and provider requests.

`Update` reads the current version and uses its number as a native CAS
precondition, requiring read and write access. Concurrent changes fail without a
driver-level retry; detected absence after CAS rejection wraps
`secret.ErrNotFound`. Metadata deletion and recreation can restart at version
`1`, so this is not a cross-recreation identity check. Native SDK retry policy
remains unchanged. `Delete` permanently deletes metadata and all history, not
just the latest value through Vault's soft-delete API.

### Native Access

If you own authentication, logging, transport, and token renewal, wrap your
existing `*api.Client` named `native` from `github.com/hashicorp/vault/api`:

```go
driver := vaultsecret.NewFrom(native.KVv2("application"))
versions := driver.Client()
```

`versions` is the shared native `*api.KVv2`, usable for version administration.
`Client()` is nil for nonnative doubles. This path has no managed login or
renewal; Start/Stop are no-ops, as they are for static-token drivers. You retain
transport cleanup. Native calls bypass payload validation, error mapping, and
facade observation. The reporting-only `vaultsecret.Config` masks its static
Token, not Address or a shared Auth formatter.

### Local Checks

Run an isolated development server with Docker. Its root token and HTTP
transport are intentionally insecure: bind loopback and use only synthetic
secrets.

```sh
docker run --rm --name arkoren-secrets-vault -p 127.0.0.1:8200:8200 hashicorp/vault server -dev -dev-root-token-id=dev -dev-listen-address=0.0.0.0:8200
```

After readiness, the server has a `secret` KV v2 mount. Use this alternative
construction for the disposable check:

```go
driver, err := vaultsecret.New(func(provider *vaultsecret.Builder) {
	provider.Address("http://localhost:8200")
	provider.Token("dev")
})
secrets := secret.New(driver)
```

Run the JSON [version check](https://arkoren.dev/docs/guides/secrets/managing.md#writing-values). A successful pinned
read checks this local mount, not production authentication or TLS policy. Stop
the server with `docker stop arkoren-secrets-vault`; `--rm` discards its data.

See the [`vaultsecret` API reference](https://pkg.go.dev/arkoren.dev/driver/vault/vaultsecret).

For shared reads and writes, see the
[`secret` API reference](https://pkg.go.dev/arkoren.dev/core/service/secret).

---

Source: https://arkoren.dev/docs/guides/secrets/managing.md
[HTML documentation](https://arkoren.dev/docs/guides/secrets/managing)

# Managing Secrets

A runtime service usually reads credentials provisioned elsewhere. Use writes
from an authorized provisioning command, administrative job, or isolated test,
not as a fallback when a required startup read fails. The `secret.Store` facade
exposes writes with payload-free observations; provider policy still determines
which identity can perform them.

These operations change secret storage. They do not create a credential in the
external system that checks it or move running consumers to its new value. Use
the [rotation procedure](https://arkoren.dev/docs/guides/secrets/rotation.md) for that coordinated change.

## Writing Values

Use a configured [store](https://arkoren.dev/docs/guides/secrets/drivers.md) named `secrets` and operation context `ctx`.
Choose names and payloads that satisfy its provider. `Put` creates a missing
logical secret or adds a new current version to an existing one; it is not
create-only and has no portable idempotency key.

> [!WARNING]
> A successful write can replace the credential readers select as current.
> Give write access only to provisioning identities, and run the fixture below
> only against an isolated, disposable store—not a production vault.

For a cross-provider fixture, encode a small JSON object. Import
`encoding/json/v2` and `crypto/rand` from the standard library:

```go
type FixtureCredential struct {
	Token string `json:"token"`
}

input := FixtureCredential{Token: "fixture"}
payload, err := json.Marshal(input)
```

After successful encoding, write it under a fresh name. Lowercase the random
suffix with standard-library `strings` so it also fits flat provider naming:

```go
name := "local-check-" + strings.ToLower(rand.Text())
written, err := secrets.Put(ctx, name, payload)
```

On success, `written.Version` is the provider-issued concrete version and
`written.Data` is a caller-owned payload copy. To verify that version, import
`secret` from `arkoren.dev/core/service/secret` and select it explicitly:

```go
read, err := secrets.Driver().Get(ctx, secret.Reference{
	Name:    name,
	Version: written.Version,
})
```

After a successful read, decode the returned JSON rather than comparing byte
formatting: Vault re-encodes object members. With the standard-library `errors`
package, the success check is:

```go
var actual FixtureCredential
if err := json.Unmarshal(read.Data, &actual); err != nil {
	return errors.New("fixture decoding failed")
}
if actual.Token != input.Token {
	return errors.New("fixture version check failed")
}
```

This verifies the stored fixture without printing it. Remove it with
`secrets.Delete(ctx, name)` after the check and handle cleanup errors. AWS and
Azure retain deleted resources according to their policies; stopping an isolated
[local container](https://arkoren.dev/docs/guides/secrets/drivers.md#local-checks) discards that container's data instead.
Use a fresh name for each run rather than relying on immediate reuse.

Input and returned byte slices are detached from storage. Clearing your copy
does not clear the stored payload or any other copy. A shallow copy of a
`secret.Value` still shares its `Data` slice; strings and decoded structs are
ordinary application values, not zeroizing containers.

## Updating Values

If a missing name indicates a provisioning mistake, use `Update` rather than
`Put`. It adds a version only to an existing secret. Here `name` is a validated,
provider-valid provisioning target and `replacement` is its encoded payload:

```go
updated, err := secrets.Update(ctx, name, replacement)
if errors.Is(err, secret.ErrNotFound) {
	return errors.New("required secret is missing; provision it separately")
}
```

On success, retain `updated.Version` if you need to verify that exact write.
On missing-secret failure, the operation creates nothing. Do not interpret
permission or authentication failures as absence and fall back to `Put`.
Other provider errors remain errors; returned diagnostics can expose names or
provider details and must not be logged blindly.

`Update` is not portable compare-and-set. Azure reads for existence, then writes
without a native version precondition. Vault reads a version and uses native
CAS, but deletion/recreation can reuse that number. Inspect the
[driver limits](https://arkoren.dev/docs/guides/secrets/drivers.md) before coordinating concurrent provisioning writers.

A failed remote write does not always mean nothing changed. For example, Google
can create a resource before adding its first version fails; a timeout can also
leave the final write outcome unknown. Reconcile through authorized provider
inspection or a protected read before deciding whether to retry. Another `Put`
or `Update` can create another version; the facade does not promise exactly-once
writes or roll back partial provider changes. Already-canceled contexts fail
without changing storage, but that guarantee does not describe cancellation
after a remote request begins.

## Version Lifetimes

A **concrete version** identifies an immutable payload within one existing
logical secret. Later writes do not edit that payload. Current-value reads
select again on each call; a `secret.Secret[T]` definition does not pin the value
that existed when the definition was declared.

To repeat work against one payload, use the returned version as shown above.
Keep the logical secret and version retained and readable until that work ends.
Saving a `secret.Reference` does not retain either. Raw driver reads bypass
facade telemetry and notifications, so own their observation policy separately.

Treat version strings as opaque: pass them back unchanged rather than parsing,
sorting, or synthesizing them. AWS uses `VersionId`, not a staging label; Google
aliases such as `latest` can move. Vault expects positive decimal selectors,
but applications should still retain its returned string rather than derive one.
A never-issued version wraps `secret.ErrNotFound`; provider retention, deletion,
or access policy can also make a previously readable version unavailable.

Permanent deletion and recreation starts a new logical secret, not a continuation
of its old history. Memory, Google, and Vault can reuse numeric selectors, so a
saved name/version pair can select different contents afterward. Provider
recovery of a retained resource is not recreation. Immutability does not cover
metadata, permissions, aliases, or permanent cross-recreation identity.

## Mutation Outcomes

To count provisioning results without recording credentials, configure your
existing telemetry service and event bus when constructing the facade. Here
`driver` is configured, `observability` is a started `*telemetry.Telemetry` from
`arkoren.dev/core/service/telemetry`, and `bus` is an `*event.Bus` from
`arkoren.dev/core/event`:

```go
secrets := secret.New(driver, func(store *secret.Builder) {
	store.Telemetry(observability)
	store.Events(bus)
})
```

Each facade `Put`, `Update`, or `Delete` records one provider-operation
observation. With telemetry enabled, spans are named `secret put`,
`secret update`, or `secret delete`; counters use `arkoren.secret.operations`,
and `arkoren.secret.operation.duration` measures seconds. The operation label is
`arkoren.secret.operation`.

Failures add only the bounded `error.type`: `secret.ErrNotFound`,
`context.Canceled`, `context.DeadlineExceeded`, or `other`. Names, versions,
values, sizes, and native error types/messages are omitted. Nil or omitted
telemetry follows the process default per call; a disabled default records
nothing. The facade does not start telemetry itself.

The optional bus receives `secret.MutationSucceeded` or
`secret.MutationFailed` after the driver returns. Each `secret.MutationEvent`
contains only `Operation` and `ErrorType`; success has an empty error class.
Delivery is synchronous and best effort. Listener errors or panics do not replace
the operation result. Nil events disables notifications.

These outcomes are observations, not an authoritative audit, durable event
delivery, global rotation notification, or proof that consumers adopted a value.
Raw driver/native calls and provider-managed rotation bypass them. A failed
mutation event does not prove absence of side effects.

Reads emit no mutation events. `Raw`, `String`, and typed `Get` each observe one
provider read, named `secret get`. JSON decoding happens afterward, so a decode
failure does not mark a successful provider read as failed. Returned errors
still retain details that the telemetry deliberately omits.

## Deleting Values

Deletion retires storage, not the credential in its issuing system. Revoke a
credential there when appropriate, and check that running consumers, retained
data, and backups no longer need its stored value before deleting it.

> [!WARNING]
> `Delete` makes the name and all versions unavailable through the driver and
> can permanently remove history. Rotate by adding a version, not by deleting
> the secret. A successful delete does not erase consumer copies or backups.

With an authorized retirement target `name`, delete the logical secret:

```go
err := secrets.Delete(ctx, name)
```

After success, reads of that name and its versions return `secret.ErrNotFound`.
Deleting a missing name returns the same sentinel. Recovery or recreation can
later make a name readable again, without guaranteeing the identity of saved
selectors across recreation.

Memory, Google, and Vault remove the secret/history through this contract. AWS
schedules deletion with its recovery window; Azure uses soft-delete retention.
Those retained names cannot be assumed immediately reusable. Review
[provider deletion behavior](https://arkoren.dev/docs/guides/secrets/drivers.md#compatibility-matrix) instead of using a
delete-and-recreate loop for tests or provisioning.

For all mutation and version-selection operations, see the
[`secret` API reference](https://pkg.go.dev/arkoren.dev/core/service/secret).

---

Source: https://arkoren.dev/docs/guides/secrets/rotation.md
[HTML documentation](https://arkoren.dev/docs/guides/secrets/rotation)

# Secret Rotation

A new secret-store version changes what the next current-value read selects.
It does not change an external credential, update an existing connection pool,
or make every application instance read again. **Rotation** coordinates creating
replacement material, moving consumers to it, and retiring the old material when
it is no longer needed.

Plan the overlap before writing: if old and new credentials can both work during
the rollout, consumers can switch independently. Not every issuer supports
overlap. If it cannot, use that system's supported cutover or maintenance
procedure rather than assuming Arkoren makes the change atomic or downtime-free.

## Credential Rollout

For a reporting API token or database credential, the issuer determines which
values are accepted. Use a separately authorized provisioning identity and keep
the existing logical secret's history intact:

1. Create a replacement at the issuer while the old credential remains valid,
   if supported—for example, a second API token or database user.
2. Validate the replacement against its intended permissions without logging it.
3. Write it as a [new secret version](https://arkoren.dev/docs/guides/secrets/managing.md#updating-values), using the
   payload shape your consumers expect.
4. Restart or refresh every consumer, including workers and scheduled jobs, and
   verify that the replacement actually works there.
5. Revoke the old credential only after consumers no longer depend on it and
   the issuer's overlap requirements are satisfied.

Track rollout state without recording values or private secret names. Provider
usage records or connection inventory can help establish that old credentials
are no longer used; a successful write or mutation notification cannot. Include
infrequent jobs and dormant instances in the verification, not just the web fleet.

If a write fails, reconcile its [possibly uncertain outcome](https://arkoren.dev/docs/guides/secrets/managing.md#updating-values)
before retrying. If consumers fail after publication, keep the old credential
valid while you investigate when the issuer permits it. Reading an old stored
version can recover its bytes only while retained and authorized; it cannot
reactivate a credential already revoked at the issuer.

After a compromise, the old credential may need immediate revocation even if
that interrupts service. Choose that incident policy explicitly; overlap extends
the time an attacker can use a leaked value. Do not delete the logical secret
as a substitute for revocation.

## Runtime Refresh

Startup-loaded credentials change when the consumer restarts. Prefer that path
when a client captures credentials during construction: a database pool, for
example, may need rebuilding and draining, not merely a new string in memory.
Read and validate credentials before building the replacement client, then
retire the old client according to its lifecycle.

For a credential read on each outgoing call, application-owned refresh can
publish a complete immutable snapshot. This is not a built-in secret-store
feature. The fragments below use the `ReportingCredentials` type and
`reportingSecret` definition from [Typed Credentials](https://arkoren.dev/docs/guides/secrets/overview.md#typed-credentials),
a configured `secrets` facade, and standard-library `errors` and `sync/atomic`.

Initialize the snapshot during startup, before any reader or refresh work runs:

```go
initial, err := secrets.Get(ctx, reportingSecret)
if err != nil || initial.Account == "" || initial.Token == "" {
	return errors.New("initial reporting credential is unavailable")
}

var active atomic.Pointer[ReportingCredentials]
active.Store(&initial)
```

The guarded success path publishes one validated credential. Keep `active` at
its application's owning scope and do not copy the atomic object after use.
Never modify a published credential through its pointer.

In each supervised refresh attempt, use a bounded operation context and replace
the snapshot only after both reading and application validation succeed:

```go
next, err := secrets.Get(ctx, reportingSecret)
if err != nil || next.Account == "" || next.Token == "" {
	return errors.New("reporting credential refresh failed")
}
active.Store(&next)
```

A failed attempt leaves the old snapshot unchanged. Schedule attempts through
your [managed background work](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md), stop
them on shutdown, and monitor failure counts and snapshot age. Choose the
interval and retry budget well inside the issuer's overlap window. Retaining
the last good snapshot preserves access only while that credential remains valid;
define when stale credentials must stop being used instead of retrying forever.

After successful startup initialization, a caller loads one snapshot for the
whole operation:

```go
snapshot := active.Load()
token := snapshot.Token
```

Use fields from that same snapshot if a request also needs `Account`; separate
loads could span a refresh. The pointer exchange is atomic in this process only.
It does not coordinate replicas, change in-flight requests, or prove the issuer
accepts the new value. Never call Load before initialization, and never print
`token` or the snapshot.

## Verification Secrets

When another system signs requests that your application verifies, overlap has
to exist in the verifier too. Follow the provider's signing format, timestamp
rules, replay policy, and supported rotation behavior. Do not replace a
provider-specific verifier with a generic string comparison.

If the protocol supports current and previous secrets together, you can keep
them in one JSON object so a read loads the pair from one stored version:

```go
type VerificationSecrets struct {
	Current  string `json:"current"`
	Previous string `json:"previous,omitempty"`
}

var verificationSecret = secret.NewSecret[VerificationSecrets]("partner-verification")
```

Here `secret` is imported from `arkoren.dev/core/service/secret`. Validate the
current field before publishing a decoded snapshot. Pass only nonempty values
to a verifier that explicitly supports overlap; this definition does not add
multi-key support to an existing webhook adapter. If your integration fixes its
secret at construction, rebuilding that verifier is part of adoption.

Keep old verification material until the sender has switched and the protocol's
allowed delivery/retry window has ended. Then remove it in a new version and
refresh consumers again. For application-owned HMAC verification, compare
computed MACs with `hmac.Equal`; use the provider's full verification procedure
for [incoming webhooks](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md), including replay defenses.

## Encryption Material

Encryption keys have a different retirement condition: data encrypted with an
old key still needs that key to decrypt. Publishing a new current key does not
rewrite stored ciphertext, and revoking an old key too soon can make retained
data or backups unreadable.

Provision replacement material explicitly using the
[key command workflow](https://arkoren.dev/docs/guides/cli/keys.md), then load the stored representation as
[typed key configuration](https://arkoren.dev/docs/guides/keys/overview.md#load-configuration). For shared
application keys, `key.ParseShared` from `arkoren.dev/foundation/key` parses the
encoded representation; encryption drivers consume `key.Shared`, not the
secret-store version string or unparsed text. Check read and parse errors before
constructing a driver. Never generate a replacement because startup loading
failed.

Follow [Encryption Keys](https://arkoren.dev/docs/guides/encryption/keys.md) for previous-key configuration
and [Storing Encrypted Data](https://arkoren.dev/docs/guides/encryption/storing.md) for re-encryption.
Retain old material until live records and retained backups no longer require
it, according to your recovery policy. A pinned [store version](https://arkoren.dev/docs/guides/secrets/managing.md#version-lifetimes)
is useful only while that logical secret and version remain readable; it does
not preserve material by itself.

For the storage operations used during rotation, see the
[`secret` API reference](https://pkg.go.dev/arkoren.dev/core/service/secret).

---

Source: https://arkoren.dev/docs/guides/database/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/database/overview)

# Overview

The database facade runs SQL statements and reads their results into Go values.
Your code works with a `*database.Database`; its driver handles the connection
and database-specific execution. The facade compiles each statement for that
driver's SQL dialect. It is not an ORM: you choose the columns and write the
queries rather than loading models with implicit relationships.

The examples assume an already configured `db` and an operation context `ctx`.
See [Drivers](https://arkoren.dev/docs/guides/database/drivers.md) for construction, connection settings, and lifecycle.
Use the request or job's context for each call so cancellation reaches the
database operation.

Import `database` from `arkoren.dev/core/service/database` and `query` from
`arkoren.dev/foundation/query`. The snippets keep error results visible but omit
routine propagation; handle errors before using results in your application.

## Reading Data

Suppose your application has a `users` table with an integer primary key `id`
and a non-null text column `email`. Tables must already exist; the facade does
not create them from Go types. Use [migrations](https://arkoren.dev/docs/guides/migrations/overview.md) to
manage the schema.

Define the fields you want to read. Each `db` tag names its SQL column:

```go
type User struct {
	ID    int64  `db:"id"`
	Email string `db:"email"`
}
```

To look up a user, describe the selected columns and filter by the primary key:

```go
lookup := query.Table("users", func(statement *query.Builder) {
	statement.Select("id", "email")
	statement.Where("id", 7)
})
```

This builds a statement without executing it. `Find` runs it and scans one row
into the type you specify:

```go
user, err := db.Find[User](ctx, lookup)
```

On success, `user` contains the matching ID and email. If several rows match,
`Find` reads the first; filtering by a unique key avoids an ambiguous result.

If nothing matches, the error matches `database.ErrNoRows`. Import the
standard-library `errors` package to distinguish absence from another failure.
For example, if your application defines `ErrUserNotFound`, translate it at the
domain boundary:

```go
if errors.Is(err, database.ErrNoRows) {
	return ErrUserNotFound
}
```

Other query and driver errors remain errors; do not treat them as a missing user.
`Find` can also read a single selected column into a scalar type, such as
`string` or `int64`, instead of a struct.

For several users, use `Select`. This query reads the first twenty users in ID
order:

```go
users, err := db.Select[User](ctx, query.Table("users", func(statement *query.Builder) {
	statement.Select("id", "email")
	statement.OrderBy("id")
	statement.Limit(20)
}))
```

`Select` returns a `collection.Slice[User]`; `users.Items()` exposes its items
as a Go slice. No matching rows means an empty collection, not `ErrNoRows`.
It loads the complete result into memory, so bound large results with
[pagination](https://arkoren.dev/docs/guides/queries/pagination.md) or use [batching and streaming](https://arkoren.dev/docs/guides/database/batching-and-streaming.md).

## Executing Statements

Suppose an HTTP endpoint accepts a new email address. Keep its input small and
specific to the operation:

```go
type UpdateEmail struct {
	Email string `json:"email" db:"email"`
}
```

Assume `input` is an `UpdateEmail` already decoded and validated by your handler.
`query.Values(input)` maps its fields to database columns using `db` tags, not
`json` tags. This input exposes only the email to update; the target ID stays
outside the request body.

Build an update with a filter that identifies the intended row. Without a
filter, an update can affect every row in the table:

```go
update := query.Table("users", func(statement *query.Builder) {
	statement.Where("id", 7)
	statement.Update(query.Values(input))
})
```

`Exec` executes a statement without scanning result rows:

```go
affected, err := db.Exec(ctx, update)
```

On success, `affected` is the count reported by the driver. An update with no
matching row is not a `Find` operation and does not return `ErrNoRows`. Do not
use a zero count alone as portable proof that a user is missing: databases can
differ in whether an unchanged value counts as an affected row.

The facade also executes insert and delete statements. See
[Query Builder](https://arkoren.dev/docs/guides/queries/query-builder.md) for constructing them, or
[Raw SQL](https://arkoren.dev/docs/guides/queries/raw-sql.md) when you need handwritten SQL. Values passed to
the builder are bound parameters; do not interpolate untrusted values into SQL.

## Data Mapping

Result columns map to exported fields by their `db` tags. Without a tag, the
field name is lowercased: `Email` maps to `email`, but `CreatedAt` maps to
`createdat`, not `created_at`. Use an explicit tag when those names differ.

Every returned column needs a destination field. An unmatched column fails with
an error matching `query.ErrUnknownColumn`; fields not selected remain at their
zero values in the newly returned struct. Selecting named columns, as above,
keeps your result shape explicit. Use `db:"-"` for a field that must not receive
a column value.

If `email` can be SQL `NULL`, replace the plain string with a nullable field.
For example, this alternative result type preserves the distinction between
a missing email and an empty one:

```go
type Contact struct {
	ID    int64   `db:"id"`
	Email *string `db:"email"`
}
```

Read it with `Find[Contact]` or `Select[Contact]`. A SQL `NULL` becomes a nil
pointer; an empty string becomes a non-nil pointer to `""`. The standard
library's `sql.NullString` is another option. Scanning SQL `NULL` into a plain
`string` fails rather than silently replacing it with an empty string.

For the complete facade contract, see the
[`database` API reference](https://pkg.go.dev/arkoren.dev/core/service/database).

---

Source: https://arkoren.dev/docs/guides/database/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/database/drivers)

# Drivers

Choose SQLite for an embedded database or isolated tests. Choose PostgreSQL or
MySQL when several application instances need a shared database server. All
three implement the database contract; their SQL dialects and coordination
boundaries still matter. There is no simulated memory driver: SQLite's memory
mode runs a real SQLite engine.

## Compatibility matrix

This compares the shipped Arkoren drivers and query compiler, not every feature
of each database product. Named locks coordinate cooperating callers, not all
SQL writers. Persistence assumes a persistent database and suitable server or
filesystem configuration; the driver does not supply replication or backups.

| Driver | Persistent Storage | Nested Transactions | Transactional DDL | Row Locks | Cross-Process Named Locks | RETURNING | Upserts |
| --- | --- | --- | --- | --- | --- | --- | --- |
| [SQLite](#sqlite) | Partial — File-backed only; memory contents end with the last connection. | Yes — Savepoints. | Yes | No | Partial — Same-host file locks on supported Unix systems; not memory or network filesystems. | Yes | Yes |
| [PostgreSQL](#postgresql) | Yes | Yes — Savepoints. | Yes | Yes | Yes — Within one database; requires direct or session-pooled connections. | Yes | Yes |
| [MySQL](#mysql) | Yes | Partial — Savepoints; DDL can end them implicitly. | No | Partial — Query locking clauses target MySQL 8. | Yes — Within one server. | No | Partial — Uses native duplicate-key update or INSERT IGNORE semantics. |

All three support typed reads, writes, cursors, retained sessions, and named
locks. [Pagination](https://arkoren.dev/docs/guides/queries/pagination.md) and
[batching](https://arkoren.dev/docs/guides/database/batching-and-streaming.md) operate through the same facade. Read
[Locking](https://arkoren.dev/docs/guides/database/locking.md) for lifetime rules and
[migration deployment](https://arkoren.dev/docs/guides/migrations/deployment.md) before relying on DDL rollback.

## SQLite

SQLite runs in your process without a database server. Install its driver from
your application's module directory:

```sh
go get arkoren.dev/driver/sqlite
```

### Connecting

Import `arkoren.dev/driver/sqlite/sqlitedatabase` and
`arkoren.dev/core/service/database`. For a local application file:

```go
driver, err := sqlitedatabase.New("app.db")
db := database.New(driver)
```

Handle the construction error before using the driver. Construction validates
the required, nonblank DSN and nonnegative pool limits but opens the database
lazily. With your startup context, verify the connection before accepting work:

```go
err := driver.Start(ctx)
```

`Start` is a connectivity check, equivalent to `Ping`, not a background worker.
After successful startup, use the shared [typed read and write examples](https://arkoren.dev/docs/guides/database/overview.md).
Constructors do not create application tables; use
[migrations](https://arkoren.dev/docs/guides/migrations/overview.md).

When work has drained and all callbacks and cursors have finished, release the
owned pool with `driver.Stop(cleanupCtx)`. SQLite's stop ignores that context
and closes through `database/sql`. In an Arkoren application, register startup
and cleanup using [application composition](https://arkoren.dev/docs/guides/application/composition.md).

### Memory Databases

For a disposable test database, use an alternative construction:

```go
driver, err := sqlitedatabase.New(":memory:")
```

`New` defaults recognized memory DSNs to one open connection. A private memory
database belongs to a connection, so overriding that default with several
connections can produce independent databases with different tables. Closing
the last connection destroys the database, including pool expiry. Avoid
connection lifetime/idle expiry when the memory database must survive.

Memory databases have only process-local named locks. File-backed databases on
Linux, macOS, and BSD also use sibling lock files for other processes on the
same host. Other operating systems and network filesystems have no supported
cross-process lock guarantee. SQLite is not a horizontally scaled database.

### Connection Settings

For a file database, bound connections explicitly when needed:

```go
driver, err := sqlitedatabase.New("app.db", func(connection *sqlitedatabase.Builder) {
	connection.MaxOpenConns(4)
	connection.MaxIdleConns(2)
})
```

Positive limits replace native settings. Zero leaves them unchanged except
`New`'s memory default. `ConnMaxLifetime` and `ConnMaxIdleTime` optionally bound
connection reuse; zero does not disable an existing supplied-pool limit.
Negative settings are rejected by `New`.

`New` adds `_time_format=sqlite` unless the DSN explicitly sets it. DATE,
DATETIME, and TIMESTAMP values round-trip using that format, but SQLite compares
timestamps as text: write UTC consistently for correct ordering and comparisons.
Configure per-connection database policies through the native DSN where needed;
a one-off PRAGMA on the root pool does not configure every connection.

For a file-backed application that needs foreign-key enforcement and a bounded
wait for concurrent writers, set native pragmas in the DSN:

```go
dsn := "file:app.db?_pragma=foreign_keys(1)&_pragma=busy_timeout(5000)"
driver, err := sqlitedatabase.New(dsn)
```

SQLite permits one writer at a time. A busy timeout lets competing writes wait
up to 5,000 milliseconds rather than immediately returning SQLITE_BUSY.
Foreign-key enforcement is per connection and is not enabled by declaring a
schema foreign key alone. Add `_pragma=journal_mode(WAL)` if your deployment
needs WAL's reader/writer behavior, after checking its filesystem requirements.

### Native Pools

If you already own a `*sql.DB` opened with the SQLite driver, transfer it with
`sqlitedatabase.NewFrom(pool)`. The Arkoren driver then owns shutdown. `NewFrom`
does not rewrite the DSN or apply the memory single-connection default; configure
both on the supplied pool yourself. Its positive builder policies apply to the
existing pool.

Native connections supplied to SQLite must implement
`database/sql/driver.ConnBeginTx`, as the pinned modernc driver does. Unsupported
native profiles fail transactions with `errors.ErrUnsupported` rather than
falling back to a BEGIN that drops cancellation. Cancellation finalizes rollback
before managed locks release, but native Commit/Rollback have no context argument
and can outlive cleanup budgets.

`driver.SQL()` returns the root `*sql.DB`, useful for `Stats()`. Native SQL bypasses
facade telemetry, transaction routing, and retained-session lifetime checks.
Keep protected work on the facade's callback context.

### Running Locally

No server is required. After `driver.Start(ctx)`, verify a facade call with
`db.Find[int](ctx, query.Raw("SELECT 1"))`, importing
`arkoren.dev/foundation/query`; the result is `1`. Stop the driver when finished.
Memory data disappears on last connection close; stopping a file-backed driver
does not delete the database. Keep `app.db`, its `-wal` and `-shm` files, and
`app.db.*.lock` sibling files out of version control.

See the [`sqlitedatabase` API](https://pkg.go.dev/arkoren.dev/driver/sqlite/sqlitedatabase).

## PostgreSQL

Use PostgreSQL for a shared SQL server with transactional DDL and advisory locks
within a database. Provision the database, a role with application permissions,
and a DSN with the TLS policy required by your deployment. Install:

```sh
go get arkoren.dev/driver/postgres
```

### Connecting

Import `arkoren.dev/driver/postgres/postgresdatabase` and
`arkoren.dev/core/service/database`. Here `dsn` is your configured PostgreSQL URL
or key-value DSN, not a hardcoded production credential:

```go
driver, err := postgresdatabase.New(dsn)
db := database.New(driver)
```

Handle construction errors before using the driver. Blank or invalid DSNs and
negative limits match `postgresdatabase.ErrInvalidConfig`. Parse diagnostics omit
supplied values, but unwrapped native errors can still expose them; do not log
those causes or the DSN.

Construction performs no network I/O. `driver.Start(ctx)` or `driver.Ping(ctx)`
verifies connectivity through pgxpool. Use that as a startup/readiness check, then
use the shared [typed operations](https://arkoren.dev/docs/guides/database/overview.md). Drain callbacks and cursors before
`driver.Stop(cleanupCtx)`: stop waits for checked-out connections. If its context
ends, it returns that error while pool closure continues in the background.

### Connection Settings

If the application's connection budget is twenty, configure the pool at construction:

```go
driver, err := postgresdatabase.New(dsn, func(connection *postgresdatabase.Builder) {
	connection.MaxOpenConns(20)
	connection.ConnMaxIdleTime(10 * time.Minute)
})
```

Import `time`. Zero retains pgxpool settings: by default the maximum is the larger
of four and the CPU count, maximum lifetime is one hour, and maximum idle time is
thirty minutes. `MaxIdleConns` is accepted for symmetry but has no effect; pgxpool
has no idle-count limit. Use idle time instead.

Nonzero native `pool_min_conns` and `pool_min_idle_conns` are rejected because
they would connect during construction; `pool_health_check_period` must be
positive. For eager minimum connections, create the native pool at application
startup and transfer ownership with `NewFrom`.

### Native Pools

With an existing `*pgxpool.Pool`, call `postgresdatabase.NewFrom(pool)`. Configure
its `pgxpool.Config` before creating it: `NewFrom` has no builder callbacks.
The Arkoren driver owns the pool and closes it on stop.

`driver.DB()` exposes that root pool for pgx features such as COPY, LISTEN, and
batches. For example, `driver.DB().Stat()` reads native utilization. SQL issued
through it bypasses facade scopes and telemetry, even if the accessor was reached
from a transaction driver. Do not use it for work protected by a managed lock.

For a native bulk import, assume an existing `imports` table with a text `email`
column and import `github.com/jackc/pgx/v5`:

```go
copied, err := driver.DB().CopyFrom(ctx, pgx.Identifier{"imports"}, []string{"email"},
	pgx.CopyFromRows([][]any{{"ada@example.com"}, {"grace@example.com"}}))
```

On success, `copied` is 2. This native operation is outside facade transactions;
use a native pgx transaction if several native operations must be atomic. For
LISTEN, acquire and retain a native connection until listening ends, then release
it; a pool-level LISTEN alone does not pin subsequent notification reads.
Consult the [pgx API](https://pkg.go.dev/github.com/jackc/pgx/v5) for native
batches and notification handling rather than assuming facade scope propagation.

> [!WARNING]
> PgBouncer transaction pooling cannot preserve session advisory locks across
> operations. Use a direct or session-pooled connection for managed named locks
> and migrations; otherwise acquisition and release can reach different sessions.

### Running Locally

With Docker installed and port 15432 free, start a disposable loopback-only server:

```sh
docker run --name app-postgres -d -p 127.0.0.1:15432:5432 -e POSTGRES_USER=app -e POSTGRES_PASSWORD=local-only -e POSTGRES_DB=app postgres:17
```

Wait for readiness using `docker exec app-postgres pg_isready -U app -d app`.
When it reports accepting connections, use this development-only DSN:

```go
driver, err := postgresdatabase.New("postgres://app:local-only@127.0.0.1:15432/app?sslmode=disable")
```

Call `driver.Start(ctx)`, wrap with `database.New(driver)`, and read
`db.Find[int](ctx, query.Raw("SELECT 1"))`; success returns 1. These credentials
and disabled TLS are only for the disposable loopback server. Stop your
application and driver before removing it. The following command deletes the
container and its anonymous data volume; never use it for data you need to keep:

```sh
docker rm -f -v app-postgres
```

See the [`postgresdatabase` API](https://pkg.go.dev/arkoren.dev/driver/postgres/postgresdatabase).
PostgreSQL's separate search adapter is configured in
[Search Drivers](https://arkoren.dev/docs/guides/search/drivers.md#postgres); constructing a database driver
does not construct or migrate that adapter.

## MySQL

Use MySQL when your application already operates a MySQL server. Provision a
database and role, then supply the native Go driver's DSN with appropriate TLS
and permissions. The query compiler's row-lock syntax targets MySQL 8; do not
infer full MariaDB compatibility from a compatible wire protocol. Install:

```sh
go get arkoren.dev/driver/mysql
```

### Connecting

Import `arkoren.dev/driver/mysql/mysqldatabase` and
`arkoren.dev/core/service/database`. With a configured native `dsn`:

```go
driver, err := mysqldatabase.New(dsn)
db := database.New(driver)
```

Handle construction errors before use. Blank or malformed DSNs and negative
limits match `mysqldatabase.ErrInvalidConfig`. Parse messages omit supplied
values; unwrapped native causes are not safe diagnostic output.

Construction opens a lazy `database/sql` pool. `driver.Start(ctx)` or
`driver.Ping(ctx)` checks connectivity before serving traffic. Use the shared
[typed operations](https://arkoren.dev/docs/guides/database/overview.md) after startup. Finish callbacks and cursors
before `driver.Stop(cleanupCtx)`, which ignores the context and closes the owned
pool through `database/sql`.

### Connection Settings

To keep this process within its connection budget:

```go
driver, err := mysqldatabase.New(dsn, func(connection *mysqldatabase.Builder) {
	connection.MaxOpenConns(20)
	connection.MaxIdleConns(5)
})
```

Positive settings replace native limits; zero leaves them unchanged. Without
configuration, `database/sql` supplies the pool defaults, including an unlimited
open-connection count. `ConnMaxLifetime` and `ConnMaxIdleTime` optionally limit
reuse. Account for held locks and sessions as well as request concurrency.

`New` adds `parseTime=true` unless the DSN's parameter list explicitly sets it.
This enables DATETIME scanning into `time.Time`; explicit `parseTime=false`
remains false. MySQL has no timezone-aware datetime type: use a consistent UTC
storage and connection policy.

### Native Pools

Transfer a `*sql.DB` opened with the native MySQL driver through
`mysqldatabase.NewFrom(pool)`. The Arkoren driver owns it afterward. `NewFrom`
applies positive builder policies but does not add `parseTime=true`; configure
the original DSN if time scanning needs it. `driver.SQL().Stats()` exposes
native utilization. Native SQL bypasses facade telemetry and context scopes.

MySQL DDL can implicitly commit and invalidate savepoints. Schema changes and
migration bookkeeping are not generally atomic; follow
[deployment and recovery](https://arkoren.dev/docs/guides/migrations/deployment.md) rather than relying on a
transaction around DDL. Named locks use `GET_LOCK`, span a server, and hash
names longer than 64 bytes. They still obey the shared managed lifetime rules.

MySQL upserts ignore the builder's conflict target: any unique key can trigger
an update. `DoNothing` uses INSERT IGNORE, which is not identical to PostgreSQL's
conflict-only behavior. `Returning` is unsupported even when a wire-compatible
server offers a native returning feature. To read a generated row, select it by
an application-owned unique key inside the same transaction.

The [MySQL ledger requirement](https://arkoren.dev/docs/guides/migrations/troubleshooting.md#mysql-ledger)
applies to migration history, not your application table collation. The runner
checks canonical binary storage and never repairs legacy history automatically.

### Running Locally

With Docker installed and port 13306 free, start a disposable loopback-only server:

```sh
docker run --name app-mysql -d -p 127.0.0.1:13306:3306 -e MYSQL_ROOT_PASSWORD=local-root-only -e MYSQL_DATABASE=app -e MYSQL_USER=app -e MYSQL_PASSWORD=local-only mysql:8.4
```

Repeat `docker exec app-mysql mysqladmin ping -h 127.0.0.1 -uapp -plocal-only --silent`
until it succeeds, then connect with the application user:

```go
driver, err := mysqldatabase.New("app:local-only@tcp(127.0.0.1:13306)/app")
```

Call `driver.Start(ctx)`, wrap with `database.New(driver)`, and read
`db.Find[int](ctx, query.Raw("SELECT 1"))`; success returns 1. This plaintext DSN
and disposable password are not production settings. After stopping the application
and driver, the following command deletes the container and anonymous data volume;
use it only for this disposable data:

```sh
docker rm -f -v app-mysql
```

See the [`mysqldatabase` API](https://pkg.go.dev/arkoren.dev/driver/mysql/mysqldatabase).

---

Source: https://arkoren.dev/docs/guides/database/transactions.md
[HTML documentation](https://arkoren.dev/docs/guides/database/transactions)

# Transactions

When several writes must succeed together, run them inside `WithTransaction`.
The callback's result decides whether to commit or roll back. The examples
assume a configured `db`, an operation context `ctx`, and the imports
`context`, `arkoren.dev/core/service/database`, and `arkoren.dev/foundation/query`.
Connection setup belongs in [Drivers](https://arkoren.dev/docs/guides/database/drivers.md).

## Related Writes

Suppose changing a user's email must also create an audit record. With existing
`updateEmail` and `insertAudit` statements, execute both using the callback context:

```go
err := db.WithTransaction(ctx, func(ctx context.Context) error {
	if _, err := db.Exec(ctx, updateEmail); err != nil {
		return err
	}
	_, err := db.Exec(ctx, insertAudit)
	return err
})
```

A nil callback result requests commit. An error requests rollback and remains
matchable through `errors.Is`; a panic rolls back and is re-raised. Cancellation
fails the transaction even if the callback returns nil. Neither an external
HTTP request nor a message already sent becomes reversible by putting it here.

The context carries the transaction, not the `db` variable. Pass it through
repository and service methods. Operations on another facade over the same
driver join it; a facade over another driver does not. Separate database
transactions are not one atomic unit. Calling `db.Driver()` bypasses routing
and facade telemetry, even inside the callback.

Use the callback context synchronously from one goroutine and only until the
callback returns. Later database I/O fails with `sql.ErrTxDone`. Close active
[cursors](https://arkoren.dev/docs/guides/database/batching-and-streaming.md#streaming-rows) before another statement or
a transaction boundary.

## Nested Work

A helper that calls `WithTransaction` with an existing transaction context
starts a nested transaction. The shipped SQLite, PostgreSQL, and MySQL drivers
use savepoints. A custom driver can reject nesting with
`database.ErrNestedTransaction`.

An ordinary nested callback error rolls back the nested work only; the outer
callback decides whether to continue or return that error. Canceling the parent
context still ends the whole transaction, including its nested work.
Successful nested work is still provisional
until the outermost commit. Nested rollback does not release named locks
acquired by that transaction; see [Locking](https://arkoren.dev/docs/guides/database/locking.md#named-locks).

## Isolation and Retries

Transactions use the driver's defaults; `WithTransaction` has no isolation-level
option. A transaction is not automatically a stable snapshot across statements.
For example, PostgreSQL's default read committed isolation can observe changes
committed between two reads. Use [row locks](https://arkoren.dev/docs/guides/database/locking.md#row-locks) when a
read followed by a write must exclude competing updates, and verify your
database's isolation policy for snapshot-sensitive work.

There is no automatic transaction retry. Retry only after classifying the
failure and establishing that repeating the entire callback is safe. A lost
connection or failed commit can leave an uncertain outcome; reconcile it using
application identifiers rather than assuming rollback.

> [!WARNING]
> If cleanup or a hook fails after a confirmed commit, `WithTransaction` wraps
> the failure with `database.ErrAfterCommit`. Do not retry that committed
> transaction: replay can duplicate its work. Check the error's origin rather
> than treating a marker returned by your own callback as proof of commit.

## After Commit

An in-process notification should not announce a change that later rolls back.
Register it with `AfterCommit`. Here `notify` is an application callback with
the type `func(context.Context) error`:

```go
err := db.WithTransaction(ctx, func(ctx context.Context) error {
	if _, err := db.Exec(ctx, updateEmail); err != nil {
		return err
	}
	return db.AfterCommit(ctx, notify)
})
```

Hooks run synchronously in registration order after the outermost commit and
before `WithTransaction` returns. They receive the outer call's context, without
the finished transaction. An enclosing session remains active. A nested hook
waits for the outer commit; nested rollback discards it. Without a matching
transaction, `AfterCommit` calls the function immediately.

Every hook runs even if an earlier one returns an error; those errors are joined
under `ErrAfterCommit`. A hook panic propagates after commit and prevents later
hooks from running. A transaction-body error carrying that marker still requests
rollback; it is not evidence that this transaction committed.

Hooks live only in memory. A crash after commit can lose them. Use them for
best-effort cache invalidation or broadcasts, not work the user is owed. For
durable delivery, write an outbox record in the same transaction and have a
worker process it with duplicate-safe delivery.

For the full contract, see the
[`database` API reference](https://pkg.go.dev/arkoren.dev/core/service/database).

---

Source: https://arkoren.dev/docs/guides/database/locking.md
[HTML documentation](https://arkoren.dev/docs/guides/database/locking)

# Locking

Use a row lock when concurrent transactions must coordinate changes to the
same records. Use a named lock when cooperating workers must exclude one another
by an application-defined name. Neither mechanism authorizes access to data;
apply your normal access filters as well.

The examples assume a configured `db` and `ctx`. Import `context`,
`arkoren.dev/core/service/database`, and `arkoren.dev/foundation/query` as needed.
Provider lock scope is compared in [Drivers](https://arkoren.dev/docs/guides/database/drivers.md#compatibility-matrix).

## Row Locks

Suppose an inventory reservation first reads a product, then changes its stock.
On PostgreSQL or MySQL 8, build a locking SELECT inside a transaction:

```go
locked := query.Table("products", func(statement *query.Builder) {
	statement.Select("stock")
	statement.Where("id", 7)
	statement.ForUpdate()
})
```

Keep the dependent update in that same callback. Here `reserveStock` is an
already built update for product 7; your application must check the quantity
before issuing it. `quantity` is a validated positive requested quantity and
`ErrInsufficientStock` is an application error:

```go
err := db.WithTransaction(ctx, func(ctx context.Context) error {
	stock, err := db.Find[int](ctx, locked)
	if err != nil {
		return err
	}
	if stock < quantity {
		return ErrInsufficientStock
	}
	_, err = db.Exec(ctx, reserveStock)
	return err
})
```

The lock lasts until the transaction ends; reading with
the outer context would bypass it. `ForShare` requests a shared row lock instead.
SQLite and ANSI compilation reject these clauses with `query.ErrUnsupported`.
PostgreSQL also rejects row locks with DISTINCT, GROUP BY, or HAVING.

When workers claim independent rows, `SkipLocked` skips rows another transaction
has locked. `NoWait` requests an immediate database error instead of waiting.
Use either with `ForUpdate` or `ForShare`, not as a substitute for one; the two
contention policies cannot be combined. Skipping
locked rows is suitable for a work queue, not a complete business report.

## Named Locks

For a report that only one worker should generate, skip rather than wait when
another worker already holds the lock. This fragment belongs in a function
with a named `err` result and imports the standard-library `errors` package:

```go
unlock, acquired, err := db.TryLock(ctx, "daily-report")
if err != nil || !acquired {
	return err
}
defer func() {
	err = errors.Join(err, unlock(context.WithoutCancel(ctx)))
}()
```

Run the protected report after acquisition. The defer preserves
release errors independently of caller cancellation; in production, bound cleanup
with an application-chosen timeout. `acquired == false` with nil error means
contention, not failure. Pool acquisition and network I/O can still wait under
`ctx`. Use `Lock` instead if the worker must wait until acquisition or cancellation.

Names must be nonempty; an empty name fails with `database.ErrEmptyLockName`.
Locks are exclusive and not reentrant, even through nested scopes. Unlock is
safe to call more than once. PostgreSQL coordinates within one database; MySQL
coordinates within one server and hashes names longer than 64 bytes. Choose
names that distinguish application databases when sharing a MySQL server.

Inside a transaction, unlock does nothing: the outermost transaction releases
its locks after commit or rollback. A lock first acquired in a nested transaction
survives that nested rollback. A session lock acquired before the transaction
remains owned by the session instead.

## Retained Sessions

When a lock and several independently committed operations must share one
physical connection, use `WithSession`:

```go
err := db.WithSession(ctx, func(ctx context.Context) error {
	_, acquired, err := db.TryLock(ctx, "daily-report")
	if err != nil || !acquired {
		return err
	}
	return generateReport(ctx)
})
```

Here `generateReport` is application code that forwards `ctx` to facade calls.
The session releases the lock and closes outstanding cursors when the callback
returns. It never reconnects or replays the callback after connection loss.
It does not begin a transaction: an error does not undo earlier autocommit work.
Transactions started with its context use the retained connection.

Use sessions synchronously from one goroutine. A callback context or cursor
used after return fails with `database.ErrSessionDone`. Nested sessions borrow
the connection but expire independently and release their own nontransactional
locks. Arbitrary SQL session state is not automatically reset. Native pool
access and `db.Driver()` bypass this protection.

> [!WARNING]
> Every held root named lock can reserve a pool connection. Sessions and
> transactions also reserve a connection while waiting. These combinations can
> exhaust a bounded pool; no pool size makes arbitrary lock nesting safe. Use
> `TryLock` in a short session and leave that session on contention so other work
> can acquire connections. Never reacquire a lock you already hold.

SQLite coordinates within one process. On Linux, macOS, and BSD hosts, file-backed
databases also use sibling lock files for same-host process coordination. Memory
databases, other operating systems, and network filesystems do not provide that
cross-process guarantee. PostgreSQL session locks require a direct or
session-pooled connection, not PgBouncer transaction pooling.

For every scope and cleanup rule, see the
[`database` API reference](https://pkg.go.dev/arkoren.dev/core/service/database).

---

Source: https://arkoren.dev/docs/guides/database/batching-and-streaming.md
[HTML documentation](https://arkoren.dev/docs/guides/database/batching-and-streaming)

# Batching and Streaming

An export or background job often needs every matching row, not one page for
a user. `Each` reads successive bounded batches; `Cursor` streams one query.
Both return fallible lazy collections: work starts when you iterate, and errors
arrive during iteration. For user-facing result windows, use
[Pagination](https://arkoren.dev/docs/guides/queries/pagination.md) instead.

Assume a configured `db`, an operation context `ctx`, and imports from
`arkoren.dev/foundation/query`. The `users` table has non-null `id` and `email`
columns, and `User` has matching `db` fields as in [Overview](https://arkoren.dev/docs/guides/database/overview.md#reading-data).

## Bounded Batches

For a large user export, order by a unique non-null key so the next batch can
start after the last row already processed:

```go
users := query.Table("users", func(statement *query.Builder) {
	statement.Select("id", "email")
	statement.OrderBy("id")
})
```

Read at most 100 rows per database query, then process them individually:

```go
for user, err := range db.Each[User](ctx, users, 100).Iter() {
	if err != nil {
		return err
	}
	if err := writeUser(user); err != nil {
		return err
	}
}
```

`writeUser` is your export sink. `Each` buffers one batch, checks cancellation
between buffered rows, and seeks after the final ordering values for the next
query. It does not revisit inserts behind that position. Stopping the loop
prevents later batches from being fetched; consuming the iterator again starts
another traversal, not a resume.

The query must be a SELECT with an order and without `Limit` or `Offset`.
Size below one yields `database.ErrInvalidBatchSize`. Missing order yields
`database.ErrNoOrder`; preset paging yields `database.ErrPaged`. Every ordered
column must be selected into a matching result field. End compound ordering
with a unique tiebreaker; uniqueness is your responsibility, not a runtime check.

All ordering values in the selected dataset must be non-NULL. Fetched NULL keys
fail with `database.ErrCursorValue`, but SQL comparisons can exclude NULL rows
before the application can detect them. Use non-null keys or explicitly filter
out NULLs. Keep ordering keys stable while the job runs; changing them can move
rows across the current position.

Each batch is a separate query, not one snapshot. A consistent view requires
database isolation that preserves a snapshot across statements; a transaction
alone, including PostgreSQL's default read committed transaction, is insufficient.

## Streaming Rows

If one query should remain open while rows are consumed, use `Cursor` with the
same `users` statement:

```go
for user, err := range db.Cursor[User](ctx, users).Iter() {
	if err != nil {
		return err
	}
	if err := writeUser(user); err != nil {
		return err
	}
}
```

This avoids collecting the complete result in application memory, but holds
the database cursor and its connection during consumption. Slow sinks prolong
that lifetime. Do not leave the iterator unconsumed inside a transaction or
session and then use it after the callback returns. Close or finish streaming
before another statement or transaction boundary on the same connection.

Exhaustion, scan failure, and an early `break` close the rows. On exhaustion or
failure, close errors are joined with the iteration error. After an early stop,
cleanup errors can only reach telemetry because the iterator cannot yield again.
If application code must receive early-close failures, use the driver's `Query`
and handle `Rows.Close` explicitly; raw driver access also bypasses facade
scope routing and telemetry, so choose the correct scoped driver deliberately.

## Partial Work

A later read, scan, or sink failure does not undo output already written.
Design exports for incomplete-file cleanup and background effects for
duplicate-safe recovery. Neither lazy iterator is a persisted checkpoint.
Do not turn a fallible stream into a success-only iterator that hides its error;
see [Fallible Collections](https://arkoren.dev/docs/guides/collections/fallible-collections.md).

For the iteration contracts, see the
[`database` API reference](https://pkg.go.dev/arkoren.dev/core/service/database).

---

Source: https://arkoren.dev/docs/guides/database/query-monitoring.md
[HTML documentation](https://arkoren.dev/docs/guides/database/query-monitoring)

# Monitoring

A slow database call can include time waiting for a connection, executing SQL,
and reading results. Facade telemetry records client-side operations so you can
relate that time to a request or job. Transaction events report outcomes, not
every query. Configure a telemetry provider through
[Telemetry](https://arkoren.dev/docs/guides/observability/telemetry.md) before expecting exported data.

## Operation Telemetry

With an existing `driver` and `signals` of type `*telemetry.Telemetry`, configure
the facade using `arkoren.dev/core/service/database`:

```go
db := database.New(driver, func(connection *database.Builder) {
	connection.Telemetry(signals)
	connection.Namespace("application")
})
```

`Namespace` labels the database name; it does not select a database or change
the connection. Without an explicit telemetry value, the facade resolves
`telemetry.Default()` on each operation. Disabled telemetry exports nothing.
The database system label comes from the driver's dialect unless overridden
with `System`.

`Exec`, `Select`, `Find`, and consumed `Cursor` operations create client spans
and record `db.client.operation.duration` in seconds. Lock acquisition uses
`LOCK` or `TRY_LOCK`; unlock is not traced. Pagination produces its underlying
count/read operations, and batching produces one read per batch. Transactions
have no separate facade span.

A cursor span lasts through iteration and ends on exhaustion, failure, or early
stop. It therefore includes time spent consuming rows, not merely server SQL
execution. Statement-build errors occur before an operation span starts.
Not-found errors count as successful outcomes. Other failures record an error
class, not error text that might contain private values.

Builder queries supply a table label. Raw SQL supplies only a recognized leading
verb; comments and CTEs are not parsed into operation names or table labels.
Keep namespaces stable rather than using user IDs as metric dimensions.

## SQL Privacy

SQL text is disabled by default. Bound argument values are never added by the
facade, but literals and comments in SQL can still contain sensitive data.
Only after auditing every statement sent through this facade, opt into verbatim
compiled text:

```go
db := database.New(driver, func(connection *database.Builder) {
	connection.Telemetry(signals)
	connection.QueryText(true)
})
```

This alternative construction adds `db.query.text` to spans, not metric labels.
It does not redact SQL. Review exporter access and retention as well as the
query itself; parameter binding alone does not make all statement text public.

## Transaction Events

To observe commits and rollbacks, pass an existing event bus:

```go
db := database.New(driver, func(connection *database.Builder) {
	connection.Events(bus)
})
```

`database.TransactionCommitted` fires after confirmed outermost commit and before
after-commit hooks. `database.TransactionRolledBack` fires when the outermost
transaction body, context, or commit fails. Nested transactions emit neither.
`TransactionEvent` contains `System`, `Namespace`, and `Err`; `Err` is nil for
a confirmed commit even if later cleanup or a hook fails. A rollback event is
not proof that a connection-loss outcome has been reconciled on the server.

Use the typed [event bus](https://arkoren.dev/docs/guides/events/overview.md) to register listeners. Events
are in-process observations, not a durable audit log. Migration events have a
separate [deployment contract](https://arkoren.dev/docs/guides/migrations/deployment.md#migration-events).

## Pool Diagnostics

When requests stall before SQL executes, inspect pool utilization alongside
operation durations. PostgreSQL exposes `driver.DB().Stat()`; MySQL and SQLite
expose `driver.SQL().Stats()`. These are native clients described in
[Drivers](https://arkoren.dev/docs/guides/database/drivers.md), not portable facade metrics.

Held named locks, unfinished sessions, transactions, and streaming cursors all
consume capacity. Finish them before increasing limits. Native pool calls and
`db.Driver()` bypass facade telemetry and context routing; do not move protected
SQL onto the root pool merely to diagnose it.

For configuration and event types, see the
[`database` API reference](https://pkg.go.dev/arkoren.dev/core/service/database).

---

Source: https://arkoren.dev/docs/guides/queries/query-builder.md
[HTML documentation](https://arkoren.dev/docs/guides/queries/query-builder)

# Query Builder

The query builder describes a SQL statement without opening a connection or
executing it. Pass the published definition to the
[database facade](https://arkoren.dev/docs/guides/database/overview.md) for execution, or compile it for a
native client. Import `arkoren.dev/foundation/query`.

## Selecting Rows

Suppose `users` contains `id`, `email`, and `active`. Select only the fields your
result needs and give bounded reads an explicit order:

```go
users := query.Table("users", func(statement *query.Builder) {
	statement.Select("id", "email")
	statement.Where("active", true)
	statement.OrderBy("id")
	statement.Limit(20)
})
```

`Table` publishes an immutable `query.Query`. Its callback edits a temporary
draft; retaining that draft cannot change the published query. Values such as
`true` become bound arguments. Valid identifiers are quoted for the chosen
dialect, not treated as raw SQL. `Select` replaces the selected columns;
`AddSelect` appends to them. Without a selection, the default is `*`.

`Where("id", 7)` means equality; `Where("id", ">", 7)` supplies an operator.
Use `WhereNull` for SQL NULL, `WhereIn` for membership, and `WhereBetween` for
inclusive bounds. Empty `WhereIn` is false; empty `WhereNotIn` is true. In
particular, an empty exclusion list does not restrict a write.

For an independent variation, derive from the published definition:

```go
recent := users.Derive(func(statement *query.Builder) {
	statement.Where("id", ">", 100)
})
```

`users` remains unchanged. Arbitrary argument objects are still caller-owned;
do not mutate them concurrently with compilation or execution. Limits reject
negatives, and `Limit(0)` requests no rows, not an unbounded result. Let the
[pagination helpers](https://arkoren.dev/docs/guides/queries/pagination.md) set limits when they own the result window.

## Targeted Writes

For a new user, expose only the fields this operation accepts:

```go
type NewUser struct {
	Email  string `db:"email"`
	Active bool   `db:"active"`
}
```

With a validated `input` of that type, convert the fields to column values:

```go
insert := query.Table("users", func(statement *query.Builder) {
	statement.Insert(query.Values(input))
})
```

`Values` uses `db` tags, not JSON tags. It includes zero values, so `false` is
an explicit value rather than an omitted patch field. `InsertRows` accepts
several value maps when you need a multirow insert; split very large inputs to
respect database parameter and statement-size limits.

For an email-only update, use a separate writable shape:

```go
type UpdateEmail struct {
	Email string `json:"email" db:"email"`
}
```

After decoding and validating that input, select the target using an authorized
ID outside the body. `Update` needs `query.Values(input)`, not the struct itself:

```go
update := query.Table("users", func(statement *query.Builder) {
	statement.Where("id", userID)
	statement.Update(query.Values(input))
})
```

Use the same explicit filtering approach with `Delete()`. UPDATE and DELETE
without a WHERE fail at compilation unless you deliberately call `Unfiltered()`.
That guard is not authorization: a permissive predicate, including an empty
`WhereNotIn`, can still affect every row. Ordering or limits on writes are
rejected rather than silently discarded.

For an atomic counter change, `Increment("visits", 1)` performs the arithmetic in
SQL instead of reading and writing a stale application value. `SetRaw` permits
trusted computed assignments with bound arguments; never build its expression
from request text.

## Returned Values

PostgreSQL and SQLite can return a newly inserted row without a separate lookup.
For this independent insert, `input` is the validated `NewUser` shape above:

```go
insert := query.Table("users", func(statement *query.Builder) {
	statement.Insert(query.Values(input))
	statement.Returning("id", "email")
})
```

With a configured `db`, operation context `ctx`, and the `User` result from
[Database Overview](https://arkoren.dev/docs/guides/database/overview.md#reading-data), use
`db.Find[User](ctx, insert)` or `db.Select[User](ctx, insert)` to scan returned
rows. `Exec` only returns the affected-row count. `Returning` fails with
`query.ErrUnsupported` on MySQL and ANSI.

For duplicate-safe inserts, `OnConflict("email").DoUpdate("active")` updates
the named columns from the proposed row; `DoNothing()` chooses the no-update
path. PostgreSQL and SQLite use ON CONFLICT. MySQL uses ON DUPLICATE KEY UPDATE
or INSERT IGNORE, whose matching and error behavior are native MySQL semantics,
not an exact conflict-target equivalent. ANSI rejects upserts.

## Joins and Totals

To display a user's team name, assume `users.team_id` references `teams.id`:

```go
members := query.Table("users", func(statement *query.Builder) {
	statement.Select("users.id", "users.email", "teams.name AS team")
	statement.Join("teams", "users.team_id", "=", "teams.id")
})
```

The result type needs fields for the returned column names, including `db:"team"`.
`LeftJoin` keeps unmatched users; represent nullable joined values with nullable
Go fields. Use `WhereColumn` to compare columns rather than binding the second
column name as a value.

For counts per team, define the aggregate expression as trusted SQL:

```go
totals := query.Table("users", func(statement *query.Builder) {
	statement.Select("team_id")
	statement.SelectRaw("COUNT(*) AS total")
	statement.GroupBy("team_id")
	statement.HavingRaw("COUNT(*) > ?", 5)
})
```

`SelectRaw` appends a verbatim expression and accepts no arguments; `HavingRaw`
binds its argument. `totals.Count()` counts the result groups, not their members.
Grouped, DISTINCT, HAVING, and raw-source count queries wrap the original SELECT,
so their selected columns need unique names.

## Reusable Filters

If a tenant condition must constrain every branch of a query, use a scope rather
than relying on the placement of an ordinary `Where` beside OR conditions:

```go
users := query.Table("users", func(statement *query.Builder) {
	statement.Scope(func(predicate *query.PredicateBuilder) {
		predicate.Where("tenant_id", tenantID)
	})
	statement.Where("active", true)
	statement.OrWhere("invited", true)
})
```

The complete user filter is grouped, then ANDed with the scope. A later
`OrWhere` cannot bypass it; scopes also apply to writes and `Count`. Obtain
`tenantID` from your authorization boundary, not an untrusted body field. Empty
scopes do nothing, and raw SQL remains trusted SQL even inside a scope. See
[Row Access](https://arkoren.dev/docs/guides/authorization/row-access.md) for policy integration.

For ordinary parentheses, use `WhereGroup` with a `PredicateBuilder`. For
existence checks, pass a published SELECT to `WhereExists` or `WhereNotExists`,
using `WhereColumn` for correlation. MySQL can reject an UPDATE or DELETE whose
subquery reads the target table with native error 1093.

## Compiling SQL

The facade chooses its driver's dialect automatically. To inspect the SQL or
use a native client, compile deliberately:

```go
compiled, err := users.BuildFor(query.Postgres)
```

`compiled.SQL()` contains placeholders and `compiled.Args()` contains values in
binding order. `Build()` uses ANSI instead. Construction problems are retained
and returned at build time under `query.ErrInvalidQuery`; a later edit cannot
repair an inherited error. Unsupported dialect features also match
`query.ErrUnsupported`. Compilation is not database execution or proof that
tables exist. Use [Raw SQL](https://arkoren.dev/docs/guides/queries/raw-sql.md) for statements outside this structured API.

For the complete builder, see the
[`query` API reference](https://pkg.go.dev/arkoren.dev/foundation/query).

---

Source: https://arkoren.dev/docs/guides/queries/raw-sql.md
[HTML documentation](https://arkoren.dev/docs/guides/queries/raw-sql)

# Raw SQL

Use handwritten SQL when a database operation does not fit the
[Query Builder](https://arkoren.dev/docs/guides/queries/query-builder.md), such as a CTE or a vendor-specific expression.
`query.Raw` and `query.RawNamed` still bind values and compile placeholders for
the driver. They do not translate SQL syntax or validate the statement's effect.

Import `arkoren.dev/foundation/query`. The examples assume a configured `db`,
`ctx`, and a `User` result with `id` and `email` fields from
[Database Overview](https://arkoren.dev/docs/guides/database/overview.md#reading-data).

## Bound Values

For a user lookup, put the ID in the argument list rather than formatting it
into the SQL string:

```go
lookup := query.Raw("SELECT id, email FROM users WHERE id = ?", userID)
user, err := db.Find[User](ctx, lookup)
```

The facade rewrites `?` to the driver's placeholder syntax, such as `$1` on
PostgreSQL. `Find` preserves the usual missing-row behavior; `Select` scans
several rows, and `Exec` returns the affected-row count without scanning rows.

Parameters bind values, not table names, column names, operators, or sort
directions. Choose those through a fixed application allowlist. Never interpolate
untrusted text into SQL; raw statements do not have the builder's unfiltered-write
guard, so retain explicit authorization and WHERE conditions on writes.

## Named Parameters

For a targeted update with several values, name the bindings. First define the
already decoded and validated input:

```go
type UpdateEmail struct {
	Email string `json:"email" db:"email"`
}
```

Keep the authorized target outside that input, then add it to the binding map:

```go
values := query.Values(input)
values["id"] = userID
update := query.RawNamed("UPDATE users SET email = :email WHERE id = :id", values)
affected, err := db.Exec(ctx, update)
```

`RawNamed` replaces each `:name` outside quotes and comments with a positional
binding. Repeating a name binds its value again. A missing name fails compilation;
extra map entries are unused. PostgreSQL's `::` cast syntax stays intact.
`Values` uses database tags, not JSON decoding tags.

## Raw Sources

When handwritten SQL produces a reusable result set, wrap it with `FromRaw`
and apply ordinary builder filtering outside it:

```go
users := query.FromRaw(query.Raw(
	"SELECT id, email FROM users WHERE active = ?", true,
), func(statement *query.Builder) {
	statement.Select("id", "email")
	statement.OrderBy("id")
})
```

This renders the raw SELECT as a subquery with alias `t` by default; `As`
changes that alias. Inner arguments bind before outer ones. The resulting
`query.Query` supports `Count` and [pagination](https://arkoren.dev/docs/guides/queries/pagination.md). Its raw source
must be a valid subquery for the database and return uniquely named columns
when wrapped for counting. A whole `RawStatement` is opaque and cannot be
passed directly to the facade's pagination methods.

For smaller expressions, use `WhereRaw`, `HavingRaw`, or `SetRaw` with trusted
SQL and `?` bindings. `SelectRaw` is different: it accepts no arguments and
copies its expression verbatim without scanning placeholders.

## Dialect Boundaries

Raw SQL preserves your grammar. PostgreSQL-only operators, MySQL functions, and
SQLite PRAGMAs remain provider-specific even though placeholders are rewritten.
Use `BuildFor(db.Dialect())` when compiling for a native client; `Build()` uses
ANSI syntax. Native execution bypasses facade telemetry and context-based
session/transaction routing.

The positional scanner counts `?` outside standard single, double, and backtick
quotes, line comments, and block comments. The count must match the arguments.
Write `??` for a literal question mark, such as a PostgreSQL JSON operator, in
scanned raw SQL. In `SelectRaw`, write the operator as a single `?` instead.

The scanner does not understand PostgreSQL dollar quoting or MySQL backslash
escapes inside strings. Pass those contents as bound values instead of embedding
them. Block comments nest only on ANSI and PostgreSQL. PostgreSQL line comments
end at CR or LF; other dialects use LF. MySQL recognizes `--` as a comment only
when followed by whitespace or a control character. SQL syntax validation still
belongs to the database; placeholder processing is not a general SQL parser.

For raw statements and compiler rules, see the
[`query` API reference](https://pkg.go.dev/arkoren.dev/foundation/query).

---

Source: https://arkoren.dev/docs/guides/queries/pagination.md
[HTML documentation](https://arkoren.dev/docs/guides/queries/pagination)

# Pagination

Use numbered pages when users need totals or direct page selection. Use cursor
windows when they move forward or backward through a large ordered result and
do not need a total. For background processing of every row, use
[Batching and Streaming](https://arkoren.dev/docs/guides/database/batching-and-streaming.md) instead.

Assume a configured `db` and request or job context `ctx`. Import
`arkoren.dev/foundation/query` and `arkoren.dev/core/pagination`. The examples
read `users` with a non-null unique `id` and a non-null `email`:

```go
type User struct {
	ID    int64  `db:"id"`
	Email string `db:"email"`
}
```

## Stable Ordering

Both helpers require a SELECT with an explicit order and no existing `Limit`
or `Offset`. Define the common query once:

```go
users := query.Table("users", func(statement *query.Builder) {
	statement.Select("id", "email")
	statement.OrderBy("id")
})
```

End any compound order with a unique tiebreaker, such as `created_at` followed
by `id`. The helper requires an order but does not verify uniqueness. Reapply
authorization and tenant filters on every request, including requests with cursors.

## Numbered Pages

To fetch the third page of twenty users:

```go
page, err := db.Paginate[User](ctx, users, pagination.PageParams{
	Page:    3,
	PerPage: 20,
})
```

`Paginate` runs a count and a separate SELECT with offset 40 and limit 20.
The result exposes `Items`, `Total`, `PerPage`, `CurrentPage`, and `LastPage`.
No matches produces an empty collection and a last page of at least one. A
request past the end keeps its requested page number instead of inventing rows.

The count uses `Query.Count`, including grouped or raw-derived result sets.
Count and items are not automatically one snapshot: concurrent changes can
make them disagree unless the database isolation policy preserves a snapshot
across both statements. Deep offsets also require the server to skip preceding
rows; use cursor pagination when that cost is inappropriate.

## Cursor Windows

To read the first window without counting the full result:

```go
page, err := db.CursorPaginate[User](ctx, users, pagination.CursorParams{
	PerPage: 20,
})
```

The helper fetches up to one extra row to detect another window. `Items` contains
at most twenty users; `NextCursor` and `PrevCursor` are opaque continuation tokens.
Pass either token unchanged as `Cursor` on the next request. An empty token means
no token was emitted; an empty result has neither token. There is no total count.
Backward queries are reversed back into the declared display order.

Every ordered column must be present in the result type and selected into it.
All ordering keys in the selected dataset must be non-NULL. Fetched NULL values,
including lookahead rows, fail with `database.ErrCursorValue`; SQL seeks can
exclude NULL rows before this check can see them. Choose non-null keys or
explicitly exclude NULLs rather than relying on the check to validate the dataset.

Tokens encode boundary values and ordering, not a snapshot. Inserts, deletes,
and ordering-key updates between requests can change traversal. Keep tokens
with the same filters and authorization context. The built-in ordering fingerprint
checks columns and directions, not filters or tenant identity.

## Request Parameters

For an existing HTTP request `r`, import `arkoren.dev/core/http/request` and
parse the query parameters before execution:

```go
params, err := request.Page(r, pagination.WithMaxPerPage(50))
```

This reads `page` and `per_page`; `request.Cursor(r)` reads `cursor` and
`per_page`. Handle parsing errors before passing parameters to the facade.
The facade normalizes again with its built-in limits, so request options can
impose stricter limits but do not raise the facade's ceilings.

For directly constructed parameters, zero page selects page 1 and zero size
selects 25. HTTP parsing distinguishes absence from explicit zero: missing values
use defaults, while supplied zero or negative values become one. Size is capped
at 100, and numbered pages above 10,000 or with
overflowing offsets fail with `pagination.ErrPageOverflow`. Cursor normalization
preserves the token unchanged; validation happens when the database helper
decodes it.

Missing order and preset limits return `database.ErrNoOrder` and
`database.ErrPaged`. Cursor fields missing from the result type return
`database.ErrCursorColumn`. Malformed, incompatible-order, or invalidly signed
tokens match `pagination.ErrCursorDecode`. Do not replace these failures with
an apparently successful first page. Encoding can fail too; never return a
partial cursor result as success.

## Signed Cursors

By default, positions are unsigned base64 JSON and clients can forge them.
To reject tampering, load private application material with
`arkoren.dev/foundation/key` and construct a codec:

```go
material, err := key.LoadShared("cursor.key")
codec, err := pagination.NewCursorCodec(material)
```

The key must contain exactly 32 bytes; follow [key provisioning](https://arkoren.dev/docs/guides/keys/overview.md)
instead of committing it. After handling those construction errors, configure
the facade using `arkoren.dev/core/service/database`:

```go
db := database.New(driver, func(connection *database.Builder) {
	connection.CursorCodec(codec)
})
```

The codec rejects unsigned, altered, and wrongly signed tokens. Enabling it
invalidates previously issued unsigned tokens. Rotate by constructing the new
codec with the current key first and previous keys afterward, then retire old
keys according to your cursor-retention policy.

Signing does not encrypt boundary values, enforce expiration, bind filters, or
authorize access. The codec provides no expiry check. Reapply access filters and
implement any additional token policy in the application. Both signed and
unsigned token formats have a 4096-byte limit.

`Items` is an eager collection for this window only. Local filtering or mapping
does not fetch rows, recompute totals, or change cursor boundaries.

See the [`database` API](https://pkg.go.dev/arkoren.dev/core/service/database) and
[`pagination` API](https://pkg.go.dev/arkoren.dev/core/pagination) for the full contracts.

---

Source: https://arkoren.dev/docs/guides/migrations/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/migrations/overview)

# Overview

A migration is a named, ordered change to your database schema or data. The
migrator executes its steps and records completion in a bookkeeping table, so
later runs can identify pending work. Your application owns registration and
execution; constructing a database driver does not run migrations.

Assume an already configured `db` from [Database Drivers](https://arkoren.dev/docs/guides/database/drivers.md).
Import `arkoren.dev/core/service/database/migrate`,
`arkoren.dev/foundation/query/schema`, and `arkoren.dev/foundation/query` where used.

## Defining Changes

To introduce a `users` table, first describe the table:

```go
users := schema.Create("users", func(table *schema.Table) {
	table.ID()
	table.String("email", 255).Unique()
})
```

This prepares a [schema operation](https://arkoren.dev/docs/guides/migrations/schema-builder.md), not a database call.
Attach it to a stable migration identity:

```go
createUsers := migrate.NewMigration("create-users", func(migration *migrate.MigrationBuilder) {
	migration.Up(migrate.Schema(users))
	migration.Down(migrate.Schema(schema.Drop("users")))
})
```

`Up` is required and contains one or more steps. `Down` is optional; omitting
it marks the migration irreversible. A drop reverses this schema addition but
destroys its data, so do not equate reversible schema with recoverable data.
Definition construction is synchronous and performs no SQL.

An ID is 1–191 printable UTF-8 bytes with no whitespace. Prefer a short change
name such as `add-users-email-index`; numeric or date prefixes do not control
execution order. Never rename a recorded ID: the new name becomes pending and
the old name becomes missing. Keep definitions append-only and preserve their
relative order after application. Do not edit an applied Up definition to change
existing databases: add a new migration instead.

## Migration Steps

Use `migrate.Schema` for schema operations and `migrate.Exec` for SQL statements.
Both accept several arguments and run them in written order. For example, a
data change can normalize existing values:

```go
normalizeEmails := migrate.NewMigration("normalize-user-emails", func(migration *migrate.MigrationBuilder) {
	migration.Up(migrate.Exec(query.Raw("UPDATE users SET email = LOWER(email)")))
})
```

This irreversible example assumes case normalization is an approved application
rule and existing uniqueness constraints permit the result. Test on representative
data before deployment; a SQL expression is not a substitute for deciding data policy.

For logic that needs the current dialect or several facade calls, use a function
step with imports `context` and `arkoren.dev/core/service/database`:

```go
step := func(ctx context.Context, db *database.Database) error {
	_, err := db.Exec(ctx, query.Raw("UPDATE users SET email = LOWER(email)"))
	return err
}
```

Pass `step` to `migration.Up(step)` instead of `migrate.Exec`. Always forward the
supplied context: it carries the migration's retained session and any active
transaction. A fresh context or direct `db.Driver()` call bypasses the lock-owning
connection and can deadlock a one-connection pool. Do not retain the context or
use it concurrently. Close streaming cursors before later statements or boundaries.

`Schema` and `Exec` report operation or statement positions on failure. Empty
step helpers and nil statements are detected when the step runs, not by `Add`.
Schema operation construction errors are checked before that helper executes any
of its operations; dialect-specific failures occur during compilation.

Use `migration.Only(query.Postgres)` only when the entire migration is intended
for PostgreSQL. `Add` rejects it on another dialect rather than silently skipping
it. For a change with different implementations per dialect, leave it unrestricted
and switch on `db.Dialect()` inside a function step.

## Applying Changes

Create a migrator and register definitions in dependency order:

```go
migrator := migrate.New(db)
err := migrator.Add(createUsers, normalizeEmails)
```

`New` performs no I/O. `Add` registers all supplied definitions or none; invalid
definitions and duplicate IDs return `migrate.ErrInvalidMigration` and
`migrate.ErrDuplicateID`. After handling registration errors, apply pending work:

```go
err := migrator.Up(ctx)
```

`Up` takes a named lock, prepares the bookkeeping table, reads history, and runs
pending definitions in `Add` order. The default table and lock name are both
`arkoren_migrations`. A successful run records completed migrations in one new
batch; another run skips recorded IDs. Each migration commits independently
when transactional DDL is supported. There is no operation-wide rollback:
earlier commits survive a later failure. Read [Deployment](https://arkoren.dev/docs/guides/migrations/deployment.md) before
using this in production or running nontransactional changes.

## Inspecting History

To compare registered definitions with recorded completion:

```go
statuses, err := migrator.Status(ctx)
```

The result lists registered definitions in `Add` order, then recorded but
unregistered IDs with `Missing`. Each entry includes `ID`, `Group`, `Ran`,
`Batch`, `RanAt`, and `OutOfOrder`. Status takes the same lock and creates the
ledger on a fresh database: it is not a read-only connection probe or a dry run.
An acquisition, session, inspection, or cleanup failure returns nil rather than
an unprotected partial snapshot.

## Reverting Changes

Before rollback, review the selected `Down` steps, take an appropriate backup,
and ensure running code tolerates the previous schema. To revert only the latest
applied migration:

```go
err := migrator.Rollback(ctx, migrate.Steps(1))
```

Without `Steps`, rollback selects the last batch. It processes newest batches
first and reverse `Add` order within a batch, not timestamp order. It validates
all targets before executing any: missing definitions and irreversible targets
fail with `ErrUnknownMigration` and `ErrIrreversible`. An unknown definition takes
precedence over known targets in its batch, including with `Steps(1)`; restore
the definitions rather than guessing which known member to revert. Preserve the original
relative `Add` order so rollback can reconstruct application order. Earlier
successful Down commits remain final if a later Down fails.

## Package Migrations

A reusable package can return definitions in its own group:

```go
definitions := migrate.NewGroup("acme.users").Migrations(createUsers)
err := migrator.Add(definitions...)
```

The group prefixes short IDs, producing `acme.users/create-users`. Grouped short
IDs cannot contain `/`, and the complete ID must still fit 191 bytes. Definitions
already assigned to a group cannot be grouped again. Each group is an independent
append-only sequence for out-of-order checks, while overall `Add` order remains
run order. Add package migrations before application changes that use their tables.

Test definitions against each supported real database engine: apply to an empty
database, rerun to check recorded work is skipped, and verify schema and data.
If Down exists, test its effects separately. Also exercise failure halfway through
nontransactional work; a successful clean install does not establish safe recovery.

For definitions, steps, and execution, see the
[`migrate` API reference](https://pkg.go.dev/arkoren.dev/core/service/database/migrate).

---

Source: https://arkoren.dev/docs/guides/migrations/schema-builder.md
[HTML documentation](https://arkoren.dev/docs/guides/migrations/schema-builder)

# Schema Builder

The schema builder describes DDL: statements that create or change database
objects. An operation is an immutable definition that compiles into one or more
statements. It does not inspect your existing schema or execute SQL. Use
[migration steps](https://arkoren.dev/docs/guides/migrations/overview.md#migration-steps) to apply operations with history
and coordination.

Import `arkoren.dev/foundation/query/schema` and `arkoren.dev/foundation/query`.

## Creating Tables

For a user table with optional biography text:

```go
users := schema.Create("users", func(table *schema.Table) {
	table.ID()
	table.String("email", 255).Unique()
	table.Text("bio").Nullable()
})
```

`ID` adds an auto-incrementing primary key named `id`. Columns are NOT NULL
unless `Nullable` is called. `String` requires a positive length; SQLite renders
it as TEXT rather than enforcing that length. Validate portable application
length and numeric range rules yourself.

Common column choices include integers, booleans, decimals, UUIDs, JSON, binary,
dates, and timestamps. `Timestamps()` adds `created_at` and `updated_at` with
current-time defaults; it does not install an automatic update trigger.
`TimestampTZ` retains provider differences: MySQL uses datetime(6), not a
timezone-aware type. Store UTC consistently, especially on SQLite where
timestamp comparisons use text.

Use `Default(value)` for scalar literal defaults and `DefaultRaw(expression)`
only for trusted database expressions. DDL defaults are rendered into SQL, not
bound parameters. Never pass request text to raw types or expressions.

## Keys and Indexes

If email must be unique per tenant rather than globally, use this alternative
table definition. Assume `tenants` already exists with a compatible `id`:

```go
users := schema.Create("users", func(table *schema.Table) {
	table.ID()
	table.ForeignID("tenant_id", "tenants").OnDelete(schema.Cascade)
	table.String("email", 255)
	table.Unique("tenant_id", "email")
})
```

`ForeignID` adds a bigint column referencing the parent's `id`; cascade deletes
remove dependent users when their tenant is deleted. Choose that destructive
behavior deliberately. Without a cascade policy, do not assume parent deletion
will clean up child rows. SQLite foreign-key enforcement is a connection policy
you must enable in the database configuration.

`Index` adds a nonunique index. `Unique` and a column's `Unique()` create unique
indexes rather than inline constraints. Generated names use the table and
columns: `users_tenant_id_email_key` for this unique index and
`users_tenant_id_fkey` for the foreign key. Schema-qualified names contribute
their schema. Names, including generated ones, must fit 63 bytes.

To remove generated objects during an `Alter`, supply the original columns to
`DropIndex`, `DropUnique`, or `DropForeign`, not the generated index name. `Primary`
supports a table-level primary key; choose key design before data depends on it.

## Adding Columns

For a rolling deployment, begin with an additive change old code can tolerate:

```go
addPhone := schema.Alter("users", func(table *schema.Table) {
	table.String("phone", 32).Nullable()
})
```

The operation adds a nullable column and leaves existing rows readable. Backfill
and application changes are separate steps; do not add a required column to
populated data without deciding how existing rows receive valid values. DDL can
lock or rebuild a table depending on the engine and operation, so assess it on
production-sized data before deployment.

`RenameColumn` and `DropColumn` belong inside `Alter`; `schema.Rename` renames
a table and `schema.Drop` removes one. Renames and drops can break old instances
still serving traffic. Follow [expand and contract](https://arkoren.dev/docs/guides/migrations/deployment.md#rolling-deploys)
and back up data before destructive changes.

## Changing Columns

PostgreSQL can change only the aspects you specify. If an existing phone column
should become nullable:

```go
changePhone := schema.Alter("users", func(table *schema.Table) {
	table.Change("phone").Nullable()
})
```

This PostgreSQL-oriented operation does not restate its type or default. SQLite
cannot compile column changes and returns `query.ErrUnsupported`. Plan a tested
table-rebuild migration when SQLite needs that transformation.

MySQL type or nullability changes use MODIFY COLUMN, which restates the whole
column. Supply a type and either `Nullable` or `NotNull` together, and restate
any default you need to retain. A default-only change uses ALTER COLUMN instead.
There is no universally safe ALTER recipe: inspect the existing attributes and
dependent objects before deciding what to restate or transform. `Change.Type`
and `Change.Using` contain trusted verbatim SQL, not validated user input.

## Inspecting Statements

Check an operation's construction error, then inspect each statement for your
target dialect. Start with the `users` operation above:

```go
err := users.Err()
```

After handling that error, compile its statements individually for PostgreSQL:

```go
for _, statement := range users.Statements() {
	compiled, err := statement.BuildFor(query.Postgres)
	// Inspect compiled.SQL() after handling err.
}
```

Create emits the table first, then one statement per index. Alter emits statements in written
order. Operation construction errors make every statement fail, while dialect
limits appear at compilation under `query.ErrUnsupported`; schema errors also
match `query.ErrInvalidQuery`. Compilation does not verify existing columns,
permissions, data conversion, or acceptable lock time.

`CreateIfNotExists` and `DropIfExists` help make some steps repeatable, but they
do not prove an existing object has the intended definition. Table IF NOT EXISTS
is unavailable on ANSI; index IF NOT EXISTS is unavailable on ANSI and MySQL.
A multi-statement operation can partially apply without transactional DDL.

Literal rendering assumes PostgreSQL `standard_conforming_strings=on` and MySQL
without `NO_BACKSLASH_ESCAPES`. MySQL text/blob/JSON defaults require MySQL 8.0.13
or newer. SQLite REFERENCES targets must be unqualified and resolve in the
referencing table's schema. Test generated SQL on each supported deployment
profile, not only against ANSI output.

For all operations and column modifiers, see the
[`schema` API reference](https://pkg.go.dev/arkoren.dev/foundation/query/schema).

---

Source: https://arkoren.dev/docs/guides/migrations/deployment.md
[HTML documentation](https://arkoren.dev/docs/guides/migrations/deployment)

# Deployment

Deploy schema changes as a dedicated job or command before new instances take
traffic. That runner constructs the database, registers the approved definitions,
calls `Up`, and exits on error. Keep a backup and a recovery plan appropriate to
the change; migration history is a completion ledger, not a backup.

This page assumes [definitions and registration](https://arkoren.dev/docs/guides/migrations/overview.md) are already in
place. Use an ordinary deployment context, not one carrying an application
transaction. The context's deadline must cover the work, not only lock acquisition.

## Controlled Runs

For a production runner, reject unexpected out-of-order changes and bound the
wait for capacity and the migration lock. Import `time` and
`arkoren.dev/core/service/database/migrate`:

```go
migrator := migrate.New(db, func(runner *migrate.Builder) {
	runner.OutOfOrder(migrate.OutOfOrderError)
	runner.LockTimeout(time.Minute)
})
```

Register definitions in their stable dependency order, check `Add` errors, then
run `migrator.Up(ctx)`. Success means the pending migrations completed and their
history was recorded under that operation's scope. On failure, stop deployment
and [inspect the outcome](https://arkoren.dev/docs/guides/migrations/troubleshooting.md#partial-changes) rather than rerunning
automatically. Keep old code compatible with the intermediate schema.

`OutOfOrderError` refuses pending definitions preceding applied ones in the same
group before any migration runs. The default, `OutOfOrderWarn`, logs and runs
them; `OutOfOrderAllow` runs without warning. Groups are independent sequences,
so a package appending a new migration does not become late merely because
application migrations were registered after the package.

Running `Up` in every instance's startup is suitable for development and simple
single-instance deployments, but production replicas would compete for the lock,
delay startup, and tie migration failures to instance health. A dedicated runner
keeps that work separate. The lock does not make an incompatible schema change
safe for already-running code.

## Lock Ownership

`Up`, `Rollback`, and `Status` all acquire the configured named lock. The default
table and lock name are `arkoren_migrations`; if you customize them, keep all
runners for the same history on the same target and lock. Different lock names
do not serialize one another.

Each acquisition attempt reserves a session and calls `TryLock` once. On
contention, a newly acquired session ends before the runner waits and retries,
returning capacity to the pool. After acquisition, preparation, history reads,
planning, steps, transactions, and bookkeeping stay on one physical connection.
Connection loss fails the operation without reconnecting or replaying its plan.

The default lock budget is five minutes. Zero selects that default; negative
`LockTimeout` waits until the caller's context ends. This budget covers pool/session
acquisition and lock contention, then stops when the lock is acquired. It does
not limit DDL or data changes. Budget expiry matches both `migrate.ErrLockTimeout`
and `context.DeadlineExceeded`; a caller deadline that ends first is returned
without the lock-timeout marker.

Cleanup releases locks with a bounded context independent of caller cancellation.
Cleanup errors join the operation error. Panics are re-raised after cleanup.
A lost or uncertain session is not silently pooled as a healthy replacement.

> [!WARNING]
> Every step must forward its supplied context to facade calls. A fresh context,
> `db.Driver()`, or a native root pool call bypasses the retained connection and
> lock scope, and can deadlock a one-connection pool. Close streaming rows before
> another statement or a transaction boundary. Do not separately acquire the
> migration lock: managed locks are not reentrant.

PostgreSQL requires a direct or session-pooled connection, not PgBouncer
transaction pooling. MySQL locks span a server; SQLite cross-process locking is
limited to supported same-host file-backed profiles. See
[provider scope](https://arkoren.dev/docs/guides/database/locking.md#retained-sessions). A caller-owned session
or transaction remains reserved during failed acquisition attempts, so prefer
an ordinary context for deployments. Other held root locks can also consume
all pool capacity.

## Commit Boundaries

PostgreSQL and SQLite normally run each migration's steps and bookkeeping in
one transaction. MySQL migrations run without that transaction because DDL can
implicitly commit. `WithoutTransaction()` also disables the per-migration
transaction on transactional-DDL dialects. Use it when a database statement
cannot run in a transaction, such as PostgreSQL `CREATE INDEX CONCURRENTLY`.
That statement can leave an invalid index after failure; `IF NOT EXISTS` alone
does not repair it.

There is no transaction around the complete `Up` or `Rollback` operation.
Earlier migration commits survive a later failure. Without a per-migration
transaction, earlier statements can survive even when the current migration
has no ledger record. Such work must be safe to repeat after deliberate recovery.
`CreateIfNotExists` or `DropIfExists` can help, but object existence alone does
not establish that a partially applied object has the correct shape.
On MySQL, `CreateIfNotExists` cannot compile when the declaration also creates
indexes, since the generated index statements have no IF NOT EXISTS support.
Keep nontransactional migrations small and give each change a specific recovery plan.

Do not pass a preexisting transaction to a deployment runner: the migrator joins
it, making final commit and lock release depend on that outer transaction.
`WithoutTransaction` cannot escape a transaction the caller already started.
An explicit nested transaction inside a transactional step creates a savepoint
or fails with `database.ErrNestedTransaction` for a driver without nesting.

A confirmed migration commit followed by cleanup or an after-commit hook failure
still leaves the change and ledger committed. Inspect history before taking a
recovery action; the returned error is not proof the migration was undone.

## Rolling Deploys

Old and new instances can run against the same schema during a deployment.
Use expand and contract to keep both working:

1. Add compatible structures, such as a nullable column or a new table, without
   removing old names. Assess index creation and table-alter locking separately.
2. Deploy code that tolerates the expanded schema and, where needed, supports
   the transition between old and new data representations.
3. Backfill and verify data with restartable, bounded work appropriate to its size.
4. After old instances and dependent jobs are gone, enforce new constraints or
   remove unused structures in a later release.

Do not rename or drop a column in the same release that stops using its old
name. Test migrations on the target engine with representative data and measure
lock duration. A valid schema operation does not guarantee an online change.

## Migration Events

To attach in-process deployment observations, provide an existing event bus:

```go
migrator := migrate.New(db, func(runner *migrate.Builder) {
	runner.Events(bus)
})
```

`migrate.Applied`, `migrate.RolledBack`, and `migrate.Failed` describe individual
migrations, not an all-or-nothing deployment. A confirmed migration commit with
post-commit failure publishes success with nil event error but stops later work.
A step error is failure even if it happens to wrap `database.ErrAfterCommit`.
Outer session cleanup does not rewrite outcomes already emitted. Events inside
a caller-owned transaction remain provisional until its outer commit.

Construction defaults to `slog.Default()` for migration logs and no event bus.
Neither logs nor events replace verifying recorded history and the resulting
schema after recovery. Continue with [Troubleshooting](https://arkoren.dev/docs/guides/migrations/troubleshooting.md) if
execution or cleanup did not complete successfully.

For execution and configuration contracts, see the
[`migrate` API reference](https://pkg.go.dev/arkoren.dev/core/service/database/migrate).

---

Source: https://arkoren.dev/docs/guides/migrations/troubleshooting.md
[HTML documentation](https://arkoren.dev/docs/guides/migrations/troubleshooting)

# Troubleshooting

When a migration fails, stop the deployment and identify where it stopped:
acquisition, ledger preparation, a user step, bookkeeping, commit, or cleanup.
The safe next action depends on that boundary. Keep the original error and
deployment identity, but do not publish credentials or sensitive native error text.

This page assumes a configured migrator with the same definitions, table, and
lock name as the failed run. [Deployment](https://arkoren.dev/docs/guides/migrations/deployment.md) explains the transaction
and session boundaries that determine whether work can already be final.

## Acquisition Failures

An error matching `migrate.ErrLockTimeout` means the configured budget expired
while acquiring a session or waiting for the lock, before protected work began.
It also matches `context.DeadlineExceeded`. A caller cancellation or deadline
that ended first is returned without the lock-timeout marker.

Check whether another approved runner is active and whether unfinished cursors,
transactions, sessions, or root named locks occupy the pool. Let the legitimate
holder finish or resolve the resource leak before increasing timeouts. Do not
acquire the migration lock manually before calling the migrator: it is not
reentrant. After correcting acquisition, verify that the next run reaches
protected preparation; a larger timeout does not fix a scope bypass or deadlock.

If PostgreSQL lock behavior is inconsistent behind PgBouncer, check pooling mode.
Use direct or session pooling for session advisory locks. For SQLite, verify that
the database and host profile support cross-process file locks; memory databases
cannot coordinate independent migration processes.

## Definition Failures

`ErrInvalidMigration` can indicate missing Up steps, invalid IDs, a dialect
restriction, invalid configuration, or MySQL ledger incompatibility. Read the
wrapped context rather than treating every instance as a malformed definition.
`ErrDuplicateID` means registration reused an ID. Fix construction before execution;
do not rename a previously applied migration to make registration pass.

For `ErrOutOfOrder`, compare pending and recorded definitions within the affected
group. Restore the intended append-only order and investigate why the earlier
definition is pending. Deliberately allowing a late change requires reviewing its
dependencies and effects on already-applied changes, not merely switching off
the error. The default policy warns and runs late work.

## History Mismatches

`Status` marks recorded IDs absent from registration as `Missing`. Restore the
original definitions and their relative order. Rollback rejects unknown targets
with `ErrUnknownMigration` before executing Down steps; do not delete ledger rows
or guess the ordering from timestamps to force it through.

`ErrIrreversible` means a rollback target has no Down steps. Restore from an
appropriate backup or deploy a reviewed forward correction. Inventing a Down
step after data has been discarded does not make that data recoverable.

With `arkoren.dev/core/service/database/migrate` imported, inspect history after
resolving connectivity and acquisition problems:

```go
statuses, err := migrator.Status(ctx)
```

Handle the error before using `statuses`. Status is protected by the migration
lock and can create the ledger on a fresh database; it is not a read-only probe.
Preparation, metadata, session-loss, or cleanup failure returns nil. On success,
compare IDs, batches, and `Ran` values with the deployed definitions and actual
schema/data, not only the last log line.

## Partial Changes

On MySQL or with `WithoutTransaction`, a failed step or ledger insert can leave
changes applied but unrecorded. `Up` sees that ID as pending and will run it again.
A later failure also leaves earlier successfully committed migrations in place.
Do not treat the whole batch as rolled back.

Before another execution:

1. Quiesce competing migration runners and relevant application writers.
2. Preserve a backup and capture the error, registered order, history, and actual
   objects/data touched by the failed step.
3. Determine which statements completed and whether the final commit or session
   outcome is known. On connection loss, consult server evidence and reconcile
   state rather than assuming the client error proves rollback.
4. Choose an owner-approved recovery: safely complete the pending idempotent
   change, restore an appropriate backup, or prepare a reviewed corrective path.
   Do not casually edit ledger rows to claim work ran.
5. Verify schema, data invariants, and ledger together before allowing deployment
   or replay. Ensure old and new application versions remain compatible.

An error after confirmed commit can leave both schema and ledger final.
`database.ErrAfterCommit` reports that boundary when produced by transaction
cleanup or hooks, but a step can itself return an error carrying the same marker.
Classify the origin and inspect state; a marker in a step's error is not proof
that its enclosing migration committed. The migrator does not reconnect or
replay after losing its retained session.

## MySQL Ledger

MySQL operations verify actual ID-column and index metadata under the retained
session and migration lock before user steps or history-row mutations. New ledgers
use `id VARBINARY(191) NOT NULL` with a full, single-column primary key on `id`.
ID matching is byte-exact, consistent with Go, rather than collation-based.

An existing character column is unsupported even with a `_bin` or NO PAD
collation. Fixed BINARY, nullable or generated ID columns, composite/prefix/
expression primary keys, and additional unique indexes are also rejected.
Nonunique secondary indexes are permitted. The library never automatically
ALTERs an existing ledger. Preparation can already issue `CREATE TABLE IF NOT EXISTS`
before rejecting a layout; rejection does not mean no SQL ran.

Column and index inspection failures preserve their cause. Missing or hidden
metadata is an inability to establish compatibility, not proof that storage
needs conversion. First verify the selected schema, metadata visibility, and
permissions. `SHOW FULL COLUMNS` and `SHOW INDEX` inspect the addressed target;
an unqualified configured table is resolved to the retained session's current
database before steps. Later `USE` statements cannot redirect bookkeeping.
ID insert/delete bindings use the original UTF-8 bytes.

### Authorized Repair

> [!WARNING]
> Changing a legacy ledger can corrupt migration identity or conceal applied
> work. Do not run a generic ALTER or delete history to bypass validation.
> Require explicit owner authorization and a reviewed, database-specific plan
> that preserves ID bytes and every history tuple.

Before an approved conversion, quiesce all migrators, ledger writers, and
conflicting administrator DDL. The migration lock coordinates cooperating
runners; it does not exclude arbitrary schema administration. Back up the full
schema and every `(HEX(id), batch, ran_at)` tuple.

Prove that stored ID bytes match the intended UTF-8 application identities,
fit within 191 bytes without truncation, and remain uniquely represented. Use
`OCTET_LENGTH`, not a character count, to check that byte limit.
Non-UTF-8 legacy storage can require an owner decision: physical bytes and
application identity cannot both be preserved by blindly changing a charset.
Never normalize, trim, rename, or discard IDs as an incidental conversion step.

Inspect constraints, triggers, generated values, indexes, and column attributes
before selecting the repair. Use strict SQL mode during the approved change.
There is no universally safe ALTER recipe; attributes and dependencies can
change what must be preserved or restated. The library performs no conversion,
and the ordinary migration role does not need new ALTER privileges for this check.

After repair, compare the row count and every saved history tuple byte-for-byte,
including full timestamp precision. Inspect canonical
column and primary-key metadata, and confirm there are no extra unique indexes.
Run `Status` with the original definitions and target to verify the protected
inspection succeeds. Reconcile any earlier failed nontransactional user steps
before authorizing `Up`: repairing the ledger does not undo or complete them.
Reconcile deployment records too: conversion cannot recover history already
lost or conflated under the old equality rules. Stop on conversion warnings or
changed tuples rather than accepting coercion or reconstructing timestamps.

For error identities and ledger behavior, see the
[`migrate` API reference](https://pkg.go.dev/arkoren.dev/core/service/database/migrate).

---

Source: https://arkoren.dev/docs/guides/validation/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/overview)

# Overview

A decoded request can have the correct Go type but still contain an empty email
or an unsupported plan. A validation **schema** defines which values your
application accepts. It can also prepare a cleaned, defaulted copy of the input.
Use the same schema for requests, configuration, CLI input, or imported records.

Install the standalone module from your application's module directory:

```sh
go get arkoren.dev/foundation/validate
```

Import `arkoren.dev/foundation/validate` as `validate`. Examples use an existing
`context.Context` named `ctx`. Focused fragments show error results without
repeating routine propagation; application code must check them before using
parsed input.

## Build a Schema

Start with the type your application already decodes. JSON tags determine
diagnostic names; schema registration uses Go field names.

```go
type Signup struct {
	Email string `json:"email"`
	Plan  string `json:"plan"`
}
```

Configure the accepted input inside `Struct`. Each callback receives a temporary,
typed **builder**, not the published schema:

```go
signup := validate.Struct(func(object *validate.StructBuilder[Signup]) {
	object.Field("Email", func(address *validate.StringBuilder[string]) {
		address.Trim()
		address.Required()
		address.Email()
	})
	object.Field("Plan", func(plan *validate.StringBuilder[string]) {
		plan.Default("free")
		plan.OneOf("free", "pro")
	})
})
```

Only registered fields are checked. Construction callbacks run once,
synchronously, in argument order; there is no separate `Build` call.
The published schema exposes `Parse`, `Validate`, and `JSONSchema`.
For nested input and shared field policies, use [Structs and Reuse](https://arkoren.dev/docs/guides/validation/structs-and-reuse.md).

## Parse Input

Use `Parse` when the application needs the prepared value rather than the
original input:

```go
input := Signup{Email: " ada@example.com "}
parsed, err := signup.Parse(ctx, input)
```

On success, `parsed.Email` is `"ada@example.com"` and `parsed.Plan` is `"free"`.
`input` is unchanged. On failure, `Parse` returns the zero result and an error,
not a partially accepted form.

If you only need a decision, call `signup.Validate(ctx, input)`. It performs the
same preparation and checks but discards the prepared value. Neither method
decodes JSON or converts an incompatible Go type.

## Choose Presence

Values are optional by default. Empty strings, zero numbers, false booleans,
nil pointers, and nil slices or maps count as absent and skip ordinary rules.
A zero `time.Time` is absent regardless of its location. `Required` rejects
absence, rather than merely requiring a JSON key.

For example, this schema rejects a missing quantity:

```go
quantity := validate.Int(func(count *validate.IntBuilder[int]) {
	count.Required()
	count.Min(1)
})
err := quantity.Validate(ctx, 0)
```

Without `Required`, zero would skip `Min(1)` and pass. Use a
[pointer field](https://arkoren.dev/docs/guides/validation/structs-and-reuse.md#explicit-zero-values) when zero is a supplied
value that must be checked. Non-nil pointees and collection entries are checked
even when zero. An allocated empty slice or map is present: `Required` accepts
it, while `Min(1)` rejects it. Nested value structs always check registered fields.

## Supply Defaults

Use `Default` when absence should select an application policy rather than fail:

```go
limit := validate.Int(func(size *validate.IntBuilder[int]) {
	size.Default(20)
	size.Between(1, 100)
})
size, err := limit.Parse(ctx, 0)
```

On success, `size` is 20. Defaults must satisfy their rules and must already be
in their final form: normalizers run before defaults, not again afterward.
`Required` and `Default` cannot coexist on one definition. `FileBuilder` has no
default setting. Slice and map defaults copy their outer containers, not
arbitrary nested objects; pointer defaults take a pointee value.
See [Normalizers](https://arkoren.dev/docs/guides/validation/normalizers.md) for preparation order.

## Share Schemas

Construct stable definitions once and share their published values across
goroutines. Publication detaches schema-owned rule containers; retained builders
are not runtime configuration objects.

This does not recursively clone arbitrary default objects or captured data.
Keep shared maps, pointers, catalogs, and callback state immutable or synchronize
access. Custom callbacks must not mutate data reachable from their input.

Bad definitions are programming errors, not rejected user input. `Struct`
aggregates recorded construction problems and panics with `ErrInvalidSchema`;
invalid standalone explicit schemas can panic when executed. See
[Definition Errors](https://arkoren.dev/docs/guides/validation/errors.md#definition-errors) before assembling dynamic rules.

## HTTP Input

With `request` imported from `arkoren.dev/core/http/request`, an HTTP handler can
decode and validate its body using the same schema:

```go
input, err := request.ParseJSON(r, signup)
```

Successful input includes normalization and defaults. Validation failures become
422 responses. If you validate separately, convert failures with
`request.ValidationProblem(err)`; raw `validate.Errors` has no HTTP status and
otherwise becomes a 500. Configure decoding and body limits through the
[request guide](https://arkoren.dev/docs/guides/http-fundamentals/requests.md).

Continue with [Scalar Rules](https://arkoren.dev/docs/guides/validation/available-methods.md) for constraints or
[Validation Errors](https://arkoren.dev/docs/guides/validation/errors.md) for structured diagnostics.
For all public contracts, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/validation/available-methods.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/available-methods)

# Scalar Rules

Rules reject input; they do not repair it. Choose them when an invalid value
must produce a diagnostic rather than a silently changed result. The
[overview](https://arkoren.dev/docs/guides/validation/overview.md) establishes construction and execution; examples import
`arkoren.dev/foundation/validate` and use an existing context named `ctx`.

Absent scalar values skip ordinary rules. Add `Required` to reject absence, or
use a [pointer](https://arkoren.dev/docs/guides/validation/structs-and-reuse.md#explicit-zero-values) when an explicit zero
must be checked. `String`, `Int`, `Uint`, `Float`, and `Bool` preserve named Go
types as well as built-in types. Without a callback to infer the type, write
`validate.String[string]()` or `validate.Int[int64]()`.

## Text Constraints

For a public display name, reject blank or overlong text without changing its
spelling:

```go
name := validate.String(func(text *validate.StringBuilder[string]) {
	text.Required()
	text.NotBlank()
	text.Max(80)
})
err := name.Validate(ctx, "Ada Lovelace")
```

This passes. Lengths count Unicode runes, not bytes. `NotBlank` rejects
whitespace-only text but does not trim it; [normalizers](https://arkoren.dev/docs/guides/validation/normalizers.md) own
deliberate transformations.

| Rule | Accepted supplied text |
| --- | --- |
| [`Min`, `Max`, `Len`](#text-constraints) | Minimum, maximum, or exact nonnegative rune count |
| [`NotBlank`](#text-constraints) | At least one non-whitespace character |
| [`OneOf`, `NotOneOf`](#text-constraints) | Membership or nonmembership in the supplied typed values |
| [`Regex`](#text-constraints) | Matches a supplied `*regexp.Regexp`; anchor it to match the whole string |
| [`Prefix`, `Suffix`, `Contains`](#text-constraints) | Has the supplied prefix, suffix, or substring |
| [`Lowercase`, `Uppercase`](#text-constraints) | Contains no uppercase or no lowercase letters; no rewriting |
| [`Alpha`, `AlphaNumeric`, `AlphaDash`](#text-constraints) | ASCII letters; letters/digits; or letters/digits/hyphens/underscores |
| [`ASCII`](#text-constraints) | ASCII code points only |

For an application plan, `plan.OneOf("free", "pro")` inside its string callback
accepts only those supplied values. There is no separate enum constructor.
`NotOneOf("admin", "root")` excludes reserved names. Negative length bounds
record [definition errors](https://arkoren.dev/docs/guides/validation/errors.md#definition-errors), not input violations.

## String Formats

A callback URL should use a scheme your application intends to handle:

```go
callback := validate.String(func(address *validate.StringBuilder[string]) {
	address.Required()
	address.URL("https")
})
err := callback.Validate(ctx, "https://example.com/events")
```

This passes. `URL` requires an absolute URL with a host. Its default schemes are
HTTP and HTTPS, compared case-insensitively. An explicit list replaces those
defaults and is copied when the setting is declared. An empty list selects the
defaults; an empty string entry never permits relative or scheme-relative URLs.
Syntax validation does not authorize a network destination: apply your own
destination policy before fetching an untrusted URL.

| Rule | What it checks |
| --- | --- |
| [`Email`](#string-formats) | Email syntax, not mailbox existence |
| [`URL`](#string-formats) | Absolute URL, host, and allowed scheme |
| [`UUID`](#string-formats) | UUID text |
| [`Domain`](#string-formats) | Domain name, not an entire URL |
| [`IP`](#string-formats) | IPv4 or IPv6 text |
| [`Duration`](#duration-values) | Go duration text accepted by `time.ParseDuration` |

These checks do not trim whitespace. Add `Trim` only when accepting and returning
cleaned text is intentional.

## Numeric Bounds

Use integral units when a quantity is naturally a count. A batch size can be
bounded without floating-point calculations:

```go
batch := validate.Int(func(count *validate.IntBuilder[int]) {
	count.Required()
	count.Between(1, 100)
	count.MultipleOf(5)
})
err := batch.Validate(ctx, 25)
```

This passes. Arguments use the schema's exact numeric type. `Int`, `Uint`, and
`Float` share these assertions:

| Rule | Supplied number must be |
| --- | --- |
| [`Min`, `Gte`](#numeric-bounds) | Greater than or equal to the bound |
| [`Max`, `Lte`](#numeric-bounds) | Less than or equal to the bound |
| [`Between`](#numeric-bounds) | Inside both inclusive bounds |
| [`Gt`, `Lt`](#numeric-bounds) | Strictly greater or strictly less |
| [`Positive`, `NonNegative`](#numeric-bounds) | Greater than zero or greater than/equal to zero |
| [`MultipleOf`](#numeric-bounds) | An exact multiple of a positive factor |
| [`OneOf`, `NotOneOf`](#numeric-bounds) | In or outside the listed typed values |

`MultipleOf(0)` panics immediately. Negative factors and non-finite floating
bounds or factors record sticky definition errors. Floating `MultipleOf` compares
exact decimal values from each Go float's shortest JSON spelling: `0.3` passes a
factor of `0.1`, without epsilon tolerance. Float32 retains its own encoding
precision. This cannot recover precision lost during an earlier calculation.

## Field Comparisons

To check confirmation input, compare the field with another exact-type field in
the nearest enclosing struct:

```go
type Passwords struct {
	Password string `json:"password"`
	Confirm  string `json:"confirm"`
}
passwords := validate.Struct(func(object *validate.StructBuilder[Passwords]) {
	object.Field("Confirm", func(text *validate.StringBuilder[string]) {
		text.Required()
		text.EqField("Password")
	})
})
```

A supplied confirmation must match `Password`. `NeField` checks inequality;
numeric builders additionally have `GtField`, `GteField`, `LtField`, and
`LteField`. Paths use Go names, not JSON names, and an unreachable target fails
the comparison. See [Field Paths](https://arkoren.dev/docs/guides/validation/conditions.md#field-paths) for nested scope.

## Boolean Intent

Agreement needs an affirmative value, not merely a non-nil field:

```go
terms := validate.Bool(func(consent *validate.BoolBuilder[bool]) {
	consent.Accepted()
})
err := terms.Validate(ctx, false)
```

This fails: `Accepted` requires true even though false normally counts as absent.
`Declined` rejects true. To require an explicitly supplied false, require a
pointer and apply `Declined` to its pointee.

## Time Windows

`Time` validates `time.Time`. `After` and `Before` are strict; `Between` includes
both bounds. `AfterField` and `BeforeField` compare other exact-type time fields.
For a future appointment, pin the clock in tests rather than sleeping:

```go
now := time.Date(2026, time.January, 1, 0, 0, 0, 0, time.UTC)
fixed := validate.NewClockContext(ctx, func() time.Time {
	return now
})
appointment := validate.Time(func(instant *validate.TimeBuilder) {
	instant.Required()
	instant.Future()
})
err := appointment.Validate(fixed, now.Add(time.Hour))
```

With the standard `time` package imported, this passes. Equality with the clock
fails both `Future` and `Past`. A zero instant is absent in every location.

## Duration Values

If your field remains a string, use `String` with `Duration()`. If your decoded
type needs a duration value that also passes through JSON, use
`validate.Duration` and `Text[validate.Duration]()`:

```go
var timeout validate.Duration
err := timeout.UnmarshalText([]byte("250ms"))
duration := timeout.Std()
text, err := timeout.MarshalText()
```

On success, `duration` is a standard `time.Duration` of 250 milliseconds and
`text` is `[]byte("250ms")`. Failed unmarshaling leaves the receiver unchanged.
Raw `time.Duration` has no representation in `encoding/json/v2`.
The wrapper's `JSONSchemaFormat()` returns `"go-duration"`; standard JSON Schema
`"duration"` uses a different grammar. `Text[T]` requires text marshaling and
unmarshaling on `*T`; it and `Any` offer common settings and custom checks,
not string-format methods.

## File Uploads

After multipart decoding, constrain the `*multipart.FileHeader` itself:

```go
avatar := validate.File(func(upload *validate.FileBuilder) {
	upload.Required()
	upload.Max(2 << 20)
	upload.Extensions("png")
	upload.Types("image/png")
})
```

Nil fails `Required`. `Min` and `Max` use the multipart parser's size in bytes.
`Extensions` ignores case and an optional leading dot. `Types` opens the file,
checks its first 512 bytes using `http.DetectContentType`, and accepts exact
types or wildcards with media-type parameters ignored. Open/read failures are
operational errors, not invalid-input violations.

Content detection is coarse: Office files detect as ZIP, and SVG can contain
scripts and is not treated as an image. There is no `Image` shorthand. Keep
antivirus checks, safe storage names, and
[multipart body limits](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#parse-uploads); these rules
do not replace them.

For every scalar method, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/validation/structs-and-reuse.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/structs-and-reuse)

# Structs and Reuse

A form can share an email policy with other forms while imposing its own
requirements. Use `Struct` to select the fields to validate and `Use` to import
an existing policy into a typed child builder. Neither operation infers rules
for unregistered fields.

Examples import `arkoren.dev/foundation/validate` and use an existing
`context.Context` named `ctx`. The [overview](https://arkoren.dev/docs/guides/validation/overview.md) covers execution.

## Select Fields

Register the Go name even when your JSON name differs:

```go
type Contact struct {
	EmailAddress string `json:"email"`
}
contact := validate.Struct(func(object *validate.StructBuilder[Contact]) {
	object.Field("EmailAddress", func(address *validate.StringBuilder[string]) {
		address.Required()
		address.Email()
	})
})
```

An email failure uses `/email`. The callback's builder must match the field's
exact Go type: a named string type requires that named type in `StringBuilder`.
Fields run in Go struct order, not registration order.

`Struct[Contact]()` is a valid schema with no registered fields. A zero
`StructSchema[Contact]` is uninitialized and invalid, including through `Ptr`.
Unknown or duplicate registrations, wrong callback types, and bad paths are
[definition errors](https://arkoren.dev/docs/guides/validation/errors.md#definition-errors).

## Reuse Policies

Define cleanup and syntax once, leaving requiredness to each form:

```go
email := validate.String(func(address *validate.StringBuilder[string]) {
	address.Trim()
	address.Email()
})
```

For the `Contact` type above, this alternative construction adds a requirement
without changing `email`:

```go
contact := validate.Struct(func(object *validate.StructBuilder[Contact]) {
	object.Field("EmailAddress", func(address *validate.StringBuilder[string]) {
		address.Use(email)
		address.Required()
	})
})
```

Pass schemas to `Use`, not `Field`. The same pattern works inside collection
`Keys` or `Values` and pointer `Value` callbacks.

If reuse concerns declaration settings rather than a published schema, supply
several callbacks to a constructor or child declaration:

```go
var policy validate.StringBuilderFunc[string] = func(text *validate.StringBuilder[string]) {
	text.Trim()
	text.Max(80)
}
name := validate.String(policy, func(text *validate.StringBuilder[string]) {
	text.Required()
})
```

Callbacks run once, synchronously, in argument order on the same fresh draft.
Every exported builder role has a matching `BuilderFunc` alias. An expanded nil
or empty callback slice makes no edits; an explicitly supplied nil function
panics, and application callback panics propagate.

## Nest Objects

A nested value object can declare its own selected fields inline:

```go
type Address struct {
	City string `json:"city"`
}
type Customer struct {
	Address Address `json:"address"`
}
customer := validate.Struct(func(object *validate.StructBuilder[Customer]) {
	object.Field("Address", func(address *validate.StructBuilder[Address]) {
		address.Field("City", func(city *validate.StringBuilder[string]) {
			city.Required()
		})
	})
})
```

`Customer{}` fails at `/address/city`: a nested value struct checks its fields
even when every field is zero. Make the address a pointer if nil should skip
the entire child schema.

Promoted fields behind a nil embedded pointer are checked as zero values.
Their `Required` rules can fail, but normalizers and defaults do not allocate
the missing intermediate pointer. Nested paths likewise never allocate merely
to reach a field.

## Explicit Zero Values

An update may omit its retry limit or explicitly set it to zero to disable
retries. A pointer preserves those two states:

```go
type Update struct {
	Retries *int `json:"retries"`
}
update := validate.Struct(func(object *validate.StructBuilder[Update]) {
	object.Field("Retries", func(pointer *validate.PtrBuilder[int]) {
		pointer.Value(func(count *validate.IntBuilder[int]) {
			count.Between(0, 10)
		})
	})
})
```

Nil passes without checking the pointee. A pointer to zero is supplied and
passes the range; a pointer to -1 fails. Add `pointer.Required()` to reject nil.
A plain pointer does not distinguish a missing JSON member from explicit null:
both decode to nil.

## Pointer Schemas

Use `Ptr` when a standalone policy already exists and you need its pointer form:

```go
count := validate.Int(func(quantity *validate.IntBuilder[int]) {
	quantity.Min(1)
})
quantity := validate.Ptr(count, func(pointer *validate.PtrBuilder[int]) {
	pointer.Required()
})
```

`quantity` accepts `*int`: nil fails presence and a pointer to zero fails the
minimum. `PtrBuilder.Default` takes a pointee value, not a pointer. Configure
pointee rules through `Value`; pointer-level checks receive `*T` instead of `T`.

## Reuse Boundaries

Ordinary `Use` requires the same concrete schema role and exact value type.
It accepts published values and non-nil pointers to them. An inferred schema
can contribute type-only documentation, but never missing child validation.
The decoded-JSON importer is a separate API, not an interchangeable typed schema.

Schema-level rules accumulate. Imported children cannot be overwritten: two
slice imports defining elements conflict, as does adding `Values` after importing
an element definition. Instead, use one local child callback and call `Use`
inside it. Map imports can contribute disjoint key and value definitions.
Repeated local child setters can replace local definitions, not imported ones.
Struct imports add registrations, so repeated fields remain errors.

A complete pointer schema can replace only a genuinely unconfigured scalar
adapter; it cannot discard a configured or composite pointee. For named slices
and maps, use the explicit [named-container adapter](https://arkoren.dev/docs/guides/validation/collections.md#named-containers).

Publication detaches owned rule containers, not arbitrary defaults or function
captures. Keep shared state immutable or synchronized, and never mutate
caller-owned nested data inside a callback.

For exact composition contracts, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/validation/collections.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/collections)

# Slices and Maps

A recipient list needs a minimum size and a valid address in every position.
Collection schemas separate these responsibilities: container rules check the
list or map, while `Values` and `Keys` configure its children. Assertions leave
the collection intact; use [Normalizers](https://arkoren.dev/docs/guides/validation/normalizers.md#clean-lists) if removing
or reordering entries is intentional.

Examples import `arkoren.dev/foundation/validate` and use an existing context
named `ctx`.

## Validate Entries

Require a nonempty list and check every supplied email address:

```go
recipients := validate.Slice(func(items *validate.SliceBuilder[string]) {
	items.Required()
	items.Min(1)
	items.Values(func(address *validate.StringBuilder[string]) {
		address.Email()
	})
})
err := recipients.Validate(ctx, []string{"ada@example.com"})
```

This passes. Nil fails `Required`; an allocated empty slice fails `Min(1)`.
Every entry is supplied, so an empty address is checked rather than skipped.
Entry diagnostics use indices such as `/0`. `Min`, `Max`, and `Len` count
entries, with inclusive bounds; they also apply to maps.

`Values` takes a concrete child callback, not a published schema. To reuse a
policy, call `address.Use(email)` inside that callback. Omitted child rules mean
no per-entry validation.

## Check Map Entries

An inventory import can constrain product codes separately from quantities:

```go
quantities := validate.Map(func(entries *validate.MapBuilder[string, int]) {
	entries.Keys(func(code *validate.StringBuilder[string]) {
		code.AlphaDash()
	})
	entries.Values(func(quantity *validate.IntBuilder[int]) {
		quantity.Min(1)
	})
})
err := quantities.Validate(ctx, map[string]int{"book": 0})
```

The supplied zero fails at `/book`. Map keys and values are checked even when
zero. Keys cannot normalize, default, or exclude: rewriting a key could change
its identity or cause a collision.

Traversal uses sorted diagnostic names. Encodable keys use their JSON-v2 object
names, including custom marshaling; other keys use `fmt.Sprint`. Names that
collide share a pointer without losing violations, but their relative order is
unspecified. Pointer segments escape `/` as `~1` and `~` as `~0`.

## Require Membership

For a map label whose existence matters regardless of its value, use `Has`:

```go
labels := validate.Map(func(entries *validate.MapBuilder[string, string]) {
	entries.Has("regulated")
})
err := labels.Validate(ctx, map[string]string{"regulated": ""})
```

This passes. `HasAny` requires at least one listed key and `HasAll` requires
every listed key. Empty key lists make `HasAny` false and `HasAll` true.

When the requirement concerns an entry's contents, use a predicate instead.
For a team roster, define the entry type once:

```go
type Member struct {
	ID    string `json:"id"`
	Admin bool   `json:"admin"`
}
```

Require at least one administrator without removing other members:

```go
team := validate.Slice(func(items *validate.SliceBuilder[Member]) {
	items.Contains(func(_ int, member Member) bool {
		return member.Admin
	})
})
```

A failed `Contains` belongs to the collection. `Every` instead requires each
entry to match and reports every failing entry path. Use `Values` when each
entry needs several rules or nested field diagnostics. Slice predicates take
`func(int, E) bool`; map predicates take the exact `func(K, V) bool`.

## Reject Duplicates

Use a projected identity when two records can differ but still name the same
account. This alternative team schema rejects repeated IDs:

```go
team := validate.Slice(func(items *validate.SliceBuilder[Member]) {
	items.UniqueBy(func(_ int, member Member) string {
		return member.ID
	})
})
```

The projection must return a comparable identity. Duplicate diagnostics identify
the later entry and its first conflict. Map `UniqueBy` takes `func(K, V) U`
and checks projected identities, not keys, which are already unique.

For plain comparable, non-interface slice elements, `Unique()` compares the
values directly. It rejects duplicates; `Dedupe()` deliberately removes them.
Maps have no `Unique` method.

## Count Matches

Total team size cannot express an exact administrator count. Count the matches
and constrain that actual integer:

```go
team := validate.Slice(func(items *validate.SliceBuilder[Member]) {
	items.CountWhere(func(_ int, member Member) bool {
		return member.Admin
	}, func(count *validate.IntBuilder[int]) {
		count.Min(2)
		count.Max(2)
	})
})
```

Exactly two administrators pass; a failed count rule is attached to the
collection. Zero matches is supplied and is checked. Map predicates receive
their exact key and value types.

Count callbacks accept pure numeric assertions: `Min`, `Max`, `Between`, `Gt`,
`Gte`, `Lt`, `Lte`, `Positive`, `NonNegative`, `MultipleOf`, `OneOf`, and
`NotOneOf`. They cannot default, normalize, require, exclude, reference fields,
add conditions, or run `Check`. `Bail` and `Describe` do not change the count.

## Empty Collections

Optional nil collections skip their own assertions. Allocated empty collections
are present: `Contains` fails, `Every` and uniqueness pass, `CountWhere` checks
zero, and `Has` fails. Add both `Required` and `Min(1)` when neither nil nor
empty is acceptable.

Predicates run during validation, not declaration. They must be pure and safe
for concurrent use. Child preparation precedes container normalizers, then
container assertions and child checks run. Diagnostics use the resulting
indices. Arbitrary Go predicates are not exported to JSON Schema; `Unique`
can export `uniqueItems`.

## Named Containers

A named slice does not have the exact type of `[]string`. Adapt the source
schema explicitly rather than converting your application field:

```go
type Names []string
names := validate.Slice(func(items *validate.SliceBuilder[string]) {
	items.Values(func(name *validate.StringBuilder[string]) {
		name.Trim()
		name.Min(1)
	})
})
named := validate.Named[Names](names)
parsed, err := named.Parse(ctx, Names{" Ada "})
```

On success, `parsed` retains type `Names` and contains `"Ada"`. For a struct
field, use `func(values *validate.NamedBuilder[Names])` and `values.Use(names)`.
This also supports named maps, including eager foundation collection types.

Underlying container types must be identical: there are no element/key
conversions or scalar adaptation. Source callbacks receive ordinary containers,
and field comparisons retain their exact-type constraints. Configure all source
rules before attachment; a second `Use` cannot replace an attached schema.

`Named` exports logical source constraints, not a custom marshaler's wire shape.
In particular, builtin-byte slices encode as base64 strings, while slice rules
describe elements. A defined `uint8` element encodes as an array instead. Use
[type inference](https://arkoren.dev/docs/guides/validation/json-schema.md#describe-output) for output descriptions, not
runtime collection validation.

For every collection assertion and adapter, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/validation/conditions.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/conditions)

# Conditions

A delivery form needs an address for shipping, but not for pickup. A
**condition** observes input without changing it or producing a violation.
The field builder chooses the effect: require, prohibit, exclude, or add rules.
Conditions read the nearest enclosing struct, so start with a
[struct schema](https://arkoren.dev/docs/guides/validation/structs-and-reuse.md).

Examples import `arkoren.dev/foundation/validate`. Construction callbacks run
once; their recorded observations run for each input.

## Require a Field

Define a delivery form whose address requirement depends on its mode:

```go
type Delivery struct {
	Mode    string `json:"mode"`
	Address string `json:"address"`
	Email   string `json:"email"`
	Phone   string `json:"phone"`
}
delivery := validate.Struct(func(object *validate.StructBuilder[Delivery]) {
	object.Field("Address", func(address *validate.StringBuilder[string]) {
		address.RequiredIf(func(condition *validate.ConditionBuilder) {
			condition.Eq("Mode", "shipping")
		})
	})
})
```

An empty address fails for shipping. Otherwise it is optional, not prohibited.
`RequiredUnless` reverses the condition. Multiple condition callbacks configure
one fresh condition draft, with observations combined using AND.

## Presence Dependencies

If contact details need at least one channel, configure the phone field with
`phone.RequiredWithout("Email")`. Here `phone` is its
`*validate.StringBuilder[string]` inside the `Delivery` declaration.

| Setting | Requires this field when |
| --- | --- |
| `RequiredWith("Email", "Address")` | Any listed field is present |
| `RequiredWithAll("Email", "Address")` | Every listed field is present |
| `RequiredWithout("Email", "Address")` | Any listed field is absent |

These shortcuts take string paths, not condition callbacks. Presence follows
typed rules: zero scalars and nil pointers/containers are absent; non-nil
pointers to zero and allocated empty collections are present. Zero time is
absent regardless of location. This is not JSON-key existence.

## Prohibit or Exclude

For pickup, decide whether an entered address is a mistake or stale form state.
To report it as a mistake, add this setting inside the address callback:

```go
address.ProhibitedIf(func(condition *validate.ConditionBuilder) {
	condition.Eq("Mode", "pickup")
})
```

A supplied address now fails for pickup. `ProhibitedUnless` allows a supplied
value only when its condition matches.

If you deliberately want to discard stale state instead, use this alternative:

```go
address.ExcludeIf(func(condition *validate.ConditionBuilder) {
	condition.Eq("Mode", "pickup")
})
```

Matching input skips the field's later rules and `Parse` clears the value in
the result. Exclusion is not rejection and does not modify the caller's input.
Do not use it when the presence of that data must be reported.

## Add Branch Rules

Some modes need stronger constraints while retaining ordinary rules. Within
the same string address callback, add a minimum only for shipping:

```go
address.Max(200)
address.When(func(condition *validate.ConditionBuilder) {
	condition.Eq("Mode", "shipping")
}, func(text *validate.StringBuilder[string]) {
	text.Min(10)
})
```

The maximum applies to every supplied address; the minimum applies only in
the branch. `When` takes one condition callback followed by callbacks for the
same concrete builder role. Supply settings, not a published schema. Branch
callbacks configure the schema once and are not rerun per input.

## Combine Observations

An explicit group expresses alternatives. For an approval field beside string
fields `Plan` and `Role`, use an OR group:

```go
approval.RequiredIf(func(condition *validate.ConditionBuilder) {
	condition.Or(func(group *validate.ConditionBuilder) {
		group.Eq("Plan", "enterprise")
		group.Eq("Role", "admin")
	})
})
```

Here `approval` is its string field builder. Either comparison requires
approval. Ungrouped observations use AND; `And` makes that grouping explicit,
and `Not` complements its nested group. Empty conditions and empty explicit
groups are definition errors.

`Eq` and `NotEq` compare a selected field with an assignable value. `In` and
`NotIn` compare against a captured value list. `Present` and `Absent` observe
typed presence. For logic clearer in Go, configure a typed predicate. Within
the delivery address callback, this requires an address for shipping when email
is absent:

```go
address.RequiredIf(func(condition *validate.ConditionBuilder) {
	condition.If(func(value Delivery) bool {
		return value.Mode == "shipping" && value.Email == ""
	})
})
```

The predicate argument must exactly match the nearest struct type. It is a pure
observation: it cannot normalize data or return an operational error.

## Field Paths

Paths use dot-separated Go names, such as `"Billing.Country"` for a field
`Billing *Address` whose address has a string `Country`. They walk structs and
pointers, including promoted fields, but cannot index collections, select
parents, or use JSON tag names. Inside an address child schema, paths begin at
that address rather than its outer form.

Bad paths and incompatible comparison types are construction errors. A nil
intermediate pointer is unreachable and is never allocated. `Eq`, `NotEq`,
`In`, and `NotIn` all return false for an unreachable target. Use an explicit
`Not` group if the missing case must count as the complement. Reachable scalar
zeros are compared normally.

`Present` is false for an unreachable path; `Absent` is true. Field assertions
such as `EqField` use the same resolver but fail when the target is unreachable,
rather than comparing an invented zero.

## Observe Collections

To require approval when any team member is an administrator, assume the
enclosing form has `Members []Member`, where `Member` contains `Admin bool`:

```go
approval.RequiredIf(func(condition *validate.ConditionBuilder) {
	condition.Contains("Members", func(_ int, member Member) bool {
		return member.Admin
	})
})
```

Slice predicates take the exact `(int, E)` types; maps take `(K, V)`. Their
results observe the collection, not validate its entries:

| Observation | Condition is true when |
| --- | --- |
| `Has`, `HasAny`, `HasAll` | A map has the selected key, any listed key, or all listed keys |
| `Contains` | At least one entry matches |
| `Every` | Every entry matches |
| `UniqueBy` | Comparable projected identities are unique |
| `CountWhere` | The actual matching count satisfies pure integer constraints |

Each takes the field path first. `CountWhere` uses the same restricted
[count rules](https://arkoren.dev/docs/guides/validation/collections.md#count-matches) as a collection assertion.

A reachable nil collection is observed as empty, unlike an optional
collection's own assertions: `Contains` is false; `Every` and `UniqueBy` are
true; count rules check zero. Nil maps lack every key. Empty key lists make
`HasAny` false and `HasAll` true. An unreachable path instead returns false for
all collection observations, including `Every` and `HasAll`.

## Effect Order

Conditions see unconditional normalization and defaults in reachable validated
children before field rules run, regardless of Go field order. A sibling with
`Trim` is observed after trimming, not as raw text.

Conditional `When` and `ExcludeIf` effects remain sensitive to declaration and
field execution order. They do not repeatedly evaluate until every dependent
field settles. Avoid mutually dependent branches that assume later effects have
already happened. An [object check](https://arkoren.dev/docs/guides/validation/custom-rules.md#object-invariants) sees
unconditional preparation, but is not a final hook after conditional effects.

Conditions and arbitrary predicates are runtime-only and omitted from JSON
Schema. Keep predicates read-only and concurrency-safe.

For exact settings and observations, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/validation/normalizers.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/normalizers)

# Normalizers

An imported tag such as `" Go "` may be usable if you store it as `"go"`.
A **normalizer** makes that deliberate replacement before validation. Use an
assertion instead when the original spelling or range must be rejected.

`Parse` returns the prepared value; `Validate` checks it and discards it.
Neither modifies caller-owned input. Examples import
`arkoren.dev/foundation/validate` and use an existing context named `ctx`.

## Clean Text

Accept surrounding whitespace and mixed case for imported tags:

```go
tag := validate.String(func(text *validate.StringBuilder[string]) {
	text.Trim()
	text.ToLower()
	text.Required()
})
parsed, err := tag.Parse(ctx, " Go ")
```

On success, `parsed` is `"go"`. `Lowercase` would instead reject uppercase
letters without changing them.

| Setting | Replacement |
| --- | --- |
| `Trim()` | Remove surrounding Unicode whitespace |
| `ToLower()`, `ToUpper()` | Change case |
| `TrimPrefix(prefix)`, `TrimSuffix(suffix)` | Remove one matching prefix or suffix |
| `Truncate(n)` | Keep at most n runes |

These preserve the builder's exact string type. Truncation does not report a
length violation: use `Max` to reject long text. Zero truncates to empty;
negative lengths are invalid definitions.

## Adjust Numbers

For an import policy that intentionally caps percentages rather than rejecting
out-of-range input, use `Clamp`:

```go
percentage := validate.Int(func(number *validate.IntBuilder[int]) {
	number.Clamp(0, 100)
})
parsed, err := percentage.Parse(ctx, 120)
```

The result is 100 on success. `Max(100)` would reject 120 instead. Signed,
unsigned, and floating builders support inclusive `Clamp`; reversed bounds
panic. Signed and floating builders also support `Abs`; the minimum signed
integer stays unchanged because its positive counterpart cannot fit.

Floating builders additionally support decimal rounding:

```go
measurement := validate.Float(func(number *validate.FloatBuilder[float64]) {
	number.Round(2)
})
parsed, err := measurement.Parse(ctx, 12.345)
```

The result is 12.35. Halves round away from zero; negative decimal places panic.
Rounding changes the quantity and cannot restore precision already lost. Do not
apply it when the original precision is part of the contract.

## Normalize Time

To store an accepted timestamp with UTC location, import the standard `time`
package for your input and configure `Time`:

```go
timestamp := validate.Time(func(instant *validate.TimeBuilder) {
	instant.UTC()
})
```

Parsing changes its location to UTC without changing the instant or original
value. Time assertions still compare instants.

## Clean Lists

Normalize entries before removing empty tags and duplicates:

```go
tags := validate.Slice(func(items *validate.SliceBuilder[string]) {
	items.Values(func(tag *validate.StringBuilder[string]) {
		tag.Trim()
		tag.ToLower()
	})
	items.RejectZero()
	items.Dedupe()
	items.Min(1)
})
parsed, err := tags.Parse(ctx, []string{" Go ", "", "go", "Queues"})
```

The result is `[]string{"go", "queues"}` on success. Entry cleanup happens
first, so deduplication sees identical `"go"` values. Diagnostics use the cleaned
slice's indices, not original import-row positions.

`Sort` takes `func(E, E) int`, such as `cmp.Compare[int]` from the standard
`cmp` package. `Filter` keeps entries matching `func(int, E) bool`; `Reject`
drops matches. `RejectZero` drops zero entries. `Dedupe` keeps the first
occurrence and requires comparable, non-interface elements.

Filtering is not `Every`, and deduplication is not `Unique`: the former change
data while the latter [report invalid collections](https://arkoren.dev/docs/guides/validation/collections.md). Maps have
custom `Normalize`, but no built-in sort/filter/deduplication operations. Map
keys never normalize, default, or exclude because that could change identity.

## Custom Replacement

For an application-specific decoration, return a same-type replacement from
`Normalize`. With the standard `strings` package imported:

```go
name := validate.String(func(text *validate.StringBuilder[string]) {
	text.Normalize(func(input string) string {
		return strings.TrimPrefix(input, "@")
	})
})
```

Parsing `"@ada"` produces `"ada"`. Every explicit rule builder supports typed
`Normalize`, including structs, pointers, maps, files, text values, and dynamic
values. Named-container adapters instead attach a configured source schema.

There is no error return. Use a [custom check](https://arkoren.dev/docs/guides/validation/custom-rules.md) if an operation
can fail. Keep callbacks concurrency-safe and do not mutate nested maps, slices,
or pointers reachable from the input: arbitrary captured/default objects remain
application-owned, not recursively cloned.

## Preparation Order

Unconditional normalizers run in their own declaration order before presence,
defaults, and ordinary rules. Placing `Trim` after `Required` in the declaration
does not make presence run first. Whitespace-only text becomes absent after
trimming, so a requirement fails or a default applies.

Defaults are not normalized again at the value where they are assigned. Supply
them in final form. Nested children prepare before enclosing conditions; slice
elements prepare before slice normalizers, and pointer pointees and map values
likewise prepare before their enclosing checks. Each normalizer runs once along
that path. Container assertions and child validation then see prepared values.

Conditional transformations remain [order-sensitive](https://arkoren.dev/docs/guides/validation/conditions.md#effect-order).
Map-key rules, condition observations, and `CountWhere` count constraints cannot
normalize or default values: they test identity or actual observations rather
than rewrite the data deciding a rule.

For exact transformation contracts, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/validation/custom-rules.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/custom-rules)

# Custom Rules

A username can satisfy every format rule and still be reserved by your
application. Add that policy with `Check`, whose callback receives the prepared
value and caller's context. Use built-in rules when they already express the
constraint; custom checks are for application logic, not a replacement schema
implementation.

Examples import `arkoren.dev/foundation/validate` and the standard `context`
package. The [overview](https://arkoren.dev/docs/guides/validation/overview.md) covers schema execution.

## Check a Value

Return nil to accept a value or `validate.Fail` to report a violation:

```go
username := validate.String(func(name *validate.StringBuilder[string]) {
	name.Trim()
	name.Required()
	name.Check(func(_ context.Context, value string) error {
		if value == "admin" {
			return validate.Fail("username.reserved", nil)
		}
		return nil
	})
})
```

`" admin "` fails at the root pointer with code `username.reserved` after
trimming. Add that code to your [message catalog](https://arkoren.dev/docs/guides/validation/errors.md#translate-messages).
Optional absent scalars skip ordinary checks; `Required` above handles absence.

`Fail` accepts a parameter map for message placeholders. Parameters need to be
JSON-serializable if you encode diagnostics; the helper does not encode or
validate them. Do not include secrets merely to improve an error message.

Every other returned error is operational: validation stops and returns it
unchanged. If a storage lookup fails, return that failure rather than reporting
that the username is unavailable. A failed dependency does not establish a fact
about user input. Pass the callback's context to storage and network calls.

## Stop Failed Checks

Put cheap syntax rules before an expensive custom check and configure `Bail`
when a malformed value must not reach that check. Inside the same typed builder,
`name.Bail()` stops this value after its first violation.

Rules otherwise collect violations in declaration order. `Bail` is a setting,
not a step whose position changes its effect; it does not stop unrelated sibling
fields. An operational error stops the whole validation regardless of `Bail`.

## Object Invariants

For a relationship involving multiple fields, a struct-level check can select
the field that needs correction. Define a booking interval:

```go
type Interval struct {
	Start int `json:"start"`
	End   int `json:"end"`
}
```

Attach a failed ordering check to the Go field name `End`:

```go
interval := validate.Struct(func(object *validate.StructBuilder[Interval]) {
	object.Check(func(_ context.Context, value Interval) error {
		if value.End < value.Start {
			return validate.At("End", "interval.order", nil)
		}
		return nil
	})
})
```

The diagnostic uses `/end`. `At` takes a Go field name, not a JSON Pointer;
an unknown name produces an error wrapping `ErrInvalidSchema` during execution.
For this simple comparison, `GteField("Start")` on an end-field builder is the
smaller [built-in alternative](https://arkoren.dev/docs/guides/validation/available-methods.md#field-comparisons). Use an
object check when the invariant needs broader application logic.

An object check sees unconditional normalization and defaults, but runs before
its child field rules. It does not see their later `When` or `ExcludeIf` effects
and is not a final post-validation hook. Keep the invariant independent of those
effects, and do not assume a child has already passed its own constraints.

## Context and Ownership

Checks run for every applicable input, unlike construction callbacks. Published
schemas support concurrent calls, so checks and captured dependencies must be
safe for concurrent use. Do not mutate the input or objects reachable from it.
Validation is not authorization, and checking a value does not make a later
database mutation atomic with that check.

Cancellation is observed between normalizers, rules, and child callbacks.
Pass the context through operations that can block; the engine cannot interrupt
an arbitrary application callback. Cancellation discards partial diagnostics
and parsed results. An operational error that already aborted execution retains
precedence over cancellation.

`Fail` and `At` errors can be reused. They snapshot supported parameter
containers, and the engine detaches them again when publishing violations.
Pointers and opaque objects remain shared; follow the full
[diagnostic ownership rules](https://arkoren.dev/docs/guides/validation/errors.md#diagnostic-ownership) rather than assuming
a general deep copy.

For all custom-check contracts, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/validation/errors.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/errors)

# Validation Errors

A form needs field-level correction advice, while a failed lookup needs
operational handling. Validation keeps those outcomes separate. Rule failures
return `validate.Errors`, a map from JSON Pointers to violation records; custom
checks can instead return their operational error unchanged.

Examples import `arkoren.dev/foundation/validate` and use an existing context
named `ctx`. Standard `errors` and `fmt` imports are named where used.

## Classify Failures

After `schema.Validate` or `schema.Parse`, use the standard `errors` package to
extract violations rather than treating every error as invalid input:

```go
var violations validate.Errors
if errors.As(err, &violations) {
	// Use violations to identify input that needs correction.
}
```

Other errors include dependency failures and cancellation. Do not translate
them into field advice. Success returns nil, never an empty `validate.Errors`.
`Parse` returns the zero typed result on failure; the decoded-JSON importer
returns nil. Neither result is partially accepted input.

HTTP handlers should use the [request validation adapter](https://arkoren.dev/docs/guides/validation/overview.md#http-input)
instead of returning raw `validate.Errors`, which has no HTTP status.
For strict byte checks, also distinguish malformed JSON and
[resource refusals](https://arkoren.dev/docs/guides/validation/encoded-json.md#classify-failures).

## Read Violations

A `Violation` has a stable `Code`, optional `Params`, and a localized `Message`.
Match codes in application logic and show messages to people. With `fmt`
imported, iterate in deterministic path order:

```go
for pointer, failures := range violations.All() {
	for _, failure := range failures {
		fmt.Println(pointer, failure.Code, failure.Message)
	}
}
```

The root pointer is `""`; `/items/0/quantity` selects a nested value. Pointer
segments escape `/` as `~1` and `~` as `~0`. Struct paths use JSON-v2 field names,
although declarations and `At` use Go names. `All` orders digit-only segments
numerically, so `/items/2` precedes `/items/10`.

`Error()` is a deterministic English summary, not a wire format or the current
locale's message representation. Do not parse it. Pointers and parameters may
reveal private member names or application data; redact diagnostics before
exposing them beyond their intended audience.

## Translate Messages

Keep codes stable across languages. Configure messages on a context rather
than changing global state:

```go
catalog := validate.Catalog{
	Default: "en",
	Locales: map[string]validate.Messages{
		"en": {"username.reserved": "This username is reserved."},
		"es": {"username.reserved": "Este nombre de usuario está reservado."},
	},
}
ctx = validate.NewCatalogContext(ctx, catalog)
ctx = validate.NewLocaleContext(ctx, "es")
```

Checks using this context render Spanish messages for `username.reserved`
without changing its code. Templates interpolate `{param}` placeholders from
the violation parameters. To change a built-in message, start from the mutable
copy returned by `validate.English()` and edit the relevant code.

Lookup tries the selected locale, its base language, the catalog default and
its base language, built-in English, then the code. `LocaleFromContext(ctx)`
returns the selected locale and whether one exists; it does not negotiate a
language. Keep catalog maps immutable while requests read them. HTTP
[locale middleware](https://arkoren.dev/docs/guides/http-capabilities/localization.md#select-languages) supplies per-request context.

For existing diagnostics, `localized := violations.Localize(ctx)` returns a
copy with rerendered messages. Codes, locations, and parameter values remain
the same; an unknown code retains its existing message. `Describe` is unrelated:
it annotates JSON Schema and never overrides a violation message.

## Add Diagnostics

If application code already knows a field-level failure, initialize the map
before appending it:

```go
failures := make(validate.Errors)
failures.Add("/email", validate.Violation{
	Code:    "email.taken",
	Message: "This email address is already registered.",
})
```

`Add` takes a JSON Pointer; [custom checks](https://arkoren.dev/docs/guides/validation/custom-rules.md#object-invariants)
use `At` with a Go field name. `Add` does not copy supplied parameters, and
`All` yields borrowed slices and parameters. Neither creates a snapshot.

## Diagnostic Ownership

`Fail` and `At` capture their parameter containers. The engine detaches supported
containers again before rendering/publishing violations, including built-in and
imported enum/const parameters. `Localize` also copies them, even when an unknown
code keeps its existing message. Mutating a copied parameter container does not
rewrite a reusable failure, schema, or another result.

Copying traverses acyclic slices, arrays, and maps with string-kind keys,
including nested containers reached through interfaces. Named types and
nil-versus-empty distinctions are preserved. Cycles along these copied
containers are unsupported and are not detected.

This is not JSON conversion or a general object deep copy. Struct fields,
pointers, and non-string-keyed maps are not traversed; opaque native objects,
functions, and captures remain application-owned. Copying invokes no custom
marshaling methods. Keep shared opaque data immutable or synchronized, and do
not mutate borrowed diagnostics concurrently with readers. Parameters need to
be JSON-serializable if you encode them; the helpers do not enforce that.

## Definition Errors

A malformed schema is a programming error, not user input needing correction.
`ErrInvalidSchema` identifies these failures, with different reporting stages:

| Operation | Invalid definition outcome |
| --- | --- |
| `Struct` construction | Aggregates recorded problems after callbacks, then panics |
| Standalone explicit `Parse` or `Validate` | Panics during compilation |
| `JSONSchema` export | Returns nil bytes and an error |
| `Infer` inspection, execution, or export | Returns its inference error rather than panicking |
| JSON Schema import | Returns a construction error |

These failures wrap `ErrInvalidSchema`. Recorded problems include missing or
duplicate fields, wrong types or paths, `Use(nil)`, negative string lengths or
multiple factors, and non-finite floating bounds/factors. They survive later
valid settings, reuse, and publication. Aggregated struct diagnostics sort by
Go field name, retaining declaration order for ties.

Some failures are immediate: `MultipleOf(0)` panics at the setter before later
callbacks run. Supplied nil functions and application callback panics also
propagate; they are not collected as ordinary registration problems.

Construct composite schemas with constructors. A zero `StructSchema[T]` remains
invalid even through `Ptr`; use `Struct[T]()` for a valid empty registration.
Constructed slices/maps may deliberately omit child rules. Struct construction
checks registered defaults against their rules, whereas export does not execute
those checks or prove default validity. Check export errors before publishing
documents; JSON encoding failures preserve the encoder's cause instead of
pretending every failure is a bad rule.

For exact error and localization contracts, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/validation/json-schema.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/json-schema)

# JSON Schema

Clients and tools may need an input description without executing Go rules.
`JSONSchema` exports the representable constraints of your typed schema.
`Infer` instead describes a Go output type, while `FromJSONSchema` constructs
a validator for decoded JSON. These are different tasks, not a lossless
round trip. For exact document bytes without defaulting, use
[Encoded JSON](https://arkoren.dev/docs/guides/validation/encoded-json.md).

Examples import `arkoren.dev/foundation/validate` and use an existing context
named `ctx`.

## Export Rules

Describe a display name alongside its runtime bounds:

```go
name := validate.String(func(text *validate.StringBuilder[string]) {
	text.Min(3)
	text.Max(80)
	text.Describe("Public display name")
})
document, err := name.JSONSchema()
```

On success, the document describes a string, both length bounds, and the
description. `Describe` sets metadata, not an error message. Export is
deterministic with sorted object keys and a 2020-12 subset; it does not add
provider-specific strict mode or `additionalProperties: false`.

Check `err` before publishing bytes. Export checks executable definitions,
including omitted runtime-only and map-key rules. Invalid definitions return
nil bytes and an error wrapping `ErrInvalidSchema`; encoding failures preserve
the JSON encoder's cause. Export never runs custom checks, normalizers,
condition/collection predicates, or declaration callbacks. It does not verify
defaults against their rules or normalize default annotations.

## Understand Export Limits

Keep the Go schema authoritative for application validation. Conditions,
cross-field comparisons, custom checks, normalizers, and predicate collection
rules are omitted. Not every time, string, file, or map-key constraint can be
represented. `URL` does not preserve its allowed scheme list; `Unique` can
export `uniqueItems`, but `UniqueBy` cannot export a Go projection.

Even representable settings need not round-trip. Runtime rules accumulate,
while repeated annotations use the last value. Go zeros can be absent where
decoded JSON zeros are checked. An exported default may violate emitted
constraints and fail import even though export succeeded. A Go regexp is
copied verbatim, but RE2-only syntax may differ from another validator's
ECMA-262 interpretation. Check downstream import and validation results.

## Describe Output

A response type needs shape metadata without input-presence rules. Use `Infer`:

```go
type Result struct {
	Name    string            `json:"name"`
	Elapsed validate.Duration `json:"elapsed"`
	Note    string            `json:"note,omitempty"`
}
result := validate.Infer[Result]()
document, err := result.JSONSchema()
```

The document describes `Elapsed` as a string with format `go-duration`.
Ordinary fields are listed as required unless omitted by `omitempty`,
`omitzero`, or a nil-capable embedded owner. This is descriptive: inferred
`Validate` adds no value checks, and `Parse` returns its input unchanged rather
than copying arbitrary objects. Both still respect cancellation.

Inference follows default JSON-v2 type encoding: text-marshaled values are
strings, `time.Time` is `date-time`, builtin-byte sequences are strings, and
ordinary structs, arrays, slices, maps, and pointers compose. Defined `uint8`
elements become array elements rather than builtin bytes. Numeric fields tagged
`string` become strings; fields with a `format` option remain `{}` because
formatter-specific encodings are not modeled. Recursive types stop at a repeated
occurrence with `{}`, not `$ref`.

Custom JSON marshalers remain opaque `{}` unless a named container explicitly
declares its native shape through `JSONSchemaType() reflect.Type`. Eager
foundation `collection.Slice` and `collection.Map` implement that protocol;
lazy collections do not. A provider must deterministically return its identical
unnamed container type independent of receiver contents. Nil, mismatched results,
or provider panics become inference errors.

Raw `time.Duration`, channels, functions, complex numbers, and unsupported map
keys cannot be inferred. `result.Err()`, execution, and export return an error
wrapping `ErrInvalidSchema` for unsupported types. Shape metadata neither proves
that a particular value encodes nor models every encoder option. For runtime
named-container constraints, use [Named](https://arkoren.dev/docs/guides/validation/collections.md#named-containers).

## Import a Contract

Use `FromJSONSchema` when your input is already decoded JSON rather than a typed
Go form. A small contract can bound a list of tags:

```go
tags, err := validate.FromJSONSchema([]byte(`{
	"type": "array",
	"maxItems": 10,
	"items": {"type": "string", "minLength": 1}
}`))
```

Check construction errors before using the validator. With `tags` successfully
constructed, validate decoded values directly:

```go
err = tags.Validate(ctx, []any{"go", "queues"})
```

This passes. Use `Parse` to retain a copied result with defaults. The input is
never modified. A zero imported validator is not usable.

The importer accepts a deliberate subset: single-string `type`; `properties`,
`required`, schema-valued `additionalProperties`, and `items`; `enum`, `const`,
`default`, `description`, `format`, and `pattern`; string-length, numeric,
array-size, and object-size bounds; `multipleOf`; and `uniqueItems`. It rejects
unknown keywords, `$schema`, `$ref`, compositions such as `oneOf`, boolean
schemas, type unions, and malformed keyword values with `ErrInvalidSchema`.
The error identifies the keyword's JSON Pointer. A pattern must be a string:
empty is valid, null is not. Patterns use Go regular expressions.

Input types are `map[string]any`, `[]any`, `string`, `float64`, `bool`, and nil.
Go `int`, `int64`, and valid numeric `encoding/json/jsontext.Value` tokens also
count as numbers. This is not reflection-based validation of arbitrary structs.
Integer schemas reject fractions; keywords without `type` apply only to their
own kinds.

## Decoded Presence

Imported application validation deliberately differs from strict JSON Schema
instance checking. `required` checks whether an object key exists before
defaults: null, empty text, and zero satisfy key existence. Null is otherwise
absent and satisfies every constraint regardless of `type`; `Parse` replaces it
with a default when one exists. Missing properties can also receive defaults,
but a missing required key still fails. Zero scalars are supplied, so
`minLength: 1` rejects an empty string.

The importer asserts `date-time` (RFC 3339), `email`, `uri` (absolute, with host,
any scheme), `uuid`, `go-duration` (`time.ParseDuration`), and standard `duration`
(RFC 3339 Appendix A). The duration grammars are not interchangeable. Other
formats annotate only. The separate encoded validator treats all formats as
annotations, so choose the boundary deliberately.

## Precision and Uniqueness

Imported numeric literals retain exact decimal values and their original spelling
for comparisons and rerendering. Numeric defaults and diagnostic parameters use
`float64` only when its JSON spelling preserves the value; otherwise they use
owned `jsontext.Value` tokens. Supplied Go numbers retain their concrete types.
Already rounded floating-point input cannot recover lost digits.

`uniqueItems` uses exact structural keys rather than pairwise comparison.
Object key order is irrelevant, array order matters, and numbers compare by
exact decimal value at every nesting level, including equivalent spellings and
signed zero. Uniqueness checks original items before defaults: parsed defaults
can make previously distinct items equal. Unsupported values or non-finite
numbers inside an item report `type` at its pointer.

For fixed-size scalar entries, expected uniqueness work/storage is linear;
nested objects add encoded-size and key-sorting work. Rerendering sorts object
keys while preserving numeric spelling; it is not RFC 8785 canonicalization.

## Bound Resource Use

For untrusted contracts or decoded input, independently limit document bytes,
numeric digits/exponents, collection sizes, nesting, and concurrent validations.
Input must be finite and acyclic. The decoded validator has no implicit global
work or memory budget; compilation has no context parameter. Context cancellation
alone is not a resource policy.

Use `maxItems` when the array cardinality is known. After type admission, an
oversized array reports only `slice.max` for that array and skips uniqueness
and child traversal; sibling values are still checked.

Pass the caller's context to execution. Traversal, equality, uniqueness, sorting,
and copying observe cancellation, which returns `ctx.Err()` and discards
accumulated violations; `Parse` returns nil. Individual scalar operations and
allocations are not preempted. Use [Encoded JSON](https://arkoren.dev/docs/guides/validation/encoded-json.md) when fixed
byte-validation budgets and exact wire semantics are the required boundary.

For the full export, inference, and import contracts, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/validation/encoded-json.md
[HTML documentation](https://arkoren.dev/docs/guides/validation/encoded-json)

# Encoded JSON

An emitted document must satisfy its contract as written. Adding a missing
member or rounding its number before checking would approve a different
document. Use `FromJSONSchemaStrict` to compile a contract, then validate the
original bytes. This is separate from [decoded application validation](https://arkoren.dev/docs/guides/validation/json-schema.md#import-a-contract),
which permits defaults and treats null as absent.

Import `arkoren.dev/foundation/validate`. Examples use an existing
`context.Context` named `ctx`.

## Compile a Contract

Require a nonnegative integer count while retaining a default as documentation:

```go
check, err := validate.FromJSONSchemaStrict([]byte(`{
	"type": "object",
	"required": ["count"],
	"properties": {
		"count": {"type": "integer", "minimum": 0, "default": 1}
	}
}`))
```

Check the construction error before using `check`. The published `EncodedSchema`
owns its compiled state, supports concurrent use, and retains no caller bytes,
application callbacks, or mutable defaults. Editing the schema byte slice
afterward cannot change it. Its zero value is invalid.

## Check Exact Bytes

With a successfully compiled check, validate a count beyond binary64's exact
integer range:

```go
err = check.Validate(ctx, []byte(`{"count":9007199254740993}`))
```

This passes without changing the document. A missing count fails `required`;
null fails the integer type; zero is supplied and passes the minimum. Defaults
never fill missing values or replace null, and default annotations need not
satisfy their schema. There is no `Parse` method and this check does not
implement typed `Schema[T]`.

Bounds, integer assertions, `multipleOf`, and nested `enum`, `const`, and
`uniqueItems` compare exact decimal values. A maximum of `9007199254740992`
rejects `9007199254740993`; a fractional token cannot round into an integer.
Equivalent spellings such as `1e3` and `1000` compare equally. Numbers such as
`1e1000` work within the budgets below, even outside `float64` range.

No byte check can recover precision already lost before encoding a Go float.
Keep an exact application representation or raw numeric tokens when those
digits are part of the contract.

## Choose Vocabulary

The strict compiler accepts self-contained draft-07 and 2020-12 documents.
Omitted or empty `$schema` selects 2020-12. Boolean schemas, type unions and
null, compositions and conditional branches, object/array constraints, and
local references/anchors are supported. Draft-07 `$ref` ignores assertion
siblings; 2020-12 evaluates them. Mixed embedded dialects fail.

References resolve only inside the supplied document: the compiler never
fetches a URL or reads a file. References must select compiled schema locations,
not arbitrary annotation values. Recursive schemas that descend through the
instance work; unproductive cycles produce a resource failure.

Unknown keywords, unsupported dialects/types, malformed keyword values, and
unresolved references fail compilation with `ErrInvalidSchema`. `x-*` extensions
are annotations. Patterns use Go's RE2 syntax, not the complete ECMA-262 dialect.
Consult the [strict vocabulary contract](https://pkg.go.dev/arkoren.dev/foundation/validate#FromJSONSchemaStrict)
before importing contracts intended for another validator.

Formats, defaults, and content keywords annotate rather than assert formats or
decoded content. `contentSchema` is compiled for validity and references but
does not inspect decoded content. No custom check or normalizer executes. Run
typed business validation separately when the application needs it; strict wire
validation does not replace application policy.

## Classify Failures

An instance mismatch returns fail-fast `validate.Errors` with a JSON Pointer
and keyword code, without instance values or schema literals. Do not assume
the typed validator's full localized messages or exhaustive violation collection.
Other failures remain distinct:

| Failure | Identity |
| --- | --- |
| Invalid UTF-8/JSON, duplicate names, trailing documents | `ErrEncodedJSON` |
| Resource refusal | `ErrEncodedLimit`, never `validate.Errors` |
| Cancellation or deadline | The context error |
| Invalid compiled definition or zero schema | `ErrInvalidSchema` |

Using the standard `errors` package inside a function returning `error`, classify
a mismatch without swallowing operational failures:

```go
var violations validate.Errors
if errors.As(err, &violations) {
	return errors.New("output does not match its contract")
}
return err
```

Returned pointers can reveal private member names. Redact them before exposing
diagnostics to an untrusted client. Do not describe resource or cancellation
failures as instructions to correct a field.

Checking output after a mutation cannot undo that mutation. A failed check is
not proof of rollback and does not justify automatic replay. The
[managed MCP adapter](https://arkoren.dev/docs/guides/mcp/tools.md#schema-limits) uses this
strict boundary and separates operational failures from mismatches.

## Apply Budgets

Both compilation and validation reject excess rather than round numbers or
accept incomplete work:

| Budget | Limit |
| --- | --- |
| Schema or instance document | 8 MiB each |
| Container nesting | 256 levels |
| Compiled schema nodes | 16,384 |
| Decoding work and evaluation work | 1,000,000 steps each |
| Nested evaluation calls | 512 |
| Numeric coefficient | 4,096 digits |
| Explicit decimal exponent | -4096 through 4096, including for zero |
| Sum of coefficient digits plus absolute exponents | 64 × 1024 × 1024 per decoded document |
| Equality/uniqueness structural-key encoding | 64 MiB cumulatively per validation |
| URI/reference and traversed-pointer lengths | 16 KiB |
| Compiled URI-index keys, including aliases | 32 MiB |

Numeric limits are checked before rational allocation. Structural-key budgets
include repeated comparisons and conservatively charged tags/length prefixes.
These are data/work limits, not exact heap limits: copies, capacity growth, and
runtime metadata also use memory. Keep application request-size and concurrency
limits as well.

Cancellation is checked between tokens, visits, comparisons, branches, and
references, not within bounded standard-library operations such as regex or
rational arithmetic. Compilation has no context parameter and performs no
lifecycle I/O. Compilation resource failures also retain `ErrEncodedLimit`.

For the full vocabulary, error identities, and limits, see the
[`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/cache/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/cache/overview)

# Cache Overview

A cache lets requests reuse a report or an upstream response instead of rebuilding
it each time. Treat cached values as replaceable copies: an entry can disappear
through expiry, eviction, deletion, or backend loss. No expiry means no scheduled
expiration, not guaranteed retention.

The `*cache.Cache` facade encodes Go values as JSON; its driver stores the bytes.
These examples assume an existing facade named `store` and an operation context
`ctx`. Choose and configure that store in [Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md).
Use `cache` from `arkoren.dev/core/service/cache`, plus standard-library `errors`
and `time`. Focused snippets keep error results visible but omit routine error
propagation; your application must handle those errors.

## Reading Values

Suppose your application builds a monthly sales report. Give its cached copy a
small, JSON-compatible shape:

```go
type Report struct {
	Month        string `json:"month"`
	Orders       int64  `json:"orders"`
	RevenueCents int64  `json:"revenue_cents"`
}
```

A typed key supplies the exact storage name, value type, and write lifetime.
Declare a ten-minute copy of March's report:

```go
key := cache.NewKey[Report]("reports.v1:2026-03", func(entry *cache.KeyBuilder) {
	entry.TTL(10 * time.Minute)
})
```

Creating this definition performs no I/O. [Cache Keys](https://arkoren.dev/docs/guides/cache/keys.md) explains literal
identities, namespaces, and reusable expiry policies. Sharing a key definition
helps readers and writers agree; its Go type does not isolate backend entries.

To publish a freshly built report, write it through the facade:

```go
err := store.Put(ctx, key, Report{
	Month:        "2026-03",
	Orders:       318,
	RevenueCents: 924500,
})
```

`Put` replaces both the value and its expiry. A key without a TTL writes without
expiry. Values must round-trip through `encoding/json/v2`; use exported fields
and JSON-compatible members, not channels or functions. Keep large blobs in
[file storage](https://arkoren.dev/docs/guides/file-storage/overview.md), rather than a cache entry.

Read the report and distinguish ordinary absence from a storage or decoding
failure:

```go
report, err := store.Get(ctx, key)
if errors.Is(err, cache.ErrNotFound) {
	// Load a replacement from the application's source of truth.
}
```

On success, `report` is a `Report`. A missing, expired, or evicted entry returns
an error wrapping `cache.ErrNotFound`. An existing value with incompatible JSON
returns a decoding error and remains stored. Writes capture a separate JSON
representation; reads decode a separate value. Mutating either your input or a
decoded result does not modify the stored copy.

To inspect expiry, use `remaining, err := store.TTL(ctx, key)`: zero means an
existing entry has no expiry; absence returns `cache.ErrNotFound`. This differs
from `key.TTL()`, which reports the declared write policy. Reads do not renew
expiry. `Has` checks existence without decoding, but cannot guard a later write.
`Delete` removes an entry and succeeds if it is already absent.

## Remembering Values

When every miss should rebuild the report, `Remember` combines lookup and
computation. Here `buildReport` is your application's loader returning
`(Report, error)`:

```go
report, err = store.Remember(ctx, key, func() (Report, error) {
	return buildReport(ctx, "2026-03")
})
```

A hit returns the decoded report without calling the loader. A miss runs the
loader and stores its result using the key's TTL. Undecodable stored JSON also
triggers rebuilding and replacement. A schema or meaning change that still
decodes successfully does not; version the physical key name or invalidate it.

Loader failure stores nothing and returns the error. If the loader succeeds but
encoding or storing its result fails, `Remember` returns that computed value
together with the error. Read or lock-acquisition failures return before the
loader runs, so an arbitrary error does not imply a usable value. Choose an
explicit fallback policy if the application must serve while the cache is down.
The computed result remains application-owned and is returned without cloning;
successful storage captures its JSON representation.

Concurrent misses sharing a backend coordinate through a ten-second
[lease](https://arkoren.dev/docs/guides/cache/locks.md). After acquiring it, each caller reads again before computing;
hits never acquire the lease. Waiting ends when `ctx` ends. Independent memory
drivers do not coordinate, even if their keys have identical names.

> [!WARNING]
> `Remember` does not renew its lease or guard the final write. A loader that
> outlasts ten seconds can overlap another loader and overwrite a newer result.
> Keep computation idempotent; move long refresh work to a schedule instead of
> relying on this lease for correctness.

After acquisition, release is best effort with a separate ten-second cleanup
timeout. Cleanup retains context values but ignores request cancellation and
deadlines; release errors are ignored because the lease expires. Return can
therefore occur after the request deadline. The cleanup timeout bounds I/O only
when the driver honors its context, and it does not bound the loader.

## Consuming Values

A one-time download grant must not be returned to two concurrent requests.
`Pull` reads and removes an entry in one atomic driver operation. With an
existing `cache.Key[string]` named `grantKey`, consume it once:

```go
grant, err := store.Pull(ctx, grantKey)
if errors.Is(err, cache.ErrNotFound) {
	// Reject the missing, expired, or already consumed grant.
}
```

For one stored entry, one concurrent caller receives its bytes; later pulls
observe absence. Removal happens before JSON decoding, so invalid JSON still
consumes the entry. A lost network response can also leave the entry consumed
without delivering the value to your application. This atomic consumption is
not a transaction with the action authorized by the grant.

## Claiming Entries

To reserve a short-lived refresh marker, use `Add` rather than a separate `Has`
and `Put`. Here `marker` is an existing `cache.Key[bool]` with a bounded TTL:

```go
claimed, err := store.Add(ctx, marker, true)
if err != nil {
	return err
}
if !claimed {
	return nil // Another caller already holds the marker.
}
```

`true, nil` means this call stored the value; `false, nil` means an existing
unexpired entry and its TTL were left untouched. This claims an entry, not
completion of subsequent work. A crash after claiming leaves the marker until
expiry or deletion. For work that must not be lost, use a
[queue job](https://arkoren.dev/docs/guides/queues/overview.md) with an idempotent handler instead.

## Guarded Writes

If another caller can retire or replace a report while an edit is pending,
check the observed value and write the replacement atomically. `CompareAndPut`
uses an exact raw-byte guard, not decoded JSON equality. Read those bytes before
preparing your replacement:

```go
raw, err := store.Driver().Get(ctx, key.Name())
if err != nil {
	return err
}
```

Raw access bypasses facade validation and telemetry. Do not decode and re-encode
the guard: the representation could change. Once your application has prepared a
`Report` named `replacement`, submit it only while those bytes still match:

```go
applied, err := store.CompareAndPut(ctx,
	cache.Match{Key: key.Name(), Value: raw}, key, replacement)
if err != nil {
	return err // A transport failure can leave the outcome unknown.
}
if !applied {
	return nil // Discard or reload the edit; never fall back to Put.
}
```

On success, the destination value and TTL are replaced together. A missing,
expired, or byte-different guard returns `false, nil` without changing either
entry or its TTL. The guard can name a different entry; a distinct guard and its
expiry remain untouched. An existing empty value can match empty bytes, but an
absent guard never matches. Invalid destination definitions fail before I/O,
including on a mismatch.

Unlike a lease, this check protects the cache write itself. It is not a monotonic
version: deleting and recreating identical guard bytes allows an old update to
match again (the **ABA problem**). Use a nonrepeating application version when
that distinction matters. Do not restore retired security records. Before using
guards for session retirement, follow the
[session writer rollout requirements](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#save-changes)
so older unconditional writers cannot bypass rejection.

`Match` and raw `Entry` descriptors expose names and bytes without redaction.
Their slices are borrowed during the call, never modified or retained afterward;
do not mutate them concurrently or log sensitive contents. A custom driver must
perform comparison and writing atomically or return `cache.ErrUnsupported`.
Name-transforming wrappers must transform both guard and destination. Network
atomicity is not exactly-once execution or durability; Redis specifically avoids
replaying a possibly executed guarded write after an uncertain response.

## Shared Infrastructure

Use [counters](https://arkoren.dev/docs/guides/cache/counters.md) for atomic integer updates and [locks](https://arkoren.dev/docs/guides/cache/locks.md) to
reduce duplicate work. [Sessions](https://arkoren.dev/docs/guides/http-capabilities/sessions.md) and
[rate limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) also depend on the cache:
losing their entries logs visitors out or resets request allowances. Independent
replica-local caches keep independent sessions and allowances.

Choose a separate backend or Redis database when state needs independent sizing
or clearing. A facade name only labels telemetry, and namespaces do not provide
bulk-clear or isolation boundaries. Do not share a session cache with entries
your application intends to clear indiscriminately. [Monitoring](https://arkoren.dev/docs/guides/cache/monitoring.md)
explains operation spans, hit ratios, and privacy boundaries.

For tests, the [memory driver](https://arkoren.dev/docs/guides/cache/drivers.md#memory) needs no server. Its clock works
with [`testing/synctest`](https://arkoren.dev/docs/guides/testing/time.md), so expiry tests need not sleep in
real time. Custom drivers can run the shared
[conformance suite](https://arkoren.dev/docs/guides/testing/conformance-suites.md), `cachetest.Run`.

For every function and option, see the [`cache` API reference](https://pkg.go.dev/arkoren.dev/core/service/cache).

---

Source: https://arkoren.dev/docs/guides/cache/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/cache/drivers)

# Cache Drivers

Choose memory when independent process-local copies are acceptable. Choose Redis
when replicas must read the same entries, share request allowances, or coordinate
work. A separate memory driver has separate state even inside the same process;
it does not become shared by giving its facade the same name.

Both drivers expose the same [typed cache operations](https://arkoren.dev/docs/guides/cache/overview.md). Differences
in state sharing, capacity, network failures, and persistence still affect your
application. Install modules from your application's Go module directory. If
core is not installed yet, run `go get arkoren.dev/core`.
Examples use `cache` from `arkoren.dev/core/service/cache`, standard-library
`time`, and an operation context `ctx`. Focused snippets keep error results
visible but omit routine propagation; handle those errors in your application.

## Compatibility Matrix

Atomicity below applies to one operation on the active backend, not exactly-once
effects across retries. Shared state means callers use the same entries,
counters, and leases. Restart columns describe state retention, not guaranteed
availability or durability.

| Driver | Shared State | Expiry | Counters | Atomic Pull | Guarded Writes | Leases | App Restart | Backend Restart |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [Memory](#memory) | No — Only callers sharing one driver instance coordinate. | Yes | Yes — Checked int64 arithmetic under one mutex. | Yes | Yes — Guard and destination share one mutex. | Yes — Within one driver instance. | No | No |
| [Redis](#redis) | Yes — Same server, database, and physical names. | Yes | Yes — Exact int64 results; counter TTLs round up to milliseconds. | Yes — Requires Redis 6.2 or later. | Yes — One Lua execution; positive TTLs round up to milliseconds. | Yes — Expiring token-owned entries. | Yes — Application restart does not clear Redis. | Partial — Depends on server persistence and recovery settings. |

Typed keys and namespaces are facade definitions, not native storage partitions.
Their [literal names and expiry policies](https://arkoren.dev/docs/guides/cache/keys.md) work with both drivers.
Neither provider promises retention for entries without expiry. Redis can evict
or lose data; memory loses all data when its process exits.

Both providers reject counter arithmetic outside `int64` without changing the
target or TTL or creating an absent target. Error identities are backend-native.
Both support atomic `Pull`, `IncrementWithTTL`, and same-key or different-key
`CompareAndPut`. A [lease](https://arkoren.dev/docs/guides/cache/locks.md) can expire while its holder still works;
none of these capabilities makes later application side effects exactly once.

## Memory

Memory is included in `arkoren.dev/core`; no additional driver module is required.
Use it for tests, local development, or replaceable data that each process can
keep independently.

### Creating Stores

Import the bundled driver and wrap it with the facade:

```go
import "arkoren.dev/core/service/cache/memorycache"

driver := memorycache.New()
store := cache.New(driver)
```

Construction cannot fail and starts no goroutines. The store is immediately
usable; follow the [report write and read](https://arkoren.dev/docs/guides/cache/overview.md#reading-values) with it.
Expired entries are never returned, regardless of whether cleanup has run.
Raw driver byte inputs and outputs are copied, so callers can mutate their own
slices without changing stored bytes.

### Expiry Cleanup

The driver has no size limit or capacity-based eviction. Give replaceable values
a TTL and account for live entries when sizing the process. By default, writes
sweep abandoned expired entries at most once per minute. To change that interval,
use an alternative construction:

```go
driver := memorycache.New(func(store *memorycache.Builder) {
	store.Cleanup(5 * time.Minute)
})
```

Zero selects the one-minute default. A negative interval disables both inline
and background sweeping; expired entries are then removed only when accessed.
Cleanup reclaims memory but does not change expiry correctness.

If an idle cache must reclaim expired entries without subsequent writes, register
the optional janitor. With an existing `composer` of type
`*application.Composer` from `arkoren.dev/core/application`, register ownership
before startup:

```go
composer.Cleanup(driver.Stop)
composer.Startup(driver.Start)
```

`Start` runs the janitor at the cleanup interval; `Stop` ends it. Without `Start`,
there is no janitor to stop. See [Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md)
for registration and shutdown ordering.

### Atomic Updates

Use the shared [counter](https://arkoren.dev/docs/guides/cache/counters.md), [guarded-write](https://arkoren.dev/docs/guides/cache/overview.md#guarded-writes),
and [lease](https://arkoren.dev/docs/guides/cache/locks.md) examples with this store. Their atomicity applies only to
callers sharing this driver instance. Two replicas with independent memory caches
each receive their own rate-limit allowance and can each acquire the same named
lease.

Counter overflow wraps `strconv.ErrRange` and leaves the value and expiry
unchanged. Subtraction supports `math.MinInt64` directly without overflowing
negation; do not assume memory's overflow error identity for other drivers.

For every function and option, see the [`memorycache` API reference](https://pkg.go.dev/arkoren.dev/core/service/cache/memorycache).

## Redis

Redis keeps entries outside the application process. Use it for shared sessions,
counters, and cached results that should outlive an application deploy. You need
network access to a Redis 6.2+ server; `Pull` uses `GETDEL`, which requires that
version. The driver wraps a standalone `*redis.Client`, not a Redis Cluster client.
Adding application replicas does not increase Redis capacity or configure high
availability.

### Installing Redis

Install the driver module:

```sh
go get arkoren.dev/driver/redis
```

This installs the Go adapter, not the Redis server. For a disposable development
server, use [Running Locally](#running-locally).

### Connecting Stores

Point the driver at your existing service, then construct the facade:

```go
import "arkoren.dev/driver/redis/rediscache"

driver := rediscache.New(func(store *rediscache.Builder) {
	store.Addr("cache.internal:6379")
})
store := cache.New(driver)
```

`New` returns a driver without error or network I/O. Defaults are TCP,
`localhost:6379`, database `0`, empty ACL credentials, and no TLS. `Network("unix")`
instead treats `Addr` as a socket path. An empty network or address restores
its default after callbacks.

With an existing `*application.Composer` named `composer`, register cleanup and
startup:

```go
composer.Cleanup(driver.Stop)
composer.Startup(driver.Start)
```

`Start` pings Redis so connection and authentication failures reject startup.
Use `driver.Ping` for a later [readiness check](https://arkoren.dev/docs/guides/application/health-and-readiness.md).
`Stop` closes the client pool, even with a canceled cleanup context; it does not
wait for application work. Finish cache users before closing their driver.
The same server, logical database, and physical names let other processes read
your [typed writes](https://arkoren.dev/docs/guides/cache/overview.md#reading-values).

### Securing Connections

For an authenticated TLS service, take credentials from application configuration,
not source code. This alternative constructor assumes an existing password
named `password` and standard-library `crypto/tls`:

```go
driver := rediscache.New(func(store *rediscache.Builder) {
	store.Addr("cache.internal:6379")
	store.Username("reports")
	store.Password(password)
	store.DB(1)
	store.TLS(&tls.Config{MinVersion: tls.VersionTLS12})
})
```

Configure certificate trust for your service; do not disable verification to
make a connection succeed. Native TLS values remain shared, not deep-cloned.
The diagnostic `rediscache.Config` is not constructor input. Its `String`,
`GoString`, and structured logging methods redact the password, but reading its
`Password` field exposes the secret. Unlike owned cache keys, it does not promise
redaction for every unsupported format verb.

### Native Clients

If you need native timeouts, retry settings, or pool configuration beyond the
builder, construct the client first. Import `redis` from
`github.com/redis/go-redis/v9`:

```go
client := redis.NewClient(&redis.Options{
	Addr:                  "cache.internal:6379",
	ContextTimeoutEnabled: true,
	ReadTimeout:           2 * time.Second,
	WriteTimeout:          2 * time.Second,
	MaxRetries:            -1,
})
driver := rediscache.NewFrom(client)
```

This variant disables native command retries and transfers ownership of the
initialized, non-nil client. `NewFrom` has no construction callbacks, makes no
connection, and preserves client settings. `Stop` closes that exact client;
do not share it with code that must outlive the driver, and arrange one shutdown
owner.

`New` uses native transport timeout/retry defaults and enables
`ContextTimeoutEnabled` so operation deadlines apply during socket I/O.
`NewFrom` does not enable that option for you. Use bounded operation contexts;
context cooperation does not interrupt arbitrary application loaders or force
noncooperative drivers to stop. Neither facade construction nor its telemetry
name adds a request timeout.

Ordinary native retry policies can repeat effects after uncertain network
failures. `CompareAndPut` specifically disables replay of possibly executed
guarded writes, even if native retries are enabled. A lost reply is an error
with an unknown outcome, not proof that no write occurred or permission to
fall back to an unconditional write.

### Native Access

For connectivity diagnostics outside the portable contract, use the underlying
client without changing stored data:

```go
status, err := driver.Client().Ping(ctx).Result()
```

On success, `status` is `"PONG"`. The native client shares the driver's owned
pool. If you only have the facade, `store.DriverAs[*rediscache.Driver]()` returns
the first matching driver through wrappers that expose `Unwrap() cache.Driver`;
check its boolean result before use. `store.Driver()` exposes raw portable
operations with literal names, bytes, and explicit TTLs.

Native and raw calls bypass facade telemetry, typed-key validation, and JSON
encoding. Do not overwrite entries with incompatible encodings or write into
the reserved `lock:` key space. Configure native options before ownership
transfer, not by mutating a running client's options.

### Storage Limits

An application restart does not clear Redis. Backend restart survival depends
on Redis persistence and recovery settings; do not assume survival. Expiry,
eviction, deletion, failover, and restoration can affect both cached values and
coordination records. Atomicity on the active server is not durable retirement
of a security record across restored earlier data.

Counter scripts return decimal text to preserve the entire `int64` range.
`Decrement` with `math.MinInt64` uses one exact-decimal script because native
`DECRBY` rejects that delta even for representable results. Native overflow errors
leave the target value and expiry untouched. The adapter does not emulate
arithmetic with client-side read/check/write calls.

`TTL` reads millisecond `PTTL` directly, retaining subsecond lifetimes. Zero
means no expiry; missing or expired entries return `cache.ErrNotFound`. Remaining
lifetimes beyond `time.Duration` saturate at its maximum. Positive
`IncrementWithTTL` and guarded-write TTLs round up to whole milliseconds without
overflowing; counter lifetime reporting also saturates.

Guarded writes compare and replace with one Lua execution, including distinct
guard and destination keys. Leases acquire with `SET NX` and release or extend
with token-checking scripts. These protect individual cache operations, not
external side effects. Review [counter retry limits](https://arkoren.dev/docs/guides/cache/counters.md#handling-failures)
and [lease safety](https://arkoren.dev/docs/guides/cache/locks.md) before relying on coordination.

### Running Locally

With Docker and Compose installed, save this as `compose.yaml` in a new empty
development directory. Do not overwrite an existing application configuration.
The unauthenticated service binds only to loopback; never expose it to an
untrusted network:

```yaml
services:
  cache:
    image: redis:7.4-alpine
    ports:
      - "127.0.0.1:6379:6379"
```

Start it and verify connectivity:

```sh
docker compose up -d
docker compose exec cache redis-cli ping
```

The check prints `PONG`; retry it if Redis is still starting. Use
`rediscache.New()` for the default local address, start the driver, and follow
the [write and read example](https://arkoren.dev/docs/guides/cache/overview.md#reading-values). When finished, run
`docker compose down`. This disposable configuration does not preserve cache
data.

For every function and setting, see the [`rediscache` API reference](https://pkg.go.dev/arkoren.dev/driver/redis/rediscache).

---

Source: https://arkoren.dev/docs/guides/cache/keys.md
[HTML documentation](https://arkoren.dev/docs/guides/cache/keys)

# Cache Keys

Readers and writers must agree on which entry holds a product card and how long
a new copy should live. A **key** is an immutable definition of that entry's
exact name, Go value type, and write lifetime. A **namespace** supplies a reusable
literal prefix and policy for keys of one type. Neither definition allocates
storage, contacts a driver, or belongs to a particular cache facade.

Use `cache` from `arkoren.dev/core/service/cache` and standard-library `time` in
these examples. [Cache Overview](https://arkoren.dev/docs/guides/cache/overview.md) owns executing operations with
the resulting keys.

## Typed Definitions

Suppose a catalog caches the public fields needed to render a product listing.
Declare the value shape separately from its storage identity:

```go
type ProductCard struct {
	SKU        string `json:"sku"`
	Name       string `json:"name"`
	PriceCents int64  `json:"price_cents"`
}
```

Choose a feature-owned name with a version when incompatible meanings need
separate entries:

```go
card := cache.NewKey[ProductCard]("catalog.v1:sku-482")
```

The resulting `cache.Key[ProductCard]` addresses exactly `catalog.v1:sku-482`.
`NewKey` does not split, escape, trim, normalize, or add separators. Even an
explicit empty name is valid. A zero-value key is not valid.

The type supplies compile-time guidance for facade calls, not a backend schema.
Two keys with different Go types or TTLs but the same completed name address the
same stored entry. If a schema change still decodes successfully, JSON decoding
cannot detect its changed meaning; version or invalidate that identity explicitly.

## Expiry Policies

Without configuration, a key has TTL zero: writes have no expiry. To expire a
product card five minutes after a write, declare that policy at construction:

```go
card := cache.NewKey[ProductCard]("catalog.v1:sku-482", func(entry *cache.KeyBuilder) {
	entry.TTL(5 * time.Minute)
})
```

This is an alternative to the preceding declaration. The TTL applies to `Put`,
successful `Add` and `CompareAndPut`, and a computing `Remember`. Reading through
a key never changes an entry's expiry. `card.TTL()` reports the policy, whereas
`store.TTL(ctx, card)` reports a stored entry's remaining lifetime.

No expiry is not a retention guarantee: eviction, deletion, or backend loss can
still remove the entry. [Counters](https://arkoren.dev/docs/guides/cache/counters.md) have separate expiry rules, and
[leases](https://arkoren.dev/docs/guides/cache/locks.md) take their own explicit duration.

If one write needs a shorter lifetime at the same physical name, derive another
definition:

```go
brief := card.Derive(func(entry *cache.KeyBuilder) {
	entry.TTL(30 * time.Second)
})
```

`brief` keeps `card`'s exact name; `card` retains its five-minute policy.
Derivation changes no stored data. The new policy matters only on a later write.
Deriving with `TTL(0)` similarly declares a no-expiry variant without modifying
the original definition.

## Shared Namespaces

When all catalog cards share a prefix and lifetime, declare the policy once:

```go
cards := cache.NewNamespace[ProductCard]("catalog.v1", func(namespace *cache.NamespaceBuilder) {
	namespace.TTL(5 * time.Minute)
})
```

Without configuration, namespaces use separator `":"` and TTL zero. Obtain a
typed key by supplying a nonempty child name:

```go
card := cards.Key("sku-482")
```

`card.Name()` is exactly `catalog.v1:sku-482`; the dot in the prefix is untouched.
A callback on `Key` can override the inherited TTL for that definition. Child
names are strings: convert numeric IDs explicitly with standard-library `strconv`,
for example `cards.Key(strconv.Itoa(id))`.

Namespaces only produce ordinary keys. They do not partition storage, authorize
access, create a bulk-clear boundary, or isolate two entries with identical
completed names. `Name()`, `Separator()`, and `TTL()` reveal the declared prefix,
future join string, and write policy—not current backend state.

## Nested Policies

If archived product cards use a longer lifetime, create a subnamespace:

```go
archive := cards.Subnamespace("archive", func(namespace *cache.NamespaceBuilder) {
	namespace.TTL(time.Hour)
})
archived := archive.Key("sku-482")
```

`archived.Name()` is `catalog.v1:archive:sku-482`. The child inherits its parent's
type, separator, and TTL before applying edits. Here the child uses one hour,
while `cards` keeps five minutes.

`Subnamespace` joins its own name with the parent's separator. A child separator
edit affects only later joins, not the already completed prefix. To keep the same
prefix but change future policy, derive a namespace instead:

```go
briefCards := cards.Derive(func(namespace *cache.NamespaceBuilder) {
	namespace.TTL(30 * time.Second)
})
```

Future keys from `briefCards` inherit thirty seconds; existing keys and entries
remain unchanged. Changing a derived namespace's separator similarly affects
future joins, never existing prefix bytes. Published definitions are detached
from their construction drafts, so retaining a builder cannot mutate them.

## Identity Safety

Literal joining can collide. Prefix `a:b` with child `c` and prefix `a` with
child `b:c` both produce `a:b:c`. An explicit empty separator means concatenation,
not the default. Validate untrusted components against an allowed alphabet or
encode each component with an unambiguous application-owned scheme before
joining. A namespace is not an authorization boundary.

Reserve physical entry names starting with `lock:` for the facade, and logical
lease names starting with `remember:` for `Remember`. Its coordination entry is
`lock:remember:<completed-key>`. These are conventions in one backend key space;
an ordinary write to a completed lock name can overwrite a lease. Check the
completed identity after all namespace joins.

### Preserving Names

If a deployed namespace uses the former dot separator, preserve it explicitly:

```go
legacy := cache.NewNamespace[ProductCard]("catalog.v1", func(namespace *cache.NamespaceBuilder) {
	namespace.Separator(".")
})
```

`legacy.Key("sku-482")` addresses `catalog.v1.sku-482`, not the current default
`catalog.v1:sku-482`. Existing entries are not renamed or read through aliases.
Standalone `NewKey` names remain unchanged.

During a rollout, keep readers, writers, deletions, and `Remember` on the same
completed identity. `Remember` also derives its lease name from that identity.
For security-sensitive entries such as webhook receipts, changing identities
can admit replays: preserve old names for their required retention period or
plan an explicit migration. Built-in session records and rate-limit counters
already select `":"` explicitly and need no migration for this default change.

### Inspecting Identities

Owned `Key` and `Namespace` definitions redact themselves in ordinary formatting,
Go-syntax formatting, and structured logging. Unsupported or mismatched `fmt`
verbs such as `%d` also format a redacted marker rather than reflecting private
names; quoting, hexadecimal formatting, width, and precision never expose the
identity.

`Name()` deliberately returns the literal identity. That string, raw driver
arguments, and raw `Match` or `Entry` descriptors are not redacted. Do not log
them indiscriminately: identities can contain personal identifiers or secrets.
[Monitoring](https://arkoren.dev/docs/guides/cache/monitoring.md#privacy-boundaries) explains the separate facade
telemetry boundary.

## Definition Errors

If policy comes from configuration, check the published definition before
serving requests:

```go
if err := cards.Err(); err != nil {
	return err
}
```

Keys expose the same `Err()` method. Zero-value keys and namespaces, empty
namespace names, and empty `Key` or `Subnamespace` child names wrap
`cache.ErrInvalidKey`. This does not prohibit the explicit empty standalone
identity accepted by `NewKey`. Negative TTLs wrap `cache.ErrInvalidTTL`.

The first declaration error survives derivation and child creation, even if a
later callback supplies a valid setting. Construct a fresh valid definition
instead of trying to repair descendants. Construction callbacks run once,
synchronously in order, even for invalid declarations; callback panics propagate
and a supplied nil callback panics. An expanded nil or empty callback slice
supplies no edits.

Facade operations reject invalid definitions before encoding, decoding, computing,
or driver I/O—even on a cache hit. An already-canceled operation context takes
precedence over definition errors. Raw driver access bypasses these definition
checks.

For every definition and option, see the [`cache` API reference](https://pkg.go.dev/arkoren.dev/core/service/cache).

---

Source: https://arkoren.dev/docs/guides/cache/counters.md
[HTML documentation](https://arkoren.dev/docs/guides/cache/counters)

# Cache Counters

Concurrent requests can lose increments if each reads a count and writes back
its own replacement. Counter operations perform the arithmetic and mutation
atomically instead. Use them for replaceable counts such as article views or
requests within a window—not as the only record of billing or inventory.

These examples assume an existing `*cache.Cache` named `store` and operation
context `ctx`. Use `cache` from `arkoren.dev/core/service/cache` and
standard-library `time`. A [shared driver](https://arkoren.dev/docs/guides/cache/drivers.md) is required for replicas
to share counts; separate memory drivers each keep their own total. Focused
snippets omit routine propagation while keeping error results visible.

## Updating Counts

An article-view counter needs a `cache.Key[int64]`. Increment it when a view
should count:

```go
views := cache.NewKey[int64]("article-views.v1:article-73")
total, err := store.Increment(ctx, views, 1)
```

On success, `total` is the new value. A missing or expired counter starts from
zero, so its first increment by one returns one. `Decrement` subtracts its delta
with the same atomic behavior:

```go
total, err = store.Decrement(ctx, views, 1)
```

This is useful for correcting a counted view; it is not a condition that prevents
negative totals. Counters store base-10 integer text, so `store.Get(ctx, views)`
can decode the count as `int64`. An existing noninteger value causes an error
and remains unchanged.

`Increment` and `Decrement` ignore the key's declared write TTL. A newly created
counter has no expiry, and an existing counter keeps its current expiry. Giving
the key a five-minute policy therefore does not make these updates expire.

## Expiring Windows

Request counts usually need to reset. A separate increment followed by an
expiry-setting call leaves a crash window in which the count never expires.
`IncrementWithTTL` updates the count and ensures an expiry in one atomic step.

For requests from an already authorized account, use a stable internal account
ID as the key component. Here `accountID` is that application's string ID:

```go
requests := cache.NewKey[int64]("export-requests.v1:" + accountID)
count, remaining, err := store.IncrementWithTTL(ctx, requests, 1, time.Minute)
```

On success, `count` is the updated total and `remaining` is time until expiry.
A missing entry starts from zero with a one-minute expiry. An existing entry
without expiry receives one minute; an existing expiring entry keeps its current
deadline. Later requests do not slide that deadline forward. The window begins
when expiry is first established, not at an aligned wall-clock minute.

The explicit TTL argument controls this operation, not the key's write policy.
It must be positive; zero or a negative duration returns `cache.ErrInvalidTTL`.
A custom driver unable to provide atomic increment-and-expiry must return
`cache.ErrUnsupported`, not emulate the operation with separate calls.

Redis rounds positive counter TTLs up to whole milliseconds, including
submillisecond durations. Its result remains exact throughout `int64`, and
remaining lifetimes beyond `time.Duration` saturate at that type's maximum.
The [rate limiting middleware](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) uses this
operation and handles request rejection for you.

## Handling Failures

Check `err` before using the returned count or lifetime. Arithmetic overflow and
noninteger contents are rejected without changing the stored value or expiry.
That rejection is different from a transport failure: Redis might have applied
an increment before its reply was lost.

Do not blindly repeat an increment after an uncertain result; doing so can count
the same action twice. Native clients can also have transport retry policies,
so a single facade call or telemetry span is not proof of one backend execution.
Choose native retry settings in [Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md#native-clients) and
an application recovery policy appropriate to the count. Exactly-once business
effects require a durable application protocol, not a cache counter.

No expiry does not guarantee retention. Backend loss, eviction, deletion, or
restart can erase counts and reset allowances. A cache-based limit must account
for those failures rather than treating a successful update as permanent state.

## Integer Bounds

All three counter operations accept only mathematical results from
`math.MinInt64` through `math.MaxInt64`, inclusive. An out-of-range result returns
a backend-native error: there is no portable overflow sentinel. Rejection leaves
the target value and TTL unchanged; a missing or expired target remains absent.
In particular, overflow in `IncrementWithTTL` does not add expiry to a persistent
counter.

Subtraction checks the mathematical result directly, without first negating the
delta. With standard-library `math`, initialize an adjustment counter at its
minimum:

```go
adjustment := cache.NewKey[int64]("view-adjustments.v1:article-73")
err := store.Put(ctx, adjustment, math.MinInt64)
```

After that successful write, subtracting the same minimum is representable:

```go
total, err := store.Decrement(ctx, adjustment, math.MinInt64)
```

On success, `total` is zero. Subtracting `math.MinInt64` from zero instead fails
and leaves zero unchanged; doing so against an absent target fails without
creating it. Do not replace subtraction with `Increment(ctx, key, -delta)`:
negating `math.MinInt64` overflows in application code before the driver sees it.
Nor should callers replace checked atomic arithmetic with a read, local range
check, and write; concurrent updates can invalidate that check.

For every counter operation and error, see the [`cache` API reference](https://pkg.go.dev/arkoren.dev/core/service/cache).

---

Source: https://arkoren.dev/docs/guides/cache/locks.md
[HTML documentation](https://arkoren.dev/docs/guides/cache/locks)

# Cache Locks

When several callers want to refresh the same catalog snapshot, a cache lock
lets one caller begin while others skip or wait. It is a **lease**: permission
that expires after a chosen duration even if its holder is still working.
Expiry recovers coordination after a crash, but also permits overlapping work.

Examples assume an existing `*cache.Cache` named `store`, an operation context
`ctx`, and `cache` from `arkoren.dev/core/service/cache`. Import standard-library
`context`, `errors`, and `time` where used. Replicas coordinate only through a
[shared backend](https://arkoren.dev/docs/guides/cache/drivers.md); memory leases coordinate callers sharing one driver
instance, not independent drivers or processes.

## Lease Safety

> [!WARNING]
> A paused holder can continue working after its lease expires and another caller
> acquires it. Backend data loss can release a lease too. Use leases to reduce
> duplicate work; protect harmful writes with a transaction, version check, or
> fencing enforced by the destination resource.

Arkoren does not issue fencing tokens: the protected database or external service
cannot identify stale writers from a cache lease. Extending or successfully
releasing a lease does not prove the work never overlapped another holder.
For work where duplication matters, make the action idempotent or protect it at
the resource itself.

Lease names are literal strings, not typed keys; key and namespace TTL policies
do not apply. The facade stores a lease named `catalog-refresh` at physical key
`lock:catalog-refresh`. An ordinary entry named `catalog-refresh` is separate,
but writing an ordinary entry named `lock:catalog-refresh` can overwrite the
lease. Reserve physical `lock:` keys and logical lease names starting with
`remember:` for the facade, including after namespace joins. These conventions
do not create an isolated namespace or access-control boundary.

## Taking Leases

If another refresh is already in progress, this caller can skip rather than wait.
Attempt acquisition once with `TryLock`:

```go
lease, held, err := store.TryLock(ctx, "catalog-refresh", time.Minute)
if err != nil {
	return err
}
if !held {
	return nil // Another caller holds the lease; skip this refresh.
}
defer lease.Unlock(ctx)
```

`lease, true, nil` means acquisition succeeded. `nil, false, nil` means another
holder owns it. The deferred release is best effort using the work context;
use the [separate cleanup context](#releasing-leases) below when cancellation
must not prevent release. The application can now run its rebuildable refresh.

The duration must be positive or acquisition fails with `cache.ErrInvalidTTL`.
An empty name fails with `cache.ErrEmptyLockName`. If you never unlock, the lease
expires after one minute. Acquisitions are not reentrant: another acquisition
with the same name contends even in the same goroutine.

`TryLock` calls the driver once, but the native client's transport policy can
still retry. A network error can leave acquisition uncertain; do not interpret
it as proof that the backend stored no lease. Expiry remains the fallback for
an acquisition whose response was lost.

## Waiting Briefly

If this request needs the refreshed result, `Lock` waits for ownership instead.
Give the wait its own deadline without replacing the context used for later
work:

```go
waiting, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()

lease, err := store.Lock(waiting, "catalog-refresh", time.Minute)
```

`Lock` polls with jittered backoff growing from 10ms to a 250ms cap. The
two-second wait budget and one-minute lease are independent: the lease lifetime
begins at acquisition, not when waiting starts. Context deadlines require the
backend to honor its context.

Handle a failed wait before using the returned lease:

```go
if errors.Is(err, context.DeadlineExceeded) {
	return nil // Choose the application's timeout fallback.
}
if err != nil {
	return err
}
defer lease.Unlock(ctx)
```

A deadline error is not proof that another caller is still refreshing, nor that
an acquisition command could not have reached the server. Only start work after
successful acquisition. Use the original `ctx` for work; a canceled wait context
would also prevent a later release through that context.

## Releasing Leases

`Unlock` removes the lease only if its random ownership token still matches.
It cannot remove a newer holder's token after yours expires. The `Lease` value
is safe for concurrent use.

If request cancellation must not prevent cleanup, use a bounded context that
retains request values but ignores its cancellation and deadline. After work
finishes, this example allows a separate one-second release budget:

```go
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Second)
defer cancel()
err := lease.Unlock(cleanup)
```

Use this release policy instead of the work-context defer above, arranging it on
every exit path after successful acquisition. Inspect `err` when cleanup failures
need reporting. A cooperative backend honors the timeout; an arbitrary driver
cannot be forcibly interrupted. If release fails, expiry is the fallback.

If ownership is already lost, `Unlock` returns `cache.ErrLockLost` once and
leaves another holder untouched. After confirmed release or confirmed loss,
later calls do nothing and return nil. A backend failure is returned and leaves
the call retryable with a fresh context.

A lost reply can mean the server released the lease even though `Unlock`
returned an error. A retry can then report `cache.ErrLockLost`. Neither that
error nor successful cleanup establishes whether protected work overlapped.

## Extending Leases

Longer work must renew before expiry. `Extend` sets the expiry to a positive
duration from now; it does not add that duration to the previous deadline.
For application-owned `batches` and `refresh`, check renewal before each batch:

```go
for _, batch := range batches {
	if err := lease.Extend(ctx, time.Minute); err != nil {
		return err // Stop: ownership is lost or cannot be confirmed.
	}
	if err := refresh(ctx, batch); err != nil {
		return err
	}
}
```

Each batch must still finish within the available lease time; a renewal before a
long pause cannot prevent expiry during that pause. `cache.ErrLockLost` means
the token no longer owns the lease, including after release. Treat it as
"another caller may also be working". Other backend errors also prevent you
from confirming renewal. A nonpositive extension returns `cache.ErrInvalidTTL`.

Arkoren does not renew in the background. An independent renewal loop that
continues while work is stuck could keep coordination occupied indefinitely,
and no renewal strategy makes lease ownership a resource-level correctness
guarantee.

## Protecting Writes

For a cache destination, an atomic
[`CompareAndPut`](https://arkoren.dev/docs/guides/cache/overview.md#guarded-writes) can reject a write whose observed
guard has changed. A lease reduces conflicts; the guard protects the write.
That comparison does not fence a database update or an external API call, and
restoring identical guard bytes can allow an old update to match again.

For database work that needs connection-scoped coordination, consider a
[database lock](https://arkoren.dev/docs/guides/database/locking.md). Unlike a lease, it has no application
TTL and releases when its owning connection/session ends or the database detects
its loss. Choose locking or transaction semantics appropriate to the protected
resource; a connection lock still does not make arbitrary external effects
exactly once.

`Remember` uses a ten-second lease named `remember:<completed-key>`, stored at
`lock:remember:<completed-key>`. It neither renews nor guards its final write;
see [loader and cleanup limits](https://arkoren.dev/docs/guides/cache/overview.md#remembering-values).
[Sessions](https://arkoren.dev/docs/guides/http-capabilities/sessions.md) also use ten-second leases to reduce
save conflicts, but additionally use guarded writes so an expired lease cannot
authorize a pending save past successful retirement. Cache loss or restoration
can still affect that state.

Locks publish no events; ownership loss is returned to the holder as
`cache.ErrLockLost`. Acquisition spans include waiting, but `Unlock` and `Extend`
have no facade spans. [Monitoring](https://arkoren.dev/docs/guides/cache/monitoring.md) explains these measurement
boundaries. For expiry tests, the [memory driver](https://arkoren.dev/docs/guides/cache/drivers.md#memory) works with
[`testing/synctest`](https://arkoren.dev/docs/guides/testing/time.md); custom drivers can run `cachetest.Run`
through the [conformance suite](https://arkoren.dev/docs/guides/testing/conformance-suites.md).

For every lease operation and error, see the [`cache` API reference](https://pkg.go.dev/arkoren.dev/core/service/cache).

---

Source: https://arkoren.dev/docs/guides/cache/monitoring.md
[HTML documentation](https://arkoren.dev/docs/guides/cache/monitoring)

# Cache Monitoring

A slow request can come from a cache miss, a waiting refresh, or a failing cache
backend. Facade telemetry distinguishes operation latency, lookup outcomes, and
error classes without recording entry names or values. The cache publishes no
events; use returned errors for application decisions and telemetry for
observation.

## Connecting Telemetry

With an existing cache `driver` and a configured `*telemetry.Telemetry` named
`signals`, select the handle when constructing the facade. Import `cache` from
`arkoren.dev/core/service/cache`; telemetry handles come from
`arkoren.dev/core/service/telemetry`:

```go
store := cache.New(driver, func(store *cache.Builder) {
	store.Telemetry(signals)
	store.Name("catalog")
})
```

Use a short fixed name such as `catalog`, not a tenant ID, token, or per-request
value. `Name` controls the `arkoren.cache.store` label only: it does not prefix
keys or isolate storage. An empty name reports `default`.

Without an explicit handle, each operation resolves the
[process default](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle). It is a
no-op until configured; installing the default after cache construction still
affects later operations. `Telemetry(nil)` also follows that default. Provider
connections and lifecycle remain in [Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md).

## Reading Signals

Each facade operation records a client span named `cache {operation}`, one
`arkoren.cache.operations` counter increment, and one
`arkoren.cache.operation.duration` histogram observation in seconds. Operation
and store labels identify the call, for example `get` on `catalog`.

The span's `arkoren.cache.hit` attribute describes a known lookup outcome:

| Operation | Hit Meaning |
| --- | --- |
| `get`, `pull`, `ttl` | Whether an unexpired entry was found. A subsequent decoding error is still a hit. |
| `has` | The returned existence result. |
| `remember` | Whether the result came from cache rather than running the loader, including a hit after waiting for another caller. |

For these lookups, `true` means found and `false` means missing or recomputed.
A read that fails before determining lookup outcome has no span hit attribute.
Writes and lock acquisitions have no lookup hit state either. The operations
counter uses string labels `"true"`, `"false"`, or `""` for unknown/inapplicable
hit state; the duration histogram has no hit label.

Compute a hit ratio for one operation and store as its `"true"` count divided by
the sum of its `"true"` and `"false"` counts. Exclude unknown states and writes;
combining all operations in the denominator understates the lookup hit ratio.
A `Remember` hit means reuse, not necessarily a request that never waited.

Errors wrapping `cache.ErrNotFound` are misses, not telemetry failures: they do
not set error status or `error.type`. Other failures set the span error status
and `error.type` using an error class, never the error text. For example, invalid
TTL declarations report `cache.ErrInvalidTTL`. A rejected guarded write is
`false, nil`, not a read miss or an error; unsuccessful `Add` and contended
`TryLock` likewise are ordinary non-error results.

For the complete attribute and metric catalog, see
[instrumented cache components](https://arkoren.dev/docs/guides/observability/instrumented-components.md#cache).

## Operation Boundaries

`Remember` produces one span around the entire operation, not child facade spans
for each internal read, acquisition, or write. Its duration can include waiting,
loading, storage, and the separate cleanup budget described in
[Remembering Values](https://arkoren.dev/docs/guides/cache/overview.md#remembering-values).

`Lock` and `TryLock` each emit one acquisition span (`cache lock` and
`cache try_lock`). `Lock` includes time spent polling for ownership. `Lease.Unlock`
and `Lease.Extend` emit no facade operation spans; report cleanup or renewal
failures explicitly if your application needs that visibility. Losing ownership
returns `cache.ErrLockLost`, not a published event.

`Driver` and `DriverAs` are untraced accessors. Calls through a raw driver or
native Redis client do not emit facade telemetry. Native instrumentation, if
configured independently, has its own measurement and privacy policy.

One operation span is not a backend command count or proof of exactly-once
effects. A facade operation can make several driver calls, and native clients
can retry transport failures. A lost response can leave a mutation's outcome
unknown even though the facade span reports an error.

## Privacy Boundaries

Facade spans and metrics never record cache keys, lock names, values, or error
messages. Error classes are used because backend error text can contain private
entry names. Store names remain caller-supplied telemetry labels, so keep them
fixed and non-sensitive.

This protection does not sanitize your application logs or raw/native client
instrumentation. Owned keys and namespaces redact formatting, but explicit
`Name()` results, raw arguments, and `Match` or `Entry` descriptors expose their
contents. Follow [identity inspection rules](https://arkoren.dev/docs/guides/cache/keys.md#inspecting-identities),
and do not attach sensitive names to your own spans to compensate for their
absence from facade telemetry.

## Testing Signals

To test instrumentation without exporting it, use the fake from
`arkoren.dev/core/service/telemetry/telemetrytest`. Inside an existing test with
`t *testing.T`, configure an isolated handle. Import `memorycache` from
`arkoren.dev/core/service/cache/memorycache`:

```go
fake := telemetrytest.New()
store := cache.New(memorycache.New(), func(store *cache.Builder) {
	store.Telemetry(fake.Telemetry(t))
})
```

With standard-library `errors`, confirm absence as part of the test before
asserting its signal:

```go
_, err := store.Get(t.Context(), cache.NewKey[string]("catalog:missing"))
if !errors.Is(err, cache.ErrNotFound) {
	t.Fatalf("Get error = %v, want ErrNotFound", err)
}
```

The fresh memory driver guarantees this read misses. Assert one facade span
with a false hit attribute, using `telemetry` from
`arkoren.dev/core/service/telemetry`:

```go
fake.AssertSpanCount(t, 1)
fake.AssertSpan(t, "cache get", telemetry.Bool("arkoren.cache.hit", false))
fake.AssertCounter(t, "arkoren.cache.operations", 1,
	telemetry.String("arkoren.cache.operation", "get"),
	telemetry.String("arkoren.cache.hit", "false"))
```

These assertions verify the facade signal, not Redis connectivity, persistence,
or exactly-once execution. The fake also supports histogram and snapshot
inspection; see [telemetry testing](https://arkoren.dev/docs/guides/testing/fakes.md).

For every instrumentation setting, see the [`cache` API reference](https://pkg.go.dev/arkoren.dev/core/service/cache).

---

Source: https://arkoren.dev/docs/guides/search/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/search/overview)

# Search Overview

A catalog search needs matching words, not only exact database-field equality.
Arkoren combines full-text matching, field filters, and ordering behind a typed
search engine. Your application chooses the documents to index; the engine
returns those documents as typed hits.

## Search Boundaries

An **index** declares a document type, its identity, and the fields available
for searching, filtering, and sorting. A **query** describes which documents
to match. The **engine**, `*search.Engine`, validates these definitions, converts
documents to and from JSON, and delegates storage and matching to a **driver**.

Choose and construct a backend in [Search Drivers](https://arkoren.dev/docs/guides/search/drivers.md). Declare your
document shape in [Indexing Documents](https://arkoren.dev/docs/guides/search/indexing-documents.md), then
[synchronize its index](https://arkoren.dev/docs/guides/search/deploying.md) before putting documents or searching.
Registration alone does not create a backend index.

Keep authoritative records in your database or another rebuildable source.
Search does not automatically follow source changes, enforce application
permissions, or replace your database. Backend ranking and capacity differ;
facets, highlighting, and typo tolerance are outside the portable contract.

## Finding Products

Assume `engine` is configured with the populated `Products` index from
[Indexing Documents](https://arkoren.dev/docs/guides/search/indexing-documents.md#document-shape), and `ctx` is your
operation context. Import `search` from `arkoren.dev/core/service/search` and
`pagination` from `arkoren.dev/core/pagination`. Focused examples leave error
results visible but omit routine propagation; handle errors in your application.

For a tenant-scoped catalog search box, take `tenantID` from trusted,
authorized application state and combine it with the text and price filter:

```go
query := search.NewQuery(Products, "red shoes", func(selection *search.QueryBuilder) {
	selection.Where("tenant_id", tenantID)
	selection.Where("price", "<=", 100)
})
```

This publishes an immutable query without performing I/O. It requires both
words, allows the last word to match a prefix, and limits matches to products
in that tenant costing at most 100. Run it to fetch one window:

```go
page, err := engine.Search(ctx, query, pagination.PageParams{Page: 1, PerPage: 20})
```

`page` is a `pagination.Page[search.Hit[Product]]`. Its items contain each hit's
stored `ID`, decoded `Document`, and `Score`; its metadata describes the bounded
result set. After checking `err`, use `hit.Document.Title` to display a product.
No matches produces an empty items collection, not `search.ErrNotFound`.

## Reading Results

Without explicit sort keys, positive text matches rank by descending relevance,
then ascending document ID. Scores are comparable only within one result, not
between queries or drivers, and are not percentages. Empty or exclusion-only
text has score zero.

Search reaches only the first 1,000 hits. [Search Queries](https://arkoren.dev/docs/guides/search/queries.md) explains
text syntax, permission filters, explicit ordering, and this result-window bound.
The [pagination guide](https://arkoren.dev/docs/guides/queries/pagination.md) explains the shared page types;
its SQL examples are not search-query syntax.

## Operation Boundaries

The engine rejects an invalid query before driver I/O. An unregistered index,
or a registered name used with a different field definition, returns
`search.ErrNotFound`. Invalid or duplicate registrations are retained at
construction and fail every operational call; `Sync` or `Check` exposes them
before serving requests.

Pass a context with a deadline when work must stop with its caller. Cancellation
is cooperative: individual key callbacks and JSON encoders or decoders finish
before the next facade check. A decode failure or cancellation during result
decoding returns an error, not a successful partial page. Cancellation after
writes begin does not imply rollback; see [batch behavior](https://arkoren.dev/docs/guides/search/indexing-documents.md#batch-behavior).

The engine is safe for concurrent use when its driver is. `Driver()` and
`DriverAs` return shared drivers, not copies. Calling them directly bypasses
facade validation, JSON conversion, telemetry, and notifications. Configure
[Monitoring](https://arkoren.dev/docs/guides/search/monitoring.md) on the engine to observe ordinary operations.

For application tests, use an independent [Memory driver](https://arkoren.dev/docs/guides/search/drivers.md#memory),
synchronize it, and put only the documents each test needs. Assert matching IDs
and explicit orders rather than backend-specific relevance scores. A custom
driver can use `searchtest.Run` from `arkoren.dev/core/service/search/searchtest`;
the [conformance guide](https://arkoren.dev/docs/guides/testing/conformance-suites.md) explains what a suite
does and does not establish. Memory tests do not prove native backend behavior.

For every function and option, see the [search API reference](https://pkg.go.dev/arkoren.dev/core/service/search).

---

Source: https://arkoren.dev/docs/guides/search/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/search/drivers)

# Search Drivers

Use Memory for isolated tests or a small rebuildable local index. Postgres
reuses an existing database and its transactions. Meilisearch provides dedicated
single-server search; OpenSearch provides a clustered backend. Benchmark your
documents and query load: sharing a facade does not make capacity, ranking, or
operational costs interchangeable.

## Compatibility Matrix

This compares Arkoren's exposed search behavior, not the vendors' full products.
Shared state means processes use the same backend namespace. Searchability after
`Put` does not guarantee durable storage or atomic batches.

| Driver | Shared State | Text Matching | Typed Filters | Explicit Sorting | Schema Sync/Check | Searchable After Put | Database Transactions |
| --- | --- | --- | --- | --- | --- | --- | --- |
| [Memory](#memory) | No | Yes | Yes | Yes | Yes | Yes | No |
| [Postgres](#postgres) | Yes | Yes | Yes | Yes | Yes | Partial — Other transactions must wait for an enclosing transaction to commit. | Yes |
| [Meilisearch](#meilisearch) | Yes | Partial — Ten positive words and phrases; multiword nonphrase exclusions are rejected; diacritics fold. | Partial — Native coercion and string length limits differ. | Partial — Long strings and IDs lose their expected ordering. | Yes | Yes | No |
| [OpenSearch](#opensearch) | Yes | Yes | Partial — Long filterable strings fail writes. | Partial — Requires native ID fielddata; long keywords fail writes. | Partial — Some mapping changes require a rebuild; Check compares a completed checkpoint. | Yes | No |

Memory loses all state on process exit. Persistent backends still need
application-owned storage, backup, replication, and recovery policies; a successful
search call is not proof of those policies. Every engine can emit optional
[administrative events](https://arkoren.dev/docs/guides/search/monitoring.md#deployment-events), independently of provider.

Examples import `search` from `arkoren.dev/core/service/search` and use the
[`Products` declaration](https://arkoren.dev/docs/guides/search/indexing-documents.md#document-shape). Run `go get`
commands in your application's module. Check constructor errors before using a
driver. Native providers require the [application lifecycle](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md)
and [deployment ordering](https://arkoren.dev/docs/guides/search/deploying.md); engine construction alone starts nothing.

## Memory

Memory is included in `arkoren.dev/core` and needs no service or credentials.
Each constructed driver owns independent process-local state.

### Engine Setup

Create the driver and register your application's indexes:

```go
import "arkoren.dev/core/service/search/memorysearch"

driver := memorysearch.New()
engine := search.New(driver, func(settings *search.Builder) {
	settings.Indexes(Products)
})
```

`memorysearch.New` has no options and cannot fail. There is no native client,
`Start`, `Stop`, or `Ping`. Run `engine.Sync(ctx)` before
[putting documents](https://arkoren.dev/docs/guides/search/indexing-documents.md#putting-documents) or
[searching](https://arkoren.dev/docs/guides/search/overview.md#finding-products). For application startup, register only
`composer.Startup(engine.Sync)`.

### Matching Limits

Search scans every document under a read lock. Text is lowercased and split on
nonletters and nondigits, without stemming or typo tolerance. All matching
documents score 1 when positive text exists, otherwise 0; explicit orders and
the ID tie-break determine their order.

`Sync` keeps documents and records the new field policy immediately; searches
read fields from stored documents. Stored and returned byte containers are
independent. Cancellation is checked between schemas, documents, and result
copies, after lock acquisition, and after sorting. Mutex waits, sorting, and
individual JSON decodes are not interrupted; canceled mutations can be partial.

For every function and option, see the [memorysearch API reference](https://pkg.go.dev/arkoren.dev/core/service/search/memorysearch).

## Postgres

Choose Postgres when you already operate PostgreSQL and want search writes to
join your database transactions. Text uses `tsvector` and a GIN index; filters
and sorting read JSON from text-matched rows rather than dedicated field indexes.
Measure those scans before choosing this backend for a large catalog.

### Engine Setup

Install the driver, then use an existing PostgreSQL-backed `*database.Database`
named `db` from your [database setup](https://arkoren.dev/docs/guides/database/drivers.md#postgresql):

```sh
go get arkoren.dev/driver/postgres
```

```go
import "arkoren.dev/driver/postgres/postgressearch"

driver, err := postgressearch.NewFrom(db)
```

`NewFrom` performs no I/O. A nil database, another SQL dialect, or an invalid
instance name returns `postgressearch.ErrInvalidConfig`. The driver borrows
`db`; `Stop` does nothing, and your application closes the database owner.
`driver.Database()` exposes that same borrowed facade.

The driver never runs DDL. Register its table declarations with your existing
application migrator before deploying:

```go
if err := migrator.Add(driver.Migrations()...); err != nil {
	return err
}
```

Create that migrator with `migrate.New(db)` from
`arkoren.dev/core/service/database/migrate` if needed. Registration creates no
tables. Run `migrator.Up(ctx)` in the release process, then start the search
driver and synchronize the logical indexes. Grant DDL privileges to the migration
identity and table read/write privileges to runtime identities.

Once construction succeeds, register the typed definition:

```go
engine := search.New(driver, func(settings *search.Builder) {
	settings.Indexes(Products)
})
```

`Start` and `Ping` verify connectivity and both tables. Missing tables return
`search.ErrNotFound` with migration guidance. Start the database owner first;
then use `driver.Start` and the release/startup policy in
[Deploying Search](https://arkoren.dev/docs/guides/search/deploying.md#release-deployment). Follow the shared
[indexing](https://arkoren.dev/docs/guides/search/indexing-documents.md#putting-documents) and
[query](https://arkoren.dev/docs/guides/search/queries.md#catalog-filters) tasks after synchronization.

### Instance Identity

For independent search stores in one database, choose a stable instance at
construction:

```go
driver, err := postgressearch.NewFrom(db, func(settings *postgressearch.Builder) {
	settings.Instance("catalog")
})
```

The default uses `search_documents` and `search_indexes`; `catalog` uses
`catalog_search_documents` and `catalog_search_indexes`. An instance is at most
40 bytes of lowercase letters, digits, and underscores; empty selects the
default. Digit-leading names are supported because generated identifiers are
quoted. There is no table-name setting or supported instance rename.

Keep deployed identities unchanged. The migration group is
`arkoren.dev:driver/postgres:postgressearch`, with `:<instance>` for a named
instance. Its IDs are `documents-create-table` and `indexes-create-table`.
Identifier quoting changes none of these identities or their advisory-lock keys.
Rolling these migrations back deletes indexed documents; recreate and repopulate
from source rather than treating rollback as a harmless configuration reset.

### Matching Limits

Text uses PostgreSQL's `simple` configuration without stemming or stop words;
punctuation splits words and relevance uses `ts_rank`. Filters preserve scalar
kinds and compare strings case-insensitively in byte order. JSONB storage
normalizes formatting and object-key order and rejects `\u0000`; see
[PostgreSQL JSON restrictions](https://www.postgresql.org/docs/current/datatype-json.html).

Without an enclosing transaction, writes commit before returning. With a
transaction in `ctx`, other transactions see changes after commit. Search reads
its window and capped count separately, so concurrent writes can make those
reads disagree; it does not create a shared snapshot for you.

`Sync` changes data, not SQL schema. It recomputes every stored search vector
when searchable fields change and uses a transaction-scoped advisory lock keyed
by the documents table, so concurrent synchronizations queue. Large batches of
`Put` use chunks of 1,000 documents; without your enclosing transaction they
are not one atomic write.

### Local Server

For a disposable development database, install Docker with Compose and save this
`compose.yaml` in an empty development directory. These credentials and disabled
TLS are for loopback-only use, not production:

```yaml
services:
  database:
    image: postgres:17-alpine
    environment:
      POSTGRES_USER: local
      POSTGRES_PASSWORD: local-password
      POSTGRES_DB: catalog
    ports:
      - "127.0.0.1:15432:5432"
```

Start the database, then retry the readiness command until it accepts connections:

```sh
docker compose up -d
docker compose exec database pg_isready -U local -d catalog
```

Use `postgres://local:local-password@127.0.0.1:15432/catalog?sslmode=disable`
with the [Postgres database constructor](https://arkoren.dev/docs/guides/database/drivers.md#postgresql), then
apply the search migrations and startup ordering above. Put a product and run
the shared search example to verify the logical index, not only connectivity.
Run `docker compose down --volumes` afterward to remove this disposable stack
and its volumes. This cleanup discards its indexed data.

For every function and option, see the [postgressearch API reference](https://pkg.go.dev/arkoren.dev/driver/postgres/postgressearch).

## Meilisearch

Use Meilisearch for a dedicated search server with native prefix matching.
Version 1.15 or later is required for string comparisons. Provision a reachable
server and a key authorized for the index, document, settings, task, and version
operations your application uses. Load credentials from configuration; do not
publish a master key in source or expose an unprotected server.

### Engine Setup

Install the module and construct the HTTP-backed driver. Here `address` and
`apiKey` are your configured server URL and secret:

```sh
go get arkoren.dev/driver/meilisearch
```

```go
import "arkoren.dev/driver/meilisearch/meilisearchsearch"

driver, err := meilisearchsearch.New(address, func(settings *meilisearchsearch.Builder) {
	settings.APIKey(apiKey)
})
```

Construction validates an absolute HTTP(S) URL and options without I/O. URL path
prefixes are supported; queries and fragments are rejected. The default client
has no overall request timeout, so give operations context deadlines.

After checking construction, register your indexes:

```go
engine := search.New(driver, func(settings *search.Builder) {
	settings.Indexes(Products)
})
```

Register `driver.Stop` as cleanup before `driver.Start` as startup, and use
`driver.Ping` for readiness. `Start` checks connectivity, the key, and the minimum
version; `Ping` checks the version endpoint but does not enforce that minimum.
Use [release synchronization](https://arkoren.dev/docs/guides/search/deploying.md#release-deployment) before the shared
[document](https://arkoren.dev/docs/guides/search/indexing-documents.md#putting-documents) and [query](https://arkoren.dev/docs/guides/search/queries.md#catalog-filters)
operations. `Stop` closes idle connections without canceling or draining in-flight
requests.

### Client Settings

To isolate environments, set `IndexPrefix` on construction; it prefixes every
logical index name and defaults to empty. A prefix holds at most 100 ASCII
letters, digits, underscores, and hyphens. Settings are fixed after construction;
API keys are redacted by the driver's configuration formatting.

If you need a proxy, custom TLS, or an overall timeout, supply your own client.
With `net/http` and `time` imported:

```go
client := &http.Client{Timeout: time.Minute}
driver, err := meilisearchsearch.NewFrom(client, address,
	func(settings *meilisearchsearch.Builder) {
		settings.APIKey(apiKey)
		settings.IndexPrefix("shop_staging_")
	})
```

Both settings apply to `NewFrom`. The HTTP client remains yours;
`driver.Client()` returns it and `Stop` still closes its idle connections.
Native requests do not automatically gain the server URL or authorization.
There is no shipped native Meilisearch SDK adapter. Keep managed settings under
`Sync`, rather than changing them through custom requests.

### Matching Limits

`Sync` manages primary key `arkoren_id`, field roles, ranking rules, the
1,000-hit pagination limit, and disabled typo tolerance. It waits for changed
settings' tasks and refuses an existing index with another primary key. `Check`
compares those settings and the primary key. Serialize deployments and native
administration: metadata reads and asynchronous tasks are not atomic.

`Put` and `Delete` wait for task completion; successful returns are searchable.
If a context expires while polling, the server can still apply the change.
Reconcile or repeat the idempotent write. HTTP status failures are not retried;
polling a pending task is completion tracking, not request retry.

The driver reserves top-level names starting with `arkoren_` and
`_rankingScore`, rejecting collisions with `search.ErrUnsupported` before its
first write request. It adds internal ID/order fields and removes them from hits.
Missing indexes return `search.ErrNotFound` rather than being created by writes.

Meilisearch folds diacritics as well as case, so `cafe` matches `café`. It accepts
at most ten positive words and phrases; excess text returns
`search.ErrUnsupported`. Nonphrase exclusions splitting into multiple words,
such as `-t-shirt`, also return that error. A phrase exclusion is not an
equivalent replacement.

Native filters can coerce different scalar kinds. Keep operands and stored
fields consistently typed, especially membership lists. Strings over 500 bytes
are not indexed for filtering/sorting and sort last; IDs over 250 bytes lose
their expected position in the byte-order tie-break. Ranking scores range from
0 to 1, but remain comparable only within one result; empty or exclusion-only
text scores zero.

### Local Server

With Docker and Compose installed, save this `compose.yaml` in an empty
development directory. The example key grants administration access; never use
it with real data or expose this service beyond loopback:

```yaml
services:
  search:
    image: getmeili/meilisearch:v1.55.0
    environment:
      MEILI_MASTER_KEY: local-search-master-key-0123456789
      MEILI_NO_ANALYTICS: "true"
    ports:
      - "127.0.0.1:17700:7700"
```

Start it and retry the health request while the server initializes:

```sh
docker compose up -d
curl --fail http://127.0.0.1:17700/health
```

A successful response reports status `available`. Construct the driver with
`http://127.0.0.1:17700` and the example API key, then start, synchronize, put,
and search a product using the shared tasks. Run `docker compose down --volumes`
afterward to remove this disposable stack and its volumes, discarding indexed data.

For every function and option, see the [meilisearchsearch API reference](https://pkg.go.dev/arkoren.dev/driver/meilisearch/meilisearchsearch).

## OpenSearch

Choose OpenSearch for an existing cluster or a workload needing clustered search.
The client supports OpenSearch 2.19 and 3.x. Synchronization needs index creation,
mapping updates, and reindexing permissions; runtime operations need document
and search permissions. Use TLS and restrict credentials to the intended index
namespace.

### Engine Setup

Install the module, then construct an owned driver with a configured cluster
`address`:

```sh
go get arkoren.dev/driver/opensearch
```

```go
import "arkoren.dev/driver/opensearch/opensearchsearch"

driver, err := opensearchsearch.New(address)
```

`New` publishes configuration without constructing the SDK client or doing I/O.
An empty address or invalid prefix fails here; native URL and certificate
validation occurs in `Start` and retains SDK error identities.

After checking construction, register the definition and lifecycle hooks with
your `*application.Composer` named `composer`:

```go
engine := search.New(driver, func(settings *search.Builder) {
	settings.Indexes(Products)
})
composer.Cleanup(driver.Stop)
composer.Startup(driver.Start)
composer.Startup(engine.Check)
composer.Check("search", driver.Ping)
```

This is the production instance path: [synchronize in the release command](https://arkoren.dev/docs/guides/search/deploying.md#release-deployment)
before starting instances. Register cleanup even if startup fails: a failed or
canceled connectivity check can leave an acquired client and background work
owned until `Stop`. An already canceled startup context prevents acquisition.

### Client Settings

Set credentials and an environment prefix at construction. Here `password`
is loaded from your secret configuration:

```go
driver, err := opensearchsearch.New(address, func(settings *opensearchsearch.Builder) {
	settings.IndexPrefix("shop_")
	settings.Username("search")
	settings.Password(password)
})
```

The prefix defaults to empty and accepts at most 64 bytes of lowercase letters,
digits, and underscores. The resulting native index name cannot start with an
underscore. `Nodes` supplies additional addresses. `Client` adjusts native
configuration for CA certificates, API keys, signing, retry, timeout, or discovery
settings; operations still need context deadlines.

The `Client` configuration hook runs once during `New`, after base validation
and settings, never again during `Start`. Keep it free of lifecycle I/O. Owned
containers and scalar pointers detach from retained drafts; native objects,
functions, and captures remain shared. Finish configuring those before concurrent
use.

`NewFrom(client, configure...)` borrows an existing
`*opensearchapi.Client` from the official v5 SDK. Only `IndexPrefix` applies;
the native hook is not invoked. `Start` checks connectivity, `Stop` leaves the
client open, and `Client()` returns it immediately. You own its lifetime.

### Native Access

For an owned driver, `Client()` is nil before native creation and after `Stop`.
Use it only after successful startup, with cleanup already registered. For
example, with `net/http` imported, read cluster information:

```go
request, err := http.NewRequestWithContext(ctx, http.MethodGet, "/", nil)
reply, err := driver.Client().Client.Request(request)
if err != nil {
	return err
}
defer reply.Body.Close()
```

Check the HTTP status and body as well as the transport error. Native requests
bypass facade validation, telemetry, and prefix mapping. Coordinate them with
shutdown: retaining a client pointer does not stop `Stop` from closing it.

Owned lifecycle calls serialize. Repeated `Start` rechecks the same client;
`Stop` before startup and repeated stops are no-ops; startup after stop can
acquire again. `Stop` waits for lifecycle serialization and SDK cleanup even
with an expired context, and does not drain application requests. Startup
cancellation does not forcibly interrupt SDK construction or native callbacks.
Native background work uses the configured native lifetime, not the startup
context. Before startup or after stop, request-bearing methods fail without
implicitly acquiring a client. `Ping` checks a node response, not cluster health
or index existence.

### Matching Limits

After startup and synchronization, use the shared
[document](https://arkoren.dev/docs/guides/search/indexing-documents.md#putting-documents) and [query](https://arkoren.dev/docs/guides/search/queries.md#catalog-filters)
operations. Writes use `refresh=wait_for` and return when searchable; the default
refresh interval is one second. Native timeouts or failed search shards return
errors, not successful partial pages. Relevance uses BM25 over one shard and
can change as documents are added.

Native `_source` wraps original JSON in an unindexed `source` string with
separate `strings`, `numbers`, and `booleans` containers. This preserves scalar
kinds for comparisons; missing, null, arrays, and objects have no scalar entry.
Nonempty, dot-free scalar keys are captured before declaration so later `Sync`
can index newly declared fields. Only declared fields are mapped. Search restores
the original JSON and external ID; no application field names are reserved.

Write through the driver, not flat JSON through the SDK, so source and indexed
containers stay consistent. Former flat-layout indexes require explicit recreation
and repopulation; `Sync` never converts or drops them automatically.

Strings use lowercase keywords for filters and sorts; values over 32,766 bytes
fail writes. Numbers normalize to float64. A keyword field cannot become
searchable in place; plan a [destructive rebuild](https://arkoren.dev/docs/guides/search/deploying.md#retiring-indexes).
Mapping changes and reindexing are not atomic, and `Check` compares only the
last completed schema checkpoint. Serialize deployments and native administration.

The ID tie-break requires `indices.id_field_data.enabled`. A cluster disabling
it rejects those searches. Enabling ID fielddata consumes heap; assess cluster
policy and capacity instead of treating it as a free compatibility switch.

### Local Server

With Docker and Compose installed and enough memory for OpenSearch, save this
`compose.yaml` in an empty development directory. Security is disabled: bind
only to loopback and never put real user data in this configuration.

```yaml
services:
  search:
    image: opensearchproject/opensearch:3.9.0
    environment:
      discovery.type: single-node
      DISABLE_SECURITY_PLUGIN: "true"
      DISABLE_INSTALL_DEMO_CONFIG: "true"
      OPENSEARCH_JAVA_OPTS: -Xms512m -Xmx512m
    ports:
      - "127.0.0.1:19200:9200"
```

Start the node and retry the request until it returns a cluster description:

```sh
docker compose up -d
curl --fail http://127.0.0.1:19200/
```

If startup fails, inspect `docker compose logs search` and the host/memory
requirements in the [OpenSearch Docker instructions](https://docs.opensearch.org/latest/install-and-configure/install-opensearch/docker/).
Use `http://127.0.0.1:19200` with the lifecycle above, synchronize, and verify a
product put/search. Check the ID-fielddata policy if search fails despite
connectivity. Run `docker compose down --volumes` afterward to remove this
disposable stack and its volumes, discarding indexed data.

For every function and option, see the [opensearchsearch API reference](https://pkg.go.dev/arkoren.dev/driver/opensearch/opensearchsearch).

---

Source: https://arkoren.dev/docs/guides/search/indexing-documents.md
[HTML documentation](https://arkoren.dev/docs/guides/search/indexing-documents)

# Indexing Documents

A search document is a projection of a record your application owns elsewhere.
Keep only the fields search needs, and retain a source from which you can rebuild
the index. Arkoren stores the JSON you send; it does not load records, follow
database changes, or remove private fields for you.

## Document Shape

For a tenant-scoped catalog, define the searchable representation explicitly.
Import `search` from `arkoren.dev/core/service/search` and standard-library
`time` for the timestamp:

```go
type Product struct {
	ID          string    `json:"id"`
	TenantID    string    `json:"tenant_id"`
	Title       string    `json:"title"`
	Description string    `json:"description"`
	Category    string    `json:"category"`
	Price       float64   `json:"price"`
	InStock     bool      `json:"in_stock"`
	Backorder   bool      `json:"backorder"`
	CreatedAt   time.Time `json:"created_at"`
}
```

Fields in a declaration refer to top-level JSON keys, not Go field names or
database `db` tags. Use UTC timestamps for text-based time comparisons and keep
one scalar kind per filterable/sortable field across documents. Searchable
fields contain strings; filterable and sortable fields contain strings, numbers,
or booleans.

The whole encoded document is stored and returned, including fields not listed
in the index policy. Excluding a field from `Searchable` does not make it private.
Do not index secrets or unnecessary personal data. Tenant fields permit
[permission filters](https://arkoren.dev/docs/guides/search/queries.md#permission-filters); they do not enforce them
automatically.

## Field Policies

Declare which fields each search operation can use, and select a stable document
ID from the source record:

```go
var Products = search.NewIndex("products", func(product Product) string {
	return product.ID
}, func(fields *search.IndexBuilder) {
	fields.Searchable("title", "description")
	fields.Filterable("tenant_id", "category", "price", "in_stock", "backorder", "created_at")
	fields.Sortable("price", "created_at")
})
```

`Products` is immutable. Builder methods append fields; publication copies
field lists and detaches retained drafts. Field getters and `Schema()` return
sorted, deduplicated copies. The key function and its captured state are not
cloned, so keep identity selection stable and safe for concurrent calls.

IDs must be unique within the whole index, including across tenants. A tenant
filter does not partition document identity; two records with the same ID
replace each other. Use source IDs that already satisfy that scope, or encode
both source and tenant identity into an allowed stable ID.

Index and field names are 1–64 bytes of lowercase letters, digits, and
underscores. A declaration needs a key function and at least one searchable
field. `Products.Err()` wraps `search.ErrInvalidName` for an invalid declaration.
Callbacks run synchronously once in order; supplied nil callbacks panic, while
expanded empty callback slices add no edits. Validation uses the completed draft.

Register `Products` with `Builder.Indexes` when constructing the engine in
[Search Drivers](https://arkoren.dev/docs/guides/search/drivers.md), then [synchronize it](https://arkoren.dev/docs/guides/search/deploying.md) before writes.
Duplicate names and invalid registrations fail every operational engine call.
Custom `search.Indexer` implementations must provide both `Schema()` and `Err()`;
registration checks `Err()` first. Keep custom definitions stable for the engine's
lifetime and report complete validation failures, not only schema-visible ones.

## Putting Documents

Assume `engine` registers `Products`, synchronization succeeded, and `ctx` is
your operation context. Index a product already validated and saved in your
source of truth:

```go
product := Product{
	ID:       "p-17",
	TenantID: "shop-4",
	Title:    "Red running shoes",
	Category: "shoes",
	Price:    85,
	InStock:  true,
}
err := engine.Put(ctx, Products, product)
```

`Put` encodes the value with `encoding/json/v2`, derives its ID with the key
function, and upserts it. After success it is searchable, subject to
[Postgres transaction visibility](https://arkoren.dev/docs/guides/search/drivers.md#matching-limits-1). A missing backend
index returns `search.ErrNotFound`; writes do not implicitly synchronize it.

To reflect a changed source record, put the complete projection again:

```go
product.Price = 79
err = engine.Put(ctx, Products, product)
```

The stored document is replaced entirely, not patched. A partially populated
`Product` replaces prior values with the JSON that value encodes. There is no
portable partial-update operation. Source writes and search writes are separate
unless you deliberately use the Postgres driver's shared database transaction.
For other backends, schedule indexing after a successful source commit and
retain enough information to retry failed updates.

## Removing Documents

When an authorized source deletion or visibility change removes products from
search, delete their stable IDs:

```go
err := engine.Delete(ctx, Products, "p-17", "p-18")
```

Missing document IDs succeed, so retries are safe. An absent index still returns
`search.ErrNotFound`. IDs are 1–255 bytes of ASCII letters, digits, hyphens, and
underscores; invalid IDs return `search.ErrInvalidID` before deletion starts.
Select deletion targets from trusted application state, not an unauthorized
request's arbitrary IDs.

`Delete` removes documents, not the index declaration. `Drop` destroys the entire
backend index and its documents; keep that action in a planned
[deployment or rebuild](https://arkoren.dev/docs/guides/search/deploying.md#retiring-indexes), never an ordinary document
update.

## Batch Behavior

For a backfill, submit a bounded source batch such as an existing `[]Product`
named `products`:

```go
err := engine.Put(ctx, Products, products...)
```

The facade validates every ID and encodes every document before the first driver
write. Invalid IDs, encoding failures, or values that do not encode as JSON
objects write nothing. Duplicate IDs within one call keep the last document.
An empty `Put` or `Delete` does no driver work after definition/context checks.

This preflight is not a transaction. Drivers can split large batches; a later
failure or cancellation can follow successful writes. Repeating the same upsert
batch finishes it safely, but coordinate retries with newer source updates so
an old retry does not overwrite a newer projection. Meilisearch can continue a
server task after the caller stops waiting. Consult the
[provider limits](https://arkoren.dev/docs/guides/search/drivers.md#compatibility-matrix) for reserved fields, field
lengths, and native task behavior.

Choose application batch sizes to bound encoding memory and retry work; there
is no portable streaming bulk writer. Cancellation is checked between key and
encoding callbacks and before passing the batch to the driver, not inside an
individual callback. Once driver work begins, reconcile outcomes rather than
assuming cancellation rolled them back.

For every function and option, see the [search API reference](https://pkg.go.dev/arkoren.dev/core/service/search).

---

Source: https://arkoren.dev/docs/guides/search/queries.md
[HTML documentation](https://arkoren.dev/docs/guides/search/queries)

# Search Queries

A search query combines text, field conditions, and result ordering. Start with
the populated [`Products` index](https://arkoren.dev/docs/guides/search/indexing-documents.md#document-shape) and a
configured `engine`. Import `search` from `arkoren.dev/core/service/search` and
`pagination` from `arkoren.dev/core/pagination`; `ctx` is your operation context.
These queries search indexed JSON documents, not SQL tables.

## Query Text

For a search box, pass the text to `NewQuery` without translating it to native
backend syntax:

```go
selection := search.NewQuery(Products, `red "running shoes" -boots`)
```

This requires a word starting with `red` and the consecutive phrase `running
shoes` in one field, while excluding `boots`. Only the last plain, nonexcluded
word is a prefix, even when a phrase or exclusion follows it.

| Text | Meaning |
| --- | --- |
| `red shoes` | Require both words; the last may match a prefix. |
| `"running shoes"` | Require consecutive words in one field. |
| `-boots` | Exclude a matching word. |
| `-"wool socks"` | Exclude a matching phrase. |
| `wool soc` | Require `wool` and a word beginning with `soc`. |

Matching lowercases words and splits punctuation without stemming. Nonphrase
words can match different searchable fields, including words split from one
punctuated term. `soc wool` requires the complete word `soc`; `wool soc` can
match `wool socks`. A quoted word is not a prefix.

Parsing never fails: an unclosed quote runs to the end, a minus not directly
followed by a word or quote is ordinary text, and terms without letters or
digits are dropped. Empty text, or text reduced to no terms, matches every
document unless filters restrict it. Use `NewQuery(Products, "")` for this;
the zero value of `Query[Product]` is invalid.

Meilisearch additionally folds diacritics, limits positive words and phrases to
ten, and rejects multiword nonphrase exclusions such as `-t-shirt` with
`search.ErrUnsupported`. Review [provider differences](https://arkoren.dev/docs/guides/search/drivers.md#matching-limits-2)
before relying on identical matches across backends.

## Catalog Filters

Keep field names under application control. For a validated catalog request,
use a typed input rather than accepting arbitrary predicates:

```go
type CatalogSearch struct {
	Text     string  `json:"text"`
	Category string  `json:"category"`
	MaxPrice float64 `json:"max_price"`
}
```

Assume `input` has already been decoded and validated. Restrict matches using
the index's declared filterable fields:

```go
selection := search.NewQuery(Products, input.Text, func(criteria *search.QueryBuilder) {
	criteria.Where("category", input.Category)
	criteria.Where("price", "<=", input.MaxPrice)
})
```

The two-argument `Where` is equality; the three-argument form accepts `=`, `!=`,
`<`, `<=`, `>`, or `>=`, either as strings or their `search.Operator` constants
such as `search.OpLessOrEqual`. Conditions append with AND. Use membership
methods, not `OpIn` passed to `Where`, for sets:

```go
selection := search.NewQuery(Products, "", func(criteria *search.QueryBuilder) {
	criteria.WhereIn("category", "boots", "socks")
	criteria.WhereNotIn("price", 15, 25)
})
```

Membership lists must be nonempty. Operands accept named or unnamed strings,
booleans, integers, floats, and exact `time.Time` values. Numbers normalize to
float64; integers above 2^53 can lose precision. Times normalize to RFC 3339 UTC
strings, so store comparable document timestamps in UTC. Booleans permit only
equality and inequality. Nil, unsupported types, NaN, and infinities invalidate
the query.

Strings compare case-insensitively. The typed contract does not equate numeric
15 with string `"15"`. Missing, null, array, and object values match no scalar
condition, including `!=` and `NOT IN`. `IN` needs one same-kind equal operand;
`NOT IN` needs every operand to have the stored kind and differ. Consequently,
a mixed-kind `NOT IN` list matches no scalar. Prefer homogeneous lists matching
the stored field. Meilisearch's native coercion remains a
[portability limit](https://arkoren.dev/docs/guides/search/drivers.md#compatibility-matrix), not a weaker contract.

## Permission Filters

Search does not authorize results. Derive the tenant from authenticated,
authorized application state, not from a client-supplied filter. For a trusted
`tenantID`, make the permission boundary part of every search:

```go
selection := search.NewQuery(Products, input.Text, func(criteria *search.QueryBuilder) {
	criteria.Where("tenant_id", tenantID)
	criteria.WhereGroup(func(availability *search.PredicateBuilder) {
		availability.Where("in_stock", true)
		availability.OrWhere("backorder", true)
	})
})
```

The expression is `tenant_id = tenantID AND (in_stock OR backorder)`. Without
the group, `tenant AND in_stock OR backorder` means `(tenant AND in_stock) OR
backorder` and can expose another tenant's documents. AND binds tighter than
OR. Never append a root `OrWhere` alternative that escapes a permission filter.

Group callbacks receive `*search.PredicateBuilder`, which supports predicates
and nested groups but no ordering. Callbacks share one draft; their final
combined group must be nonempty.

For a cheaper view, derive a query without changing this permission-scoped base:

```go
discounts := selection.Derive(func(criteria *search.QueryBuilder) {
	criteria.Where("price", "<", 50)
})
```

`Derive` preserves text, existing conditions, orders, and the first error; it
adds settings rather than replacing existing restrictions. This is a convenience,
not an authorization object: later application code must still preserve the
logical boundary. If permissions can change before reindexing, recheck them
against authoritative state before disclosing sensitive documents. Filtering the
fetched page afterward does not repair leaked counts or a wrongly scoped search.

## Result Ordering

To show cheaper products first and newer ones first at the same price, publish
an ordered variant of your scoped query:

```go
ordered := selection.Derive(func(criteria *search.QueryBuilder) {
	criteria.OrderBy("price")
	criteria.OrderByDesc("created_at")
})
```

Only declared sortable fields are accepted. Orders apply in declaration order,
then descending relevance, then ascending document ID by bytes. Strings sort by
lowercase bytes, numbers numerically, and missing values last in either
direction. Keep one scalar kind per field; mixed-kind ordering is undefined.

With no explicit orders, positive text ranks by relevance then ID. Empty or
exclusion-only text has score zero and falls back to ID. Scores are comparable
only within one result, not across queries or drivers. Meilisearch's string/ID
length limits and OpenSearch's ID-fielddata requirement can affect this ordering;
see [Search Drivers](https://arkoren.dev/docs/guides/search/drivers.md#compatibility-matrix).

## Result Windows

Run the query with page-number parameters to fetch a bounded window:

```go
page, err := engine.Search(ctx, ordered, pagination.PageParams{Page: 3, PerPage: 20})
```

Zero values normalize to page 1 and 25 items; the default per-page maximum is
100. The [pagination guide](https://arkoren.dev/docs/guides/queries/pagination.md) explains these shared page
types and normalization, not search text or backend execution.

`search.MaxHits` is 1,000. A normalized page times per-page above that bound
returns `search.ErrInvalidQuery` before driver I/O. `Total` is exact up to
1,000 and never reports more. `LastPage` uses that capped total, but does not
override the window check: at 30 items per page and a total of 1,000, its value
is 34 and requesting page 34 fails because its window ends at 1,020. Use a page
size that divides 1,000, such as 20 or the default 25, to navigate the full cap.
Ask readers to narrow large searches instead of offering arbitrarily deep pages.
A window past the last match is empty, not an error, when it stays within the
bound.

Pages are not a snapshot or cursor traversal. Concurrent document changes can
alter contents between requests. The bound limits reachable results, not the
backend's indexing work or scan cost.

After checking the search error, extract documents from the current page:

```go
products := page.Items.Map(func(_ int, hit search.Hit[Product]) Product {
	return hit.Document
})
```

`Map`, `Filter`, and `GroupBy` are eager, local collection operations. They visit
only the fetched window, make no search requests, and do not recompute page
metadata. Put conditions in the search query when they must change server-side
counts or ordering. JSON serializes items as an array with the page metadata.

## Query Validation

`NewQuery` publishes an immutable value without I/O. Its first declaration
error is retained by `Err()` and returned by `Search` before reaching the driver:

```go
if err := selection.Err(); err != nil {
	return err
}
```

Undeclared filter/sort fields, unsupported values/operators, empty membership
lists, and empty groups wrap `search.ErrInvalidQuery`. A zero query also wraps
the missing index's `search.ErrInvalidName`; deriving it does not repair that
failure. Registration and schema matching are checked separately by the engine.
Native provider limits can still return `search.ErrUnsupported` for a valid
query. Never retry by silently dropping a filter, especially a permission filter.

Callbacks run synchronously once in order even for invalid bases. Supplied nil
callbacks panic; expanded empty callback slices omit edits. Empty groups remain
invalid. Publication, derivation, and `Inspect()` detach owned text, condition,
group, membership, and order containers; editing a retained draft or inspection
cannot change a published query.

When user input makes a query invalid, classify its error at your
[HTTP boundary](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md). Prefer fixed fields and
validated values instead of exposing the complete predicate builder to clients.

For every function and option, see the [search API reference](https://pkg.go.dev/arkoren.dev/core/service/search).

---

Source: https://arkoren.dev/docs/guides/search/deploying.md
[HTML documentation](https://arkoren.dev/docs/guides/search/deploying)

# Deploying Search

An index declaration describes the fields your code expects; the backend must
be prepared to serve them. `Sync` creates missing logical indexes and applies
changed definitions. `Check` verifies those definitions without changing the
backend. Neither happens implicitly during a query or document write.

Use the configured `engine`, registered definitions, and concrete `driver` from
[Search Drivers](https://arkoren.dev/docs/guides/search/drivers.md). Examples use your operation context `ctx` and
import `search` from `arkoren.dev/core/service/search` plus standard-library
`errors` where shown.

## Preparing Indexes

Prepare infrastructure before synchronizing logical indexes. For Postgres,
the application registers `driver.Migrations()` and executes its migrator before
`driver.Start`; the search driver never runs DDL. `Sync` writes definitions and
search vectors, not tables. Follow [Postgres setup](https://arkoren.dev/docs/guides/search/drivers.md#postgres) for
permissions and immutable migration identities.

After starting the provider, synchronize all registered definitions. Stop the
deployment if synchronization fails:

```go
if err := engine.Sync(ctx); err != nil {
	return err
}
```

Successful synchronization keeps existing documents, can reindex them, and is
safe to repeat with the same intended definition. It never prunes backend
indexes removed from registration. Before synchronization, nonempty document
writes and searches needing a missing index fail with `search.ErrNotFound`.

An invalid declaration, nil custom registration, or duplicate name fails before
driver I/O. Applying migrations or repeating synchronization cannot fix that:
correct the declaration and construct a new engine. For valid definitions,
provider-specific changes can still require an explicit rebuild rather than
an in-place sync.

For one process owning a local backend, run `Sync` as a startup hook after
driver startup. Memory has no lifecycle methods; register only
`composer.Startup(engine.Sync)` and repopulate its process-local documents.
Persistent shared backends should use the release path below.

## Startup Checks

An instance should refuse startup if the backend does not match its code. Use
`Check` to classify a missing or changed index:

```go
err := engine.Check(ctx)
switch {
case errors.Is(err, search.ErrNotFound):
	return err // The release must prepare the missing index.
case errors.Is(err, search.ErrSchemaChanged):
	return err // The release must apply the intended definition.
case err != nil:
	return err
}
```

After successful `Sync`, `Check` passes for those same definitions within the
provider's comparison scope. Additional backend indexes are ignored. OpenSearch
compares the last completed schema checkpoint, not every live mapping setting or
document representation; it is not an external-administration detector.

Run this once at startup, not on every readiness probe. During a rolling
deployment, old instances still have the previous definitions, so repeated
`Check` calls would make them unhealthy as soon as synchronization updates the
backend. Use the concrete driver's `Ping` for readiness instead.

## Release Deployment

For a shared persistent backend, use one serialized release command containing
the new release's definitions. Apply shipped table migrations first, start the
driver, run `Sync`, and require a successful `Check` before rolling out instances.
Do not run migrations or synchronization independently on every replica.

The instance path starts the driver and checks schema once. With `application`
imported from `arkoren.dev/core/application`, and an existing composer:

```go
composer.Cleanup(driver.Stop)
composer.Startup(driver.Start)
composer.Startup(engine.Check)
composer.Check("search", driver.Ping)
```

These hooks apply to concrete native drivers, not Memory. Start Postgres's
database owner before its search driver and close that owner after application
operations finish. Register owned OpenSearch cleanup even if `Start` fails:
acquired SDK resources remain owned until `Stop`. See
[native ownership](https://arkoren.dev/docs/guides/search/drivers.md#native-access) for its cancellation and wait limits.

Do not treat a failed or canceled `Sync` as a rollback. Some settings or indexes
can already have changed, and Meilisearch server tasks can continue after the
caller stops waiting. Stop rollout, inspect the backend, reconcile with the
intended definition, retry synchronization, and check again. Keep original source
records available for a rebuild.

Postgres synchronizes under a transaction-scoped advisory lock keyed by its
documents table, so concurrent syncs queue. That does not coordinate your whole
release. Meilisearch metadata reads and tasks are not atomic. OpenSearch mapping
edits can become visible before reindexing finishes; failure leaves the previous
completed checkpoint so a later sync can retry. Serialize deployments and native
administration rather than assuming one cross-provider migration lock.

For release visibility, use [deployment events](https://arkoren.dev/docs/guides/search/monitoring.md#deployment-events)
and telemetry while still checking returned errors directly.

## Changing Fields

Plan definition changes around old and new instances overlapping. To add a
field, first deploy writers that include it in stored JSON. Synchronize the new
definition, then deploy readers that query it. If older documents lack the value,
backfill them from source with [batched `Put`](https://arkoren.dev/docs/guides/search/indexing-documents.md#batch-behavior).
Reindexing can use stored values but cannot invent values never indexed.

To remove a field, first deploy readers that stop querying it, then remove its
role from the declaration and synchronize. Verify the old/new query paths before
rollout; successful schema synchronization alone does not prove application
compatibility.

Field changes can be expensive: Postgres recomputes vectors when searchable
fields change; native providers can run reindexing work. OpenSearch cannot
promote an existing keyword field to searchable text in place. Its former flat
document layout also needs explicit recreation and repopulation for the current
typed wrapper. Schedule those changes as rebuilds, not ordinary startup repairs.

## Retiring Indexes

Removing a declaration from code leaves its backend index intact. To retire it,
run a release command that still registers the exact definition, then explicitly
drop it:

```go
if err := engine.Drop(ctx, Products); err != nil {
	return err
}
```

`Drop` permanently removes the index and every indexed document. It succeeds if
the backend index is already absent, but the engine still requires registration.
Only remove the definition after this step succeeds and no running code uses it.

For a destructive rebuild, plan a maintenance window: drop the index,
synchronize it empty, then repopulate from source in bounded batches. Search is
unavailable or incomplete until repopulation finishes. Neither `Sync` nor
`Check` verifies that every source record has been copied; confirm expected
documents and representative searches before restoring traffic. Never run this
sequence as an automatic response to a request-time error.

For every function and option, see the [search API reference](https://pkg.go.dev/arkoren.dev/core/service/search).

---

Source: https://arkoren.dev/docs/guides/search/monitoring.md
[HTML documentation](https://arkoren.dev/docs/guides/search/monitoring)

# Search Monitoring

Search requests and index deployment need different signals. Telemetry measures
every engine operation; optional events report returned `Sync` and `Drop`
outcomes. Neither signal proves that the index contains every source record or
that a multi-step change committed atomically.

## Shared Telemetry

Use your application's [shared telemetry handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle)
when constructing the engine. With an existing `driver`, the
[`Products` definition](https://arkoren.dev/docs/guides/search/indexing-documents.md#document-shape), and `search`
imported from `arkoren.dev/core/service/search`:

```go
engine := search.New(driver, func(settings *search.Builder) {
	settings.Indexes(Products)
	settings.Telemetry(app.Telemetry())
	settings.Name("catalog")
})
```

`Telemetry(nil)`, including the omitted setting, resolves `telemetry.Default()`
per operation rather than freezing the default at construction. Disabled
telemetry emits no signals. `Name` is a stable engine label, not an index rename;
empty uses `search.DefaultName`, which is `default`.

## Operation Signals

`Sync`, `Check`, `Put`, `Delete`, `Drop`, and `Search` each emit one client span
named `search {operation}`, the `arkoren.search.operations` counter, and the
`arkoren.search.operation.duration` histogram in seconds. A logical batch is
one facade operation even if a provider sends several requests.

Attributes identify operation, engine, and index; `Sync` and `Check` operate
across registrations and use an empty index label. Search records the number
of returned hits on its span, not the entire capped match count. Returned errors
use classified `error.type` values instead of recording native error messages.
See [instrumented components](https://arkoren.dev/docs/guides/observability/instrumented-components.md#search)
for exact attribute names and instrument details.

Validation failures still belong to engine operation telemetry. `Driver()` and
`DriverAs` emit nothing, and calls made through their returned shared driver
bypass facade instrumentation, validation, JSON conversion, and notifications.

## Deployment Events

For an in-process release observer, opt in to an event bus at construction.
Import `event` from `arkoren.dev/core/event`:

```go
events := event.New()
engine := search.New(driver, func(settings *search.Builder) {
	settings.Indexes(Products)
	settings.Events(events)
})
```

These are alternative engine constructions; combine settings in one callback
if you need both telemetry and events. Omitted or nil `Events` disables
notifications independently of telemetry.

Listen for administrative failures using the typed payload. Here `log` is the
standard-library package and `context` supplies the listener context:

```go
_, err := events.Listen(search.Failed, func(ctx context.Context, outcome search.MutationEvent) error {
	log.Print(outcome.Operation, outcome.ErrorClass, outcome.Duration)
	return nil
})
```

Check listener registration before running the release action. `search.Synced`
reports returned sync success; `search.Dropped` reports returned drop success,
including an absent index; `search.Failed` reports a returned driver error from
either operation. The payload contains only `Operation` (`sync` or `drop`),
driver-call `Duration`, and bounded `ErrorClass`. Success has an empty error
class. Context failures use `canceled` or `deadline_exceeded`; search sentinels
use their stable names; unknown errors use `error`.

These synchronous, best-effort notifications are not a durable audit log.
Listener errors and panics cannot replace the operation result. Driver panics
propagate without an outcome event. Invalid registrations never reach the
driver and emit no administrative event; `Check`, document operations, and
queries emit none either.

A successful sync can change nothing. A failed sync can follow partial changes;
an event is not rollback evidence or proof of index consistency. Duration excludes
listener execution. Keep handling the returned operation error and follow the
[deployment recovery path](https://arkoren.dev/docs/guides/search/deploying.md#release-deployment) before continuing
a rollout.

## Privacy Boundaries

Automatic telemetry excludes query text, document bodies, document IDs, and
native error messages. Event payloads additionally exclude index and engine
names. Telemetry does contain the configured engine and index labels, so keep
those names stable and nonsensitive rather than deriving them from request text
or private identifiers.

Application logging and native SDK instrumentation have their own policies.
Do not log entire queries, documents, or raw backend errors merely to add context
to these bounded signals. Apply your [telemetry privacy policy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md)
to exports and access controls, and verify native instrumentation separately.

For every function and option, see the [search API reference](https://pkg.go.dev/arkoren.dev/core/service/search).

---

Source: https://arkoren.dev/docs/guides/file-storage/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/file-storage/overview)

# Overview

A generated report needs a stable name so another request can retrieve it later.
The `storage` facade stores an **object**, a sequence of bytes, under a **key**
such as `reports/2026-q1.csv`. It validates portable names and delegates the
bytes to your chosen driver; it does not track users, filenames, or access rights.

Examples assume an existing `files *storage.Storage` and a non-nil operation
context `ctx`. [Drivers](https://arkoren.dev/docs/guides/file-storage/drivers.md) owns construction and backend lifecycle.
Import `storage` from `arkoren.dev/core/service/storage`; the other packages
named below are from the standard library. Focused fragments leave routine
error results visible; handle them before using results in application code.

## Object Keys

Choose keys from application-owned identifiers, not client filenames. A valid
key is a literal UTF-8 relative path of at most 1024 bytes, with `/` between
segments. Validation checks syntax, not whether the caller owns the object.
Authorize access separately before reading, replacing, or deleting a key.

Keys cannot contain empty, `.` or `..` segments, control characters, backslashes,
or `:`, `<`, `>`, `"`, `|`, `?`, `*`. A segment cannot end in a dot or space.
Reserved Windows device names are rejected case-insensitively, including their
extension forms: `CON`, `PRN`, `AUX`, `NUL`, `COM1`–`COM9`, and `LPT1`–`LPT9`.
The port digits `¹`, `²`, `³` and exact names `CONIN$`, `CONOUT$` are also
reserved; `COM0` and `COM10` are not blanket-rejected.

These rules apply on every backend. `storage.ValidateKey` checks a key without
I/O; `storage.ValidatePrefix` also allows an empty prefix or one trailing `/`.
Neither function cleans, escapes, case-folds, or repairs a name. Native filesystem
path-length and case-sensitivity constraints can still differ.

Do not store both `a` and `a/b`: a filesystem cannot represent both. If only
`reports/2026/q1.csv` exists, `reports/2026` is a prefix, not an object;
`Open` and `Stat` report it missing and `Delete` leaves its children alone.

If existing native names violate these rules, inventory them through the backend
and migrate bytes and application references to collision-free valid names.
Preserve ownership checks; silently normalizing a name can address another
user's object. Check configured backend prefixes too.

## Writing Objects

To save a completed report, give `Put` its key and an `io.Reader`. This small
report uses `strings.NewReader`, which also supports seeking:

```go
contents := strings.NewReader("month,total\n2026-01,1250\n")
info, err := files.Put(ctx, "reports/2026-q1.csv", contents)
```

On success, `info` contains `Key`, byte `Size`, and `ModifiedAt`. `Put` consumes
the reader once and replaces any existing object at that key. It does not close
the source: if you supply an open file, you own its cleanup. `PutBytes` accepts
an existing `[]byte` instead; do not mutate the bytes during the call.

> [!WARNING]
> S3 buffers a non-seekable source fully in memory. Bound its size or supply a
> seekable file; use [signed uploads](https://arkoren.dev/docs/guides/file-storage/signed-urls.md#direct-uploads) when large
> transfers should bypass your application.

A source read failure leaves the previous object intact and publishes no new
logical object. This does not mean every returned error proves that nothing
changed: filesystem replacement and S3 upload can succeed before the subsequent
metadata lookup fails. Inspect the key with `Stat` before recovery; do not
blindly repeat a replacement. Backend staging cleanup and crash durability are
separate concerns described on [Drivers](https://arkoren.dev/docs/guides/file-storage/drivers.md).

For a multipart upload, use the [HTTP upload helpers](https://arkoren.dev/docs/guides/file-storage/uploads-and-downloads.md)
rather than deriving a key from the submitted filename.

## Reading Objects

Keep the stored key in your application's record, alongside the display name and
owner. After loading and authorizing that record, open its contents. Here
`report` is a `StoredReport` and `destination` is your existing `io.Writer`:

```go
type StoredReport struct {
	Key          string
	OriginalName string
	OwnerID      string
}
```

Open and close the body in the same operation. Import `io` for copying and
`errors` for joining transfer and cleanup errors:

```go
object, err := files.Open(ctx, report.Key)
if err != nil {
	return err
}
_, err = io.Copy(destination, object.Contents)
err = errors.Join(err, object.Contents.Close())
```

The caller owns every returned body, including cleanup after a failed copy.
Metadata describes the opened object; acquiring it does not mean its bytes have
been consumed. Keep the operation context alive while reading a backend stream.
Seeking is optional: check for `io.Seeker` when you need it, rather than assuming
all drivers provide it.

If you only need metadata or an existence check, use `Stat` without opening a
body:

```go
info, err := files.Stat(ctx, report.Key)
if errors.Is(err, storage.ErrNotFound) {
	// The authorized report record has no stored object.
}
```

Treat other errors as failures, not absence. Storage preserves the driver's
not-found identity; it does not reinterpret a permission failure as a missing
object. For HTTP delivery with automatic body cleanup, use
[Downloads](https://arkoren.dev/docs/guides/file-storage/uploads-and-downloads.md#serving-files).

## Listing Objects

For housekeeping across reports, `Walk` flattens pages into a fallible lazy
[collection](https://arkoren.dev/docs/guides/collections/fallible-collections.md). Constructing it performs
neither validation nor listing. To consume at most ten metadata records:

```go
objects, err := files.Walk(ctx, storage.ListOptions{
	Prefix: "reports/",
	Limit:  100,
}).Take(10).Items()
if err != nil {
	return err // objects is an incomplete successful prefix.
}
```

`Limit` caps each page, not the whole traversal; nonpositive values use the
driver default. `Take` and breaking iteration stop future facade page requests,
not work already performed for the current page. Memory and filesystem drivers
can scan and sort matching keys; S3 can read ahead to find valid continuation.

Traversal preserves lexical source order and follows opaque cursors even after
empty or short pages. Cancellation is checked before requests and between
buffered objects. A page error or cancellation is yielded once, then traversal
ends without retry. `Items` returns the successful prefix alongside the error.
Reusing a walk starts fresh I/O with its original options and captured context;
create a new walk for a new operation context.

If your API returns individual pages, call `List` instead:

```go
page, err := files.List(ctx, storage.ListOptions{
	Prefix: "reports/",
	Limit:  100,
})
```

After checking `err`, use `page.Objects` and return `page.NextCursor` to the
caller. Continue with that cursor and the same driver and prefix until it is
empty, even after a short or empty page. Do not parse or rewrite cursors.
Prefixes are literal strings: `reports` also matches `reports-old/a.csv`;
`reports/` selects the slash-delimited namespace.

Pages and repeated walks provide no snapshot under concurrent changes. Native
entries outside portable key syntax are omitted. Keep owner/date lookup data in
your database rather than using storage listings as an authorization index.

## Deleting Objects

After authorizing removal, delete the record's exact key:

```go
err := files.Delete(ctx, report.Key)
```

Deleting an absent object succeeds, so absence does not require a preliminary
`Stat`. This removes one object, not everything below a matching prefix.

## Handling Errors

Use `errors.Is` with `storage.ErrNotFound` for missing reads and
`storage.ErrInvalidKey` for invalid keys or prefixes. The signing page explains
`ErrUnsupported`, `ErrInvalidTTL`, and `ErrInvalidSignOptions`. Other errors can
report operational failures such as denied access, a full disk, or unavailable
storage; retain their wrapped causes when handling them.

An already-canceled context takes precedence over invalid input and prevents
backend work. Signing then checks key, TTL (including the backend maximum),
method, bindings, and capability in that order. Later cancellation is cooperative:
it cannot force an arbitrary blocked reader or native operation to return.

For every operation and option, see the
[`storage` API reference](https://pkg.go.dev/arkoren.dev/core/service/storage).

---

Source: https://arkoren.dev/docs/guides/file-storage/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/file-storage/drivers)

# Drivers

Choose storage by where your application runs and which processes must share
its files. Memory suits isolated tests; filesystem storage suits one host with
a persistent volume; S3 suits replicas that need the same objects and clients
that transfer directly through signed URLs.

All three use the same [object operations](https://arkoren.dev/docs/guides/file-storage/overview.md) and portable key rules.
The facade borrows its driver: `storage.New` neither starts nor stops it.
Examples import `storage` from `arkoren.dev/core/service/storage` and use a
non-nil operation context `ctx`. Run installation commands in your application's
Go module, and check constructor errors before wrapping a driver.

Use memory for isolated tests, then run
[integration tests](https://arkoren.dev/docs/guides/testing/integration-tests.md) against the driver you
deploy; passing memory tests does not establish provider permissions or limits.

## Compatibility Matrix

This compares behavior exposed by Arkoren, not every native provider feature.
Download ranges refer to the application-served `Download` and `Inline` helpers.

| Driver | Shared Across Hosts | Survives App Restart | Seekable Reads | Download Ranges | Signed Downloads | Signed Uploads |
| --- | --- | --- | --- | --- | --- | --- |
| [Memory](#memory) | No | No | Yes | Yes | No | No |
| [Filesystem](#filesystem) | Partial — Requires the same mounted volume. | Partial — Requires the same persistent root. | Yes — Regular files only. | Yes | No | No |
| [Amazon S3](#amazon-s3) | Yes | Yes | No | No | Yes — Requires a presigner. | Partial — PUT only; the presigner must preserve requested bindings. |
| [S3-Compatible](#s3-compatible) | Yes | Partial — Depends on service persistence. | No | No | Partial — Requires compatible SigV4 signing. | Partial — Requires compatible signing and bound-header enforcement. |

Shared access assumes the same bucket/prefix or root and appropriate permissions.
Restart support does not promise backups, retention, or recovery from backend
failure. An ephemeral container directory is not persistent storage.
S3-compatible services use the S3 driver with a custom endpoint.

Every driver supports paged listing and the facade's [lazy traversal](https://arkoren.dev/docs/guides/file-storage/overview.md#listing-objects).
Neither provides a multi-page snapshot. Stopping a walk prevents later facade
requests, not work already done preparing a page.

## Memory

The bundled memory driver stores objects only in this process. A new driver
starts empty, needs no lifecycle hooks, and is safe for concurrent use:

```go
import "arkoren.dev/core/service/storage/memorystorage"

files := storage.New(memorystorage.New())
```

Use the shared [write and read tasks](https://arkoren.dev/docs/guides/file-storage/overview.md#writing-objects) with this
store. Bytes are copied into storage and into opened snapshots; later writes or
deletions do not change an already opened body. Bodies support seeking and
HTTP ranges, but objects consume application memory without a total-size quota.
Signing returns `storage.ErrUnsupported`.

If housekeeping needs a different default page size, configure the driver at
construction:

```go
driver := memorystorage.New(func(store *memorystorage.Builder) {
	store.ListLimit(500)
})
```

The default is 100; nonpositive configured values restore it. An explicit
positive `ListOptions.Limit` overrides it for that request.

See the [`memorystorage` API](https://pkg.go.dev/arkoren.dev/core/service/storage/memorystorage).

## Filesystem

Choose filesystem storage when one host owns a persistent directory. The
application's operating-system identity must be able to create and access that
directory. Keep the root application-controlled and separate from unrelated data;
portable key validation is not a substitute for filesystem permissions.

```sh
go get arkoren.dev/driver/filesystem
```

Import `filesystemstorage` from
`arkoren.dev/driver/filesystem/filesystemstorage`. Inside an existing
`compose *application.Composer` (from `arkoren.dev/core/application`), construct
the driver and register its lifecycle:

```go
driver := compose.Must(filesystemstorage.New("./data/uploads"))
compose.Cleanup(driver.Stop)
compose.Startup(driver.Start)
files := storage.New(driver)
```

`New` rejects an empty root and resolves an absolute path without disk I/O.
`Start` creates the directory if absent and opens it with `os.Root`; a
nondirectory root fails startup. Operations before `Start` or after `Stop` fail.
There are no optional construction settings. Follow
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md) for shutdown
ordering: finish active work before `Stop`, which closes the root without
joining readers or writers. Already opened bodies remain yours to close.

### Stored Files

After startup, the [shared write task](https://arkoren.dev/docs/guides/file-storage/overview.md#writing-objects) stores
`reports/2026-q1.csv` at `<root>/reports/2026-q1.csv`; intermediate directories
are created as needed. New directories request mode `0750`, subject to umask,
and stored files are set to `0640` where the platform supports Unix modes.
Existing directory permissions are not repaired. Restrict root ownership and
access explicitly in your deployment.

Replacement writes a temporary file beside the destination, syncs and closes
it, then renames it into place. Atomic replacement depends on the volume's
rename semantics; it does not promise crash durability on every filesystem.
A later metadata failure can report an error after replacement succeeded.

New staging names start with `.storage-` and end in a dot, outside portable key
syntax, so listings omit unpublished bytes. A valid name such as
`.storage-report` is an ordinary object. Never remove files by prefix alone:
stop writers, inspect ownership and leftovers, and remove only confirmed staging
files. Back up the root before native migrations or cleanup.

### Path Containment

Operations through `os.Root` cannot traverse symbolic links outside the root;
such attempts fail with an operational error, not `storage.ErrNotFound`.
Internal symlinks can be followed for object operations. Listings skip symlinks
encountered during walking, including links that point inside the root; a
listing prefix that traverses an outside link fails.

`Open` and `Stat` accept regular files only. Directories are missing logical
objects; FIFOs, sockets, and other special entries report operational errors.
On Unix, FIFO opens do not wait for a writer. Regular bodies support seeking
and [download ranges](https://arkoren.dev/docs/guides/file-storage/uploads-and-downloads.md#headers-and-ranges).

Listings walk the directory implied by the prefix, sort matching portable keys,
and default to 100 objects per page. Native regular files with portable names
can appear as objects; invalid names and special entries are omitted.

> [!WARNING]
> Shared network volumes can have different rename behavior and competing
> writers can overwrite one another. Prefer S3 for replicas on separate hosts;
> do not infer shared-write safety from a successful local filesystem test.

No emulator is needed locally. Write and read a test object, restart with the
same persistent root to verify persistence, then delete only that test key.
Stopping the driver does not remove stored data.

See the [`filesystemstorage` API](https://pkg.go.dev/arkoren.dev/driver/filesystem/filesystemstorage).

## Amazon S3

Use a pre-existing bucket and grant only needed actions: `s3:ListBucket` on
the bucket, and `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject` on allowed
objects. `Put` also reads metadata after uploading, so it needs read permission.

```sh
go get arkoren.dev/driver/aws
```

Import `awsstorage` from `arkoren.dev/driver/aws/awsstorage`. Configure a region
and optional namespace while constructing the driver:

```go
driver, err := awsstorage.New(ctx, "private-documents", func(store *awsstorage.Builder) {
	store.Region("eu-west-1")
	store.Prefix("reports")
})
```

After checking the error, wrap it with `files := storage.New(driver)`. `New`
validates the bucket and prefix and loads AWS configuration; loading can read
local configuration or consult configured credential services. Object requests
begin with operations, not construction. No `Start` or `Stop` is required, and
construction does not verify bucket access.

Credentials and an omitted region use the AWS default configuration chain,
including environment, shared files, and workload roles. `Builder.Credentials`
accepts a shared `aws.CredentialsProvider` when that chain is unsuitable.

The prefix defaults empty; a configured `reports` becomes `reports/` before
logical keys. A logical `annual.pdf` addresses `reports/annual.pdf` in the bucket,
including for signing. Prefixes separate names, not permissions: enforce access
in application authorization and bucket policy.

### Transfer Limits

Use the shared [write and read tasks](https://arkoren.dev/docs/guides/file-storage/overview.md#writing-objects). `Put` streams
from a seekable source but buffers a non-seekable reader fully in memory before
upload. Bound source sizes; choose a seekable file or
[direct signed upload](https://arkoren.dev/docs/guides/file-storage/signed-urls.md#direct-uploads) for large transfers.

An upload is followed by a metadata request. If that request fails, the new
object can already exist despite the returned error; inspect with `Stat` before
deciding recovery. No facade write event is emitted for a failed call.

`Open` returns a non-seekable response body. App-served downloads stream the
whole object without ranges or conditional handling; a signed GET lets S3 serve
the client directly. Signing supports GET and PUT, at most seven days, and
temporary credentials can shorten that validity. See
[Signed URLs](https://arkoren.dev/docs/guides/file-storage/signed-urls.md) for bindings and authorization.

`List` maps positive limits to `MaxKeys`; S3 returns at most 1,000 keys per page.
Nonportable native keys are omitted. Continuation checks can read ahead through
physical pages, adding requests while preserving native opaque cursor strings.
The empty cursor means no further matching portable object was found.

### S3-Compatible

To use another S3 implementation, set its endpoint. This also enables path-style
addressing. For the disposable local server below:

```go
driver, err := awsstorage.New(ctx, "dev", func(store *awsstorage.Builder) {
	store.Endpoint("http://localhost:9001")
	store.Region("us-east-1")
})
```

Supply that service's credentials through the default chain or
`Builder.Credentials`. Check its persistence, listing, SigV4, and signed-header
enforcement rather than assuming all S3 behavior. A successful emulator test
does not establish cloud IAM permissions or durability.

See the [`awsstorage` API](https://pkg.go.dev/arkoren.dev/driver/aws/awsstorage).

## Native Access

If SDK retry, HTTP timeout, or middleware settings need customization, configure
a native client first. With `awsconfig` imported from
`github.com/aws/aws-sdk-go-v2/config` and `s3` from
`github.com/aws/aws-sdk-go-v2/service/s3`:

```go
settings, err := awsconfig.LoadDefaultConfig(ctx)
if err != nil {
	return err
}
client := s3.NewFromConfig(settings)
driver, err := awsstorage.NewFrom(client, "private-documents", func(store *awsstorage.Builder) {
	store.Prefix("reports")
})
```

Only `Prefix` applies to `NewFrom`; region, endpoint, and credential settings
already belong to the client. The driver neither copies nor closes it. Arkoren
adds no timeout or retry policy over SDK settings; use operation deadlines and
configure native policies before sharing the client.

For an operation outside the facade, recover the concrete driver with
`files.DriverAs[*awsstorage.Driver]()`. Both this result and `files.Driver()`
are shared, not copied. With your existing bucket name `bucket`:

```go
if driver, ok := files.DriverAs[*awsstorage.Driver](); ok {
	if client := driver.Client(); client != nil {
		_, err := client.HeadBucket(ctx, &s3.HeadBucketInput{Bucket: &bucket})
		if err != nil {
			return err
		}
	}
}
```

`Client()` is nil for non-native injected clients. A successful bucket check
does not verify every object permission. Native operations bypass facade
validation, prefix mapping, events, and telemetry. Large non-seekable streams
can use this client with the AWS upload manager instead of unbounded `Put`.

A native client supplied to `NewFrom` must provide presigning to sign URLs.
For bound uploads, its presigner must report the exact requested content type
as the sole signed `Content-Type` value; missing or changed bindings return
`storage.ErrUnsupported`, not a weaker URL. Arkoren does not rewrite borrowed
middleware. See [upload bindings](https://arkoren.dev/docs/guides/file-storage/signed-urls.md#upload-bindings).

The reporting-only `awsstorage.Config` redacts credential-provider details, not
endpoint or prefix text. Native diagnostics, middleware, and provider errors
are not sanitized by facade telemetry; keep secrets out of those settings and
review native logging separately.

## Local S3

For disposable local storage, install Docker with Compose/BuildKit and the AWS
CLI. The original pinned `minio/minio` image is no longer served by Docker Hub.
Build this release from its
[official artifacts](https://github.com/minio/minio/releases/tag/RELEASE.2025-09-07T16-13-09Z),
not an unverified mirror. This fixture supports Linux amd64/arm64 containers,
not a production MinIO deployment.

Save this `Dockerfile` in a new empty development directory. Keep the pinned
base and binary checksums; a mismatch or unsupported architecture must fail:

```dockerfile
FROM alpine:3.22.7@sha256:2e0b81d60597f449baa56dc25d355367869d2dca4cd57e866f1e2b52af3eb55f
ARG TARGETARCH
RUN set -eu; \
    case "${TARGETARCH}" in \
        amd64) checksum=7c5bd8512c6e966455b1d198209358b2d191c77a83ab377c4073281065fb855f ;; \
        arm64) checksum=5c83cd2cf151717ba0243f73e1c7802ff36e272b67144bdd7f1f7d684fd6f03d ;; \
        *) printf 'Unsupported MinIO fixture architecture: %s\n' "${TARGETARCH}" >&2; exit 1 ;; \
    esac; \
    release=RELEASE.2025-09-07T16-13-09Z; \
    wget -q -O /usr/local/bin/minio \
        "https://github.com/minio/minio/releases/download/${release}/minio.linux-${TARGETARCH}.${release}"; \
    printf '%s  /usr/local/bin/minio\n' "${checksum}" | sha256sum -c -; \
    chmod 0755 /usr/local/bin/minio
ENTRYPOINT ["minio"]
```

Beside it, save `compose.yaml`, not over an existing application configuration.
These public credentials are for disposable local data only; retain loopback
binding:

```yaml
services:
  storage:
    build: .
    command: server /data
    environment:
      MINIO_ROOT_USER: localstorage
      MINIO_ROOT_PASSWORD: localstorage-password
    ports:
      - "127.0.0.1:9001:9000"
```

Start the service and configure the local client's credentials:

```sh
docker compose up -d --build
export AWS_ACCESS_KEY_ID=localstorage
export AWS_SECRET_ACCESS_KEY=localstorage-password
export AWS_DEFAULT_REGION=us-east-1
```

After the endpoint is ready, create the bucket; reuse it if it already exists:

```sh
aws --endpoint-url http://localhost:9001 s3 mb s3://dev
```

Use the [S3-compatible constructor](#s3-compatible), write and read a small test
object through the facade, then inspect it with
`aws --endpoint-url http://localhost:9001 s3 ls s3://dev`. The signing endpoint
must also be reachable by the client using the URL.

Run `docker compose down` when finished. This fixture does not preserve objects
after its container is removed.

For facade operations, see the
[`storage` API reference](https://pkg.go.dev/arkoren.dev/core/service/storage).

---

Source: https://arkoren.dev/docs/guides/file-storage/uploads-and-downloads.md
[HTML documentation](https://arkoren.dev/docs/guides/file-storage/uploads-and-downloads)

# Uploads and Downloads

An upload brings untrusted bytes into your store; a download exposes those bytes
to another client. Storage's HTTP helpers generate upload keys and control how
objects are presented, but your handler still decides who may write or read
each object and which content your application accepts.

Examples assume a configured `files *storage.Storage`, `r *http.Request`, and
`w http.ResponseWriter` in an error-returning router handler. Import `request`
from `arkoren.dev/core/http/request` and `validate` from
`arkoren.dev/foundation/validate`. Backend construction belongs on
[Drivers](https://arkoren.dev/docs/guides/file-storage/drivers.md).

## Accepting Uploads

For an avatar upload, authorize the current user's profile before storing a
file. Parse the whole multipart request with a limit that includes fields and
multipart overhead, not only file bytes:

```go
form, err := request.Multipart(r, 5<<20)
if err != nil {
	return err
}
```

This accepts at most 5 MiB in the whole body; exceeding the limit returns a
413 problem. [Parse Uploads](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#parse-uploads)
owns general multipart parsing, repeated-call behavior, and temporary-file
cleanup. Through Arkoren's router, cleanup runs after the response. Outside
that adapter, the caller must arrange `form.RemoveAll`; consume and close file
handles before cleanup, and do not queue file headers for later use.

Set a route ceiling with [Body Limits](https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md) as
well. A negative `request.Multipart` limit uses the contextual middleware limit,
or the default when none is set; it does not mean unlimited. The parser keeps
up to 8 MiB of file content in memory before spilling larger parts to disk.

The whole-body limit does not express an avatar policy. Define a separate
policy requiring a PNG of at most 2 MiB:

```go
avatar := validate.File(func(upload *validate.FileBuilder) {
	upload.Required().Max(2 << 20).Types("image/png")
})
```

Validate the actual `*multipart.FileHeader` before storage. A missing field
returns nil from `form.File`; `Required` rejects it before the storage helper
can dereference it:

```go
header := form.File("avatar")
if err := avatar.Validate(r.Context(), header); err != nil {
	return request.ValidationProblem(err)
}
info, err := files.PutFile(r.Context(), "avatars", header)
```

On success, persist `info.Key` with the owner and any display metadata. The
validation adapter maps rule violations to an HTTP problem while preserving
operational errors; see [Validation Errors](https://arkoren.dev/docs/guides/validation/errors.md).
[File Upload Rules](https://arkoren.dev/docs/guides/validation/available-methods.md#file-uploads) explain
per-file size, claimed extension, and detected-type checks. Detection examines
leading bytes, not the client's `Content-Type` header; it is not malware scanning
or proof that an image decoder will accept the whole file.

## Choosing Keys

`PutFile` combines your directory with 26 random lowercase base32 characters
and an extension inferred from the first 512 bytes. The client filename never
becomes the key. Collisions are extremely unlikely, not impossible: the
underlying `Put` still replaces a colliding key.

The directory may be empty; otherwise it must satisfy the
[object key rules](https://arkoren.dev/docs/guides/file-storage/overview.md#object-keys) without a trailing slash. An invalid
directory fails before reading the upload. Unknown or octet-stream content gets
no extension. HTML disguised as `photo.png` can get `.html`, so sniffing alone
does not make unrestricted static serving safe.

If your application needs a stable replacement key, use `PutFileAs`. Here
`userID` is the authorized, application-issued portable profile identifier, and
the preceding PNG policy has succeeded:

```go
info, err := files.PutFileAs(r.Context(), "avatars/"+userID+".png", header)
```

This replaces exactly that key. Choose the extension from your accepted format,
never `header.Filename`. Both helpers open and close their read-only multipart
source themselves, passing its seekable file to the driver. Source close is
best-effort and cannot undo or change a successful write's result; do not repeat
the write to compensate for source cleanup.

Returned `ObjectInfo` has a key, size, and modification time, not a stored MIME
type or original filename. Keep `header.Filename` separately if your application
needs a display name. Serve untrusted objects through `Download`, `Inline`, or
an isolated origin rather than an unrestricted static route on your app origin.

## Raw Uploads

For a client that sends bytes directly, `Put` accepts `r.Body`. Before this
fragment, enforce a [body limit](https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md), authorize
the target, and validate content according to your application's format policy.
Here `key` is that record's application-owned key, not an unchecked path value:

```go
info, err := files.Put(r.Context(), key, r.Body)
```

`Put` does not detect content or close the reader; the HTTP server closes the
request body after the handler returns. Reading beyond an enforced body limit
fails the reader, so no partial logical object replaces the prior one. Other
failures can occur after replacement, as explained in
[Writing Objects](https://arkoren.dev/docs/guides/file-storage/overview.md#writing-objects).

> [!WARNING]
> S3 buffers a non-seekable body fully in memory. Keep this path bounded and
> small, use a seekable multipart source, or choose a
> [direct signed upload](https://arkoren.dev/docs/guides/file-storage/signed-urls.md#direct-uploads) for large objects.

Cancellation is cooperative after work begins; it cannot force an arbitrary
blocked reader to return. Keep the request context attached to the operation.

## Serving Files

Load a document by your application's identifier and verify that the current
user may read it. Do not expose an arbitrary requested storage key. With an
authorized `document` of this shape:

```go
type StoredDocument struct {
	Key          string
	OriginalName string
	OwnerID      string
}
```

Return the download helper's result from your handler:

```go
return files.Download(w, r, document.Key, document.OriginalName)
```

The helper opens and always closes the object body. It sets attachment
disposition so the browser offers a download. An empty display name uses the
key's last segment; supplied names get an ASCII fallback and UTF-8 presentation
where needed. The display name changes presentation, not the stored key.

## Headers and Ranges

The response MIME type comes from the display filename's extension, falling back
to the first 512 bytes when the extension gives no type. A recognized extension
therefore takes precedence over sniffing. Choose trustworthy display metadata
and validate accepted formats; the helper adds `X-Content-Type-Options: nosniff`
but does not validate an entire payload.

Memory and filesystem bodies can seek, so the helper uses `http.ServeContent`:
range requests can return 206, `HEAD` omits bytes, and `Last-Modified` with
`If-Modified-Since` can produce 304. Native handling also answers invalid ranges
and failed size probes with its own HTTP error responses.

S3 bodies cannot seek. The helper streams the whole object with its known
`Content-Length`; `HEAD` receives headers only. It does not process ranges or
conditional requests for that body. Use a [signed GET](https://arkoren.dev/docs/guides/file-storage/signed-urls.md#signed-downloads)
when clients need direct S3 range downloads.

## Inline Content

For an authorized avatar, `Inline` requests browser display instead of saving:

```go
return files.Inline(w, r, document.Key)
```

Every inline object response includes `Content-Security-Policy: sandbox`.
Only PNG, JPEG, GIF, WebP, AVIF, BMP, ICO, PDF, audio, video, and plain text are
allowed inline. Other types, including HTML, SVG, XML, JavaScript, CSS,
WebAssembly, JSON, and unknown content, are forced to attachment disposition.
The offered name comes from the key's final segment.

These restrictions reduce active-content risk on your origin; they are not
authorization or content inspection. Use an isolated origin without application
cookies when your threat model requires stronger separation.

## Transfer Errors

A missing object returns `storage.ErrObjectNotFound`, a 404 problem that also
wraps `storage.ErrNotFound`. Open failures and initial sniff/rewind failures
return before response commitment, so the router can still render an error.

Source, seek, write, short-write, and body-close failures remain discoverable
through `errors.Is`; cleanup errors join transfer errors. Premature EOF during a
seekable transfer matches `io.ErrUnexpectedEOF`. Native `ServeContent` error
responses are retained, not replaced by a problem body.

An error after commitment does not always mean truncation: a copy failure can
truncate the body, while a close failure can arrive after every byte was sent.
Return errors for logging, but never append another response or automatically
retry a committed transfer. See
[Committed Failures](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md#committed-failures)
for that HTTP boundary.

For all helper options, see the
[`storage` API reference](https://pkg.go.dev/arkoren.dev/core/service/storage).

---

Source: https://arkoren.dev/docs/guides/file-storage/signed-urls.md
[HTML documentation](https://arkoren.dev/docs/guides/file-storage/signed-urls)

# Signed URLs

A large transfer need not pass through your application server. A **signed URL**
grants a client GET or PUT access to one object directly at the provider. Your
application authorizes the recipient and signs the request; the provider serves
the bytes without that client presenting your storage credentials.

The URL is itself a bearer credential. Anyone holding it can use the granted
operation while the provider accepts it, so return it only to an authorized
recipient and keep it out of logs, analytics, and shared diagnostics. Facade
[telemetry](https://arkoren.dev/docs/guides/file-storage/events-and-telemetry.md#privacy-boundaries) excludes signed URLs.

Examples assume `files *storage.Storage` backed by the
[S3 driver](https://arkoren.dev/docs/guides/file-storage/drivers.md#amazon-s3) and a non-nil operation context `ctx`. Import
`storage` from `arkoren.dev/core/service/storage`, plus standard `time`,
`net/http`, and `errors` where used. Memory and filesystem drivers do not sign
provider URLs.

## Signed Downloads

After loading and authorizing a private invoice record, use its stored key to
issue a short-lived download. Here `invoice.Key` is an application-owned string:

```go
address, err := files.Sign(ctx, invoice.Key, storage.SignOptions{
	TTL: 15 * time.Minute,
})
```

An omitted method means GET. Return the string only after checking `err`;
`SignURL` provides a parsed `*url.URL` instead. Do not edit the signed path or
query. The configured S3 prefix applies, so this addresses the same logical
object as `Open` or `Put`.

Signing does not open the object or check existence. A signed GET for a missing
key fails when fetched; use `Stat` first if your issuance flow needs that check.
Provider requests do not pass through your app's authorization middleware or
`Download`/`Inline` presentation safeguards. Review bucket permissions, stored
content metadata, and the origin that serves untrusted content separately.

TTL must be positive. S3 accepts at most seven days; larger values wrap
`storage.ErrInvalidTTL`. Temporary AWS credentials can expire sooner, making the
URL unusable before its requested TTL. Prefer short grants and issue new ones
when needed. This API does not supply one-time consumption or per-link
revocation; do not infer either from the TTL.

## Direct Uploads

A signed PUT permits replacing one key and can be reused while valid. Before
issuing it, authorize the upload target and allocate an application-owned key
that does not expose another user's object. Do not accept an arbitrary key or
filename from the client.

For an avatar, accept a decoded request with a claimed exact size. The format
policy is PNG, selected by your application rather than the client's MIME claim:

```go
type AvatarUpload struct {
	Size int64 `json:"size"`
}
```

With `input AvatarUpload`, check its positive size against the application limit
before signing. Import `problem` from `arkoren.dev/foundation/problem` for this
handler fragment:

```go
if input.Size <= 0 || input.Size > 2<<20 {
	return problem.UnprocessableContent().Derive(func(details *problem.Builder) {
		details.Detail("The avatar must contain between 1 byte and 2 MiB.")
	})
}
```

Here `key` is the upload key you allocated for the authorized user. Bind both
the expected type and exact byte length:

```go
destination, err := files.Sign(ctx, key, storage.SignOptions{
	Method:        http.MethodPut,
	TTL:           10 * time.Minute,
	ContentType:   "image/png",
	ContentLength: input.Size,
})
if err != nil {
	return err
}
```

Only expose `destination` after success. The uploader must send PUT,
`Content-Type: image/png`, and exactly `input.Size` bytes with the matching
`Content-Length`. Signing does not send these headers for the client. Binding
the MIME header does not establish that the bytes are a valid or safe image.

For browser uploads, configure bucket CORS to allow your application origin,
PUT, and the `Content-Type` request header; follow the
[S3 CORS instructions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html).
CORS is not authorization. Allow your real origin rather than every origin.
Given the selected browser `File` named `file`, whose size matches the signed
request, upload with:

```js
const uploaded = await fetch(destination, {
  method: "PUT",
  headers: { "Content-Type": "image/png" },
  body: file,
});
if (!uploaded.ok) {
  throw new Error(`Upload failed: ${uploaded.status}`);
}
```

The browser manages `Content-Length`; do not try to set that forbidden request
header manually.

## Upload Bindings

An empty `ContentType` leaves the type unbound. Zero `ContentLength` leaves
size unbound, not restricted to an empty upload. A positive length is an exact
match, not a maximum. S3 presigned PUT cannot express a size range; presigned
POST policies are a different mechanism that Arkoren does not offer.

Content type can be bound without length, but use both for user uploads.
Otherwise the URL holder selects any unbound type or size. A negative length,
or type/length bindings on GET, wraps `storage.ErrInvalidSignOptions`.

With a client supplied through [Native Access](https://arkoren.dev/docs/guides/file-storage/drivers.md#native-access), the
presigner must report the requested type as the sole signed `Content-Type`
value in its
[`v4.PresignedHTTPRequest.SignedHeader`](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/signer/v4#PresignedHTTPRequest).
A missing or changed type produces `storage.ErrUnsupported` instead of a weaker
URL. Arkoren does not modify borrowed middleware or close the supplied client.

For an [S3-compatible service](https://arkoren.dev/docs/guides/file-storage/drivers.md#s3-compatible), verify the provider
actually enforces signatures and bound headers. Signing success alone does not
test server enforcement. Never recover from unsupported bindings by silently
retrying without them.

## Confirming Uploads

A direct client write bypasses facade upload validation, write events, and
transfer telemetry. Keep the allocated key unconfirmed until your application
has verified the upload. When the client reports completion, authorize that
confirmation against the same owner's pending upload and inspect it:

```go
info, err := files.Stat(ctx, key)
```

After checking `err`, compare the size with the pending record and validate
contents before publishing the object's application record. `Stat` establishes
metadata, not content safety. A holder can reuse the PUT URL while valid, so a
completion callback alone does not make the object immutable. If your workflow
requires a validated immutable asset, design a separate trusted promotion step
rather than serving a still-writable pending key.

## Unsupported Signing

Memory and filesystem stores return an error wrapping `storage.ErrUnsupported`;
an S3 client without presigning, or unable to retain the requested type binding,
does too. Call the operation and classify its result:

```go
destination, err := files.Sign(ctx, key, options)
if errors.Is(err, storage.ErrUnsupported) {
	// Offer an application-served transfer with the same authorization policy.
}
```

Here `options` is your validated `storage.SignOptions`. Do not return
`destination` on error. A fallback can accept a
[bounded upload](https://arkoren.dev/docs/guides/file-storage/uploads-and-downloads.md#accepting-uploads), not a less
restricted signed request. Test the fallback with memory storage; exercise
provider signing separately against the deployment backend.

Cancellation precedes input validation. Otherwise validation checks key, TTL
and backend cap, method, bindings, then signing capability. Only GET and PUT are
supported; unsupported methods also wrap `ErrUnsupported`. An invalid request
does not become valid merely because the driver cannot sign.

## App-Served Links

If the backend has no direct URL, or your application must check policy on each
request, sign an application route with
[Secure Links](https://arkoren.dev/docs/guides/http-capabilities/secure-links.md) and serve its authorized
record using [Download](https://arkoren.dev/docs/guides/file-storage/uploads-and-downloads.md#serving-files).
This is a separate mechanism: your route verifies the link and your server
transfers the bytes. Storage signing does not install route authentication,
create a receiver, or add download-count enforcement.

For signing options and error identities, see the
[`storage` API reference](https://pkg.go.dev/arkoren.dev/core/service/storage).

---

Source: https://arkoren.dev/docs/guides/file-storage/events-and-telemetry.md
[HTML documentation](https://arkoren.dev/docs/guides/file-storage/events-and-telemetry)

# Events and Telemetry

Deleting an image can require thumbnail cleanup, while a slow upload requires
timing evidence. Storage exposes these as different observations: **events**
announce successful facade writes and deletions; **telemetry** measures facade
operations. Neither is a durable delivery receipt or a record of every native
backend change.

Examples use `storage` from `arkoren.dev/core/service/storage`, your
[configured driver](https://arkoren.dev/docs/guides/file-storage/drivers.md), and an existing application `app`.

## Connecting Observers

Pass the application's bus and telemetry when constructing the facade:

```go
files := storage.New(driver, func(store *storage.Builder) {
	store.Events(app.Events())
	store.Telemetry(app.Telemetry())
	store.Name("uploads")
})
```

The fixed store name labels events and telemetry; omitted or empty names report
`default`. A name does not select a bucket or provide tenant isolation.
Use a small stable set of names, not user identifiers.

Events require an explicit bus. Without explicit telemetry, each operation uses
the [shared handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle), initially a
no-op. Observe through the facade: direct driver or native-client calls bypass
its instrumentation and notifications.

## Object Events

When your application needs to react to storage changes, subscribe to the typed
topics on the same bus passed to `Builder.Events`:

| Topic | Payload | Successful Observation |
| --- | --- | --- |
| `storage.ObjectWritten` | `storage.ObjectWrittenEvent` | `Put` returns metadata after storing an object. |
| `storage.ObjectDeleted` | `storage.ObjectDeletedEvent` | `Delete` succeeds, including for an absent object. |

The topic names are `storage.object.written` and `storage.object.deleted`.
Written payloads contain `Store` and `Object` metadata: key, byte size, and
modification time. Deleted payloads contain `Store` and `Key`. They contain no
object bytes or signed URLs, but keys can still identify private documents.

For thumbnail cleanup, import standard `context`. Here `thumbnails` is your
existing service with `Delete(context.Context, string) error`, accepting the
source image key:

```go
unsubscribe, err := app.Events().Listen(storage.ObjectDeleted,
	func(ctx context.Context, payload storage.ObjectDeletedEvent) error {
		if payload.Store != "uploads" {
			return nil
		}
		return thumbnails.Delete(ctx, payload.Key)
	})
```

Check the registration error and retain `unsubscribe` until the listening
component stops; call it during that component's cleanup. See the
[event bus guide](https://arkoren.dev/docs/guides/events/overview.md) for listener ownership.

Listeners run synchronously before the storage call returns, on its goroutine
and with its context. Keep them short; hand slow work to a queue when needed.
An error or panic is logged by the bus and stops subsequent listeners for that
publication, but never changes the completed storage result. Notifications are
not a substitute for durable work: arrange durable reconciliation when missing
cleanup would leave important state inconsistent.

`PutBytes`, `PutFile`, and `PutFileAs` reuse `Put` and publish the same write
event. Failed calls publish nothing, even if replacement succeeded before a
metadata lookup failed. `Delete` events do not prove the object existed.
Concurrent writes can publish in a different order from backend replacement;
do not reconstruct the current object from event order.

Reads, listings, and signing publish no object event. A client using a
[signed PUT](https://arkoren.dev/docs/guides/file-storage/signed-urls.md#direct-uploads) writes directly to the provider and
also publishes none.

## Operation Timing

Each facade call records one client span named `storage {operation}` and two
metrics. Operations are `open`, `put`, `stat`, `delete`, `list`, and `sign`.
Upload helpers reuse `put`; download helpers reuse `open`; `Sign` and `SignURL`
reuse `sign` rather than adding helper spans.

`open` ends when the object is acquired, not after its contents are consumed or
closed. Later download read/write/close failures belong to the caller's HTTP
operation, not the completed storage span. `put` includes consuming the source
and backend work, so a slow source can explain a long write duration. `sign`
measures URL creation, not the client's later transfer.

[`Walk`](https://arkoren.dev/docs/guides/file-storage/overview.md#listing-objects) records one `list` per requested page.
Construction records nothing, and there is no separate traversal instrument.
Early stopping avoids observations for pages never requested; S3 read-ahead
inside one driver page remains part of that single facade operation.

The counter `arkoren.storage.operations` counts calls. The histogram
`arkoren.storage.operation.duration` records seconds, with buckets from 0.001
to 60 seconds. Both label operation, store, and failure class when present:

| Attribute | Meaning |
| --- | --- |
| `arkoren.storage.operation` | The facade operation. |
| `arkoren.storage.store` | Configured name or `default`. |
| `error.type` | Error class on failure. |
| `arkoren.storage.object.size` | Successful write size in bytes; span only. |

See [Instrumented Components](https://arkoren.dev/docs/guides/observability/instrumented-components.md#storage)
for the shared instrument catalog and [Telemetry](https://arkoren.dev/docs/guides/observability/telemetry.md)
for exporter setup.

## Failure Classification

`Open` or `Stat` returning `storage.ErrNotFound` is an ordinary missing-object
outcome: the span is not marked failed and no `error.type` is set. Other errors
mark the span failed and report their class, not their text.

Known classes include `storage.ErrInvalidKey`, `storage.ErrUnsupported`,
`storage.ErrInvalidTTL`, and `storage.ErrInvalidSignOptions`. Cancellation uses
the context error class; other errors use their Go type. A permission failure
is not silently reclassified as absence.

If development uses memory storage and intentionally falls back from signing,
expect `sign` failures classified as `storage.ErrUnsupported`. Keep this known
capability difference separate from operational alerts; do not hide unexpected
provider failures under the same fallback.

## Privacy Boundaries

Automatic storage telemetry excludes keys, prefixes, cursors, signed URLs,
payloads, and raw error text. Correlate through a suitably chosen application
identifier and trace ID rather than adding a private key to every span. Event
payloads include keys for listeners; review destinations before forwarding them.

Facade exclusions do not sanitize native SDK middleware, credential providers,
endpoints, or returned errors. If you add AWS `otelaws` instrumentation to a
client passed through [Native Access](https://arkoren.dev/docs/guides/file-storage/drivers.md#native-access), SDK request
spans can nest under facade spans, but follow their own data policies. Review
native attributes and HTTP/body logging separately; never record signed URLs
or credential-bearing endpoints in ordinary diagnostics.

## Testing Observations

For a test with `t *testing.T`, import `telemetrytest` from
`arkoren.dev/core/service/telemetry/telemetrytest`, `telemetry` from
`arkoren.dev/core/service/telemetry`, and `memorystorage` from
`arkoren.dev/core/service/storage/memorystorage`. Construct an isolated store:

```go
fake := telemetrytest.New()
files := storage.New(memorystorage.New(), func(store *storage.Builder) {
	store.Telemetry(fake.Telemetry(t))
	store.Name("uploads")
})
```

Write a small object and assert the observation rather than exporting it:

```go
if _, err := files.PutBytes(t.Context(), "reports/a.txt", []byte("ready")); err != nil {
	t.Fatal(err)
}
fake.AssertSpan(t, "storage put",
	telemetry.String("arkoren.storage.store", "uploads"),
)
fake.AssertCounter(t, "arkoren.storage.operations", 1,
	telemetry.String("arkoren.storage.operation", "put"),
)
```

This checks facade reporting, not backend durability or client transfer success.
For events, attach a real in-process bus, capture typed payloads, and assert
your listener's effect; include failure and absent-delete paths when relevant.

For every setting, see the
[`storage` API](https://pkg.go.dev/arkoren.dev/core/service/storage),
[`event` API](https://pkg.go.dev/arkoren.dev/core/event), and
[`telemetry` API](https://pkg.go.dev/arkoren.dev/core/service/telemetry).

---

Source: https://arkoren.dev/docs/guides/collections/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/collections/overview)

# Overview

A dispatch report often needs a subset of orders, their identifiers, and a total
for each warehouse. The `collection` package connects those operations as typed
Go calls. An eager `Slice[T]` holds ordered items; an eager `Map[K, V]` holds
values addressed by comparable keys. Their operations run when you call them.

Use collections to reshape values your application already has. They do not
translate callbacks into SQL or fetch more data. Apply database selection and
ordering before loading records, then shape the returned values in Go. For one
operation, a loop or the standard-library `slices` and `maps` packages may be
clearer than a chain.

## Creating Collections

The standalone module requires Go 1.27 and works without the rest of Arkoren.
Install it from your application's module directory:

```sh
go get arkoren.dev/foundation/collection
```

Import `arkoren.dev/foundation/collection`. For a dispatch report, wrap the
orders already loaded by your application:

```go
type Order struct {
	ID        string
	Warehouse string
	Units     int
	Ready     bool
}

orders := collection.NewSlice([]Order{
	{ID: "ord-104", Warehouse: "north", Units: 3, Ready: true},
	{ID: "ord-105", Warehouse: "south", Units: 2, Ready: false},
	{ID: "ord-106", Warehouse: "north", Units: 5, Ready: true},
})
```

`NewSlice` wraps without copying. `Slice[T]` is a defined `[]T`, so you can
index it and range over it directly. `Items()` returns the underlying `[]T`
for standard-library calls. Both the wrapper and that slice share storage.
`NewSliceAs` accepts a named slice type; `ItemsAs` converts back to one.

## Selecting Orders

Keep the ready orders before projecting their IDs. Slice callbacks receive the
input index and item; `_` makes an unused index explicit:

```go
ready := orders.Filter(func(_ int, order Order) bool {
	return order.Ready
})
ids := ready.Map(func(_ int, order Order) string {
	return order.ID
})
```

`ids` is a `collection.Slice[string]` containing `ord-104` and `ord-106`, in
that order. `Filter` and `Map` allocate result storage and leave `orders`
unchanged. Methods can introduce a new type parameter, so projection changes
the collection's element type without losing the chain.

If you need one order rather than another collection, use a lookup:

```go
order, found := orders.FirstWhere(func(_ int, order Order) bool {
	return order.Warehouse == "south"
})
```

Here `found` is true and `order.ID` is `ord-105`. If nothing matches, the result
is the zero `Order` and false; checking `found` distinguishes absence from a
stored zero value. `First` and `Last` use the same value-plus-boolean convention.

## Summarizing Dispatch

Group ready orders when each warehouse needs its own dispatch list:

```go
warehouses := ready.GroupBy(func(_ int, order Order) string {
	return order.Warehouse
})
```

`warehouses["north"]` contains the two ready orders in source order. The outer
map has no iteration-order guarantee. Use `CountBy` for the number of orders per
warehouse, or `KeyBy` for one order per identifier; repeated `KeyBy` keys keep
the last item.

To count units rather than orders, fold the selected records with `Reduce`:

```go
units := ready.Reduce(func(total, _ int, order Order) int {
	return total + order.Units
}, 0)
```

`units` is 8. The initial value is also the result for empty input, so an empty
dispatch has zero units without a separate branch.

## Working With Maps

Choose a map when the key, rather than a position, identifies each value. Map
callbacks receive `(key, value)`. For example, select warehouses with available
packing capacity:

```go
capacity := collection.NewMap(map[string]int{"north": 12, "south": 0})
available := capacity.Filter(func(_ string, slots int) bool {
	return slots > 0
})
```

`available` contains only `north: 12`. `Get` reports both a value and whether
its key exists; a stored zero is different from an absent key. `MapValues`
changes values while retaining keys, and `MapKeys` changes keys while retaining
values. Colliding transformed keys overwrite earlier entries. With a Go-map
source, the winner is unspecified because traversal order is unspecified.

For explicit overrides, `Merge` gives the argument precedence regardless of
iteration order:

```go
adjusted := capacity.Merge(collection.NewMap(map[string]int{"south": 4}))
```

`adjusted["south"]` is 4; `capacity["south"]` remains 0. The result has independent
outer map storage. Keep the returned map: `Merge` does not modify its receiver.

To display keys in a repeatable order, import `strings` and sort them:

```go
names := collection.Keys(capacity).Sort(strings.Compare)
```

`names` contains `north`, then `south`. `Keys` and `Values` are package
functions, not eager-map methods. Map traversal remains unordered; sorting
the extracted keys does not reorder the map.

## Copies And Windows

Wrapping and extracting do not copy. Assigning `orders[0]` changes the slice
passed to `NewSlice`, and writes through `orders.Items()` change the collection.
Allocated transformations such as `Filter`, `Map`, `Sort`, `Reverse`, and
`Merge` own their outer result storage. Copies remain shallow: pointers, maps,
and slices inside elements can still refer to shared data.

Slice windows behave differently. `Take`, `Skip`, `SkipWhile`, `SkipUntil`,
and `ForPage` share the source array, as do eager `Chunk` and `Sliding` windows.
A write through a window can change the source; retaining a small window can
also keep a large array alive. To keep an independent dispatch prefix, import
the standard-library `slices` package and clone it:

```go
batch := collection.NewSlice(slices.Clone(ready.Take(1).Items()))
batch[0].Units = 1
```

`ready[0].Units` is still 3. `Sort` already makes a shallow copy, but it is not
stable: equal items need not retain their relative order. Supply a comparison
that includes a tie-breaker if that distinction matters.

Nil eager collections support empty reads. `First` reports false, `Len` is
zero, and a filter with no matches returns nil slice storage. Constructors and
`Items` preserve nil versus allocated-empty input; windows preserve their
source representation. A nil map still panics on indexed writes: allocate it
first, or keep a populated `Merge` result. Concurrent access follows ordinary
Go slice and map rules; synchronize writes and shared mutable elements.

## Choosing Evaluation

Eager operations suit data already held in memory. Use [Lazy Collections](https://arkoren.dev/docs/guides/collections/lazy-collections.md)
when callbacks should run only as results are consumed, especially when you
need only a prefix. Use [Fallible Collections](https://arkoren.dev/docs/guides/collections/fallible-collections.md) when a
source or callback can fail during traversal. Neither conversion copies the
underlying application data or supplies cancellation automatically.

For selection, windowing, and serialization details, use
[Collection Methods](https://arkoren.dev/docs/guides/collections/available-methods.md). Validate untrusted window sizes:
`Chunk` and `Sliding` require positive sizes, and `Nth` and `Sliding` require
positive steps. Invalid values panic. `Take` and `Skip` clamp counts instead;
`ForPage` does not validate page inputs or arithmetic overflow.

## Native Results

Arkoren data results can already be collections. Database `Select` returns an
eager slice; [cursors and batch traversal](https://arkoren.dev/docs/guides/database/batching-and-streaming.md)
return fallible lazy slices. [Paginated results](https://arkoren.dev/docs/guides/queries/pagination.md)
contain only the returned window: filtering its `Items` does not change the
server's total, query ordering, or navigation tokens.

Inspection results, including router routes, scheduler entries, telemetry
records, and [workflow definitions](https://arkoren.dev/docs/guides/workflows/running-workflows.md#inspecting-definitions),
retain their producer's ordering and copy depth. The collection wrapper adds
no extra isolation. [Storage listings](https://arkoren.dev/docs/guides/file-storage/overview.md) and
[model streams](https://arkoren.dev/docs/guides/llm/streaming.md) retain their producer's cleanup and replay
rules; traversing a model source again can repeat billed work.

Eager collections use their native container's JSON shape through
`encoding/json/v2`. Default nil non-byte slices encode as `[]`, maps as `{}`,
and byte slices as base64 strings. Lazy JSON encoding materializes the source;
it is not network streaming. For schemas, eager collections declare their
native shape to [output description](https://arkoren.dev/docs/guides/validation/json-schema.md#describe-output).
Attach logical rules through the [named-container adapter](https://arkoren.dev/docs/guides/validation/collections.md#named-containers).

For exact declarations, see the [`collection` API reference](https://pkg.go.dev/arkoren.dev/foundation/collection).

---

Source: https://arkoren.dev/docs/guides/collections/lazy-collections.md
[HTML documentation](https://arkoren.dev/docs/guides/collections/lazy-collections)

# Lazy Collections

If a packing station needs only the next two ready parcels, transforming every
parcel first does unnecessary work. A `LazySlice` is an iterator: streaming
steps such as `Filter` and `Map` run as you consume their result, not when you
build the chain. `LazyMap` applies the same model to keyed entries.

Use eager collections when you want an immediately available result. Use lazy
collections when deferring callbacks or stopping before the end matters. A lazy
chain does not cache values, add concurrency, or make a source replayable.

## Reading A Prefix

Import `arkoren.dev/foundation/collection`. These parcels are already in memory;
`Lazy()` changes when later operations run, without copying the source:

```go
type Parcel struct {
	ID    string
	Ready bool
}

parcels := collection.NewSlice([]Parcel{
	{ID: "pkg-41", Ready: false},
	{ID: "pkg-42", Ready: true},
	{ID: "pkg-43", Ready: true},
	{ID: "pkg-44", Ready: true},
})
```

Observe the source before filtering to distinguish records read from records
selected:

```go
read := 0
next := parcels.Lazy().TapEach(func(_ int, _ Parcel) {
	read++
}).Filter(func(_ int, parcel Parcel) bool {
	return parcel.Ready
}).Map(func(_ int, parcel Parcel) string {
	return parcel.ID
}).Take(2)
```

`read` is still zero. `TapEach` observes each item without changing it; on a
lazy collection it runs only during traversal. Materialize the selected IDs
when the station is ready to use them:

```go
ids := next.Items()
```

`ids` is `[]string{"pkg-42", "pkg-43"}` and `read` is 3. `Take(2)` stops after
two matches; it does not read `pkg-44`. `Eager()` instead returns an eager
`collection.Slice[string]` with methods. Both materializers collect new outer
storage and copy elements shallowly. A slice with no yielded values is nil;
an empty materialized map is allocated.

You can also range over a lazy collection directly, or pass `Iter()` to a
standard-library iterator consumer. Breaking the loop stops traversal, provided
the source honors the iterator's stop signal.

## Full Input Operations

Not every operation can produce its first result from a prefix:

| Operation | Source Consumption |
| --- | --- |
| `Filter`, `Map`, `Take`, `Skip` | During traversal, as downstream asks for values. |
| `First`, `FirstWhere`, `Contains`, `IsEmpty` | At the call, stopping when an answer is known. |
| `Items`, `Eager`, `Len`, `Last`, `Reduce`, `Each` | At the call, through the end unless the operation can stop. |
| `Sort`, `Reverse` | The whole source before yielding, on every traversal. |
| `GroupBy`, `KeyBy`, `CountBy`, `Partition` | At the method call, buffering their results. |

Keep full-input operations off infinite sources. Sorting, reversing, grouping,
and partitioning also retain data in memory; laziness does not remove that
cost. In particular, `source.Sort(compare).Take(2)` still reads and sorts every
item before it yields two. `Unique` streams but retains a set of seen keys,
so its memory use can grow with the number of distinct keys.

## Supplying An Iterator

`NewLazySlice` accepts an iterator yielding `(index, item)` pairs. For example,
an application can expose an existing ordered set of bin labels without first
building a collection result:

```go
bins := collection.NewLazySlice(func(yield func(int, string) bool) {
	for index, label := range []string{"A-01", "A-02", "B-01"} {
		if !yield(index, label) {
			return
		}
	}
})
```

The constructor retains the function and its captures without invoking them.
`bins.Take(1).Items()` returns only `A-01`. The source must return when `yield`
returns false; continuing to yield violates the Go iterator protocol.
`NewLazyMap` accepts an `iter.Seq2[K, V]`, such as `maps.All(existingMap)` from
the standard-library `maps` package. A Go-map source retains unspecified order.

Supply a non-nil function. Lazy zero values and constructors given nil are nil
functions, not iterable empty collections; traversing them panics. For an empty
source, use `collection.NewSlice[Parcel](nil).Lazy()` or
`collection.NewMap[string, int](nil).Lazy()`.

## Source Ownership

Wrapping an iterator does not acquire or close its resources. If your source
opens a file or cursor inside its iterator, release it on every return,
including early stop and failure. If the source captures an already open
resource, its existing owner still arranges cleanup. For I/O that can fail,
use [Fallible Collections](https://arkoren.dev/docs/guides/collections/fallible-collections.md) rather than hiding read
errors in an infallible iterator.

Collection operations run synchronously in the consuming goroutine. They start
no goroutines and supply no context, timeout, or cancellation signal. A blocking
source must capture and observe the caller's context itself. Stopping downstream
cannot interrupt work blocked before the next yield.

The source and callbacks also determine concurrency safety. Do not mutate a
captured slice or map while another goroutine traverses it without
synchronization. A lazy wrapper is not a snapshot of its captured data.

## Callback Indexes

A callback on `LazySlice` receives the index yielded by its immediate input.
Custom sources can supply arbitrary indexes. Transformations yield dense output
indexes starting at zero, so an index is not a stable record identifier:

```go
labels := parcels.Lazy().Filter(func(_ int, parcel Parcel) bool {
	return parcel.Ready
}).Map(func(index int, parcel Parcel) string {
	return fmt.Sprintf("%d: %s", index, parcel.ID)
}).Take(2).Items()
```

With the standard-library `fmt` package imported, `labels` contains
`0: pkg-42` and `1: pkg-43`. The rejected parcel no longer occupies a position
in `Map`'s input. Carry the parcel ID in the value when later steps need its
identity. Map callbacks receive keys instead of indexes.

## Sampling And Windows

Use `Nth` to sample positions, or `Sliding` to inspect complete neighboring
groups. With the parcels above:

```go
sample := parcels.Lazy().Nth(2, 1).Items()
windows := parcels.Lazy().Sliding(2, 3)
```

`sample` contains `pkg-42` and `pkg-44`: offset 1 is zero-based. `Nth` defaults
to offset zero and clamps negative offsets to zero. Selection counts sequence
positions, not the indexes a custom source supplies.

`windows` is an `iter.Seq2[int, []Parcel]`. It yields only the first two-parcel
window here; the next start has no complete window. `Sliding` defaults its step
to one and omits incomplete final windows. Use `Chunk` for nonoverlapping
groups that include a final short group. Lazy chunks and windows own independent
outer slice storage, unlike eager windows; nested references remain shared.

Validate sizes and steps before calling these methods: `Chunk` and `Sliding`
panic on nonpositive sizes, and `Nth` and `Sliding` panic on nonpositive steps.
Positive steps through `math.MaxInt` are valid without wrapping. A lazy source
still reads intervening items to reach the next selection; a large step does
not skip I/O. Stopping after a yielded selection or window avoids reading the
following gap.

## Repeating Traversal

Every traversal invokes the retained iterator again. Reusing `next.Items()`
from the prefix example repeats its callbacks and increments `read` again.
A slice-backed source can be traversed again, but sees shared element data,
not an isolated snapshot.

An already open file, channel, or network response may yield only its unread
remainder on a second pass. A source that opens a fresh request can repeat I/O,
including billed model work. Treat a source as single-use unless its contract
says otherwise. If you need to inspect the same result twice, materialize it
once and reuse that eager value rather than traversing again.

For exact declarations, see the [`collection` API reference](https://pkg.go.dev/arkoren.dev/foundation/collection).

---

Source: https://arkoren.dev/docs/guides/collections/fallible-collections.md
[HTML documentation](https://arkoren.dev/docs/guides/collections/fallible-collections)

# Fallible Collections

An import can fail after several records have succeeded. `TryLazySlice[T]`
yields `(item, error)` pairs so that failure remains part of the traversal.
`TryLazyMap[K, V]` yields `(MapEntry[K, V], error)` pairs, where `MapEntry`
contains `Key` and `Value`. Ignore the item or entry whenever its error is
non-nil.

These are lazy iterators, not saved results. Streaming transformations forward
source and callback errors; the consumer decides whether to stop or continue.
Ordinary terminal methods stop on the first error. Methods ending in `All`
continue while the source permits it and return joined errors. Neither policy
retries a failed operation or recovers a panic.

## Parsing Import Values

Import `arkoren.dev/foundation/collection` and the standard-library `strconv`
package. Start with typed import lines so that source identity is available
even if callback indexes change:

```go
type QuantityLine struct {
	Line  int
	Units string
}

lines := collection.NewSlice([]QuantityLine{
	{Line: 1, Units: "3"},
	{Line: 2, Units: "unknown"},
	{Line: 3, Units: "5"},
})
```

`Try()` converts an eager or infallible lazy collection to its fallible lazy
form. It does not parse anything yet. Make parsing a step that can return an
error:

```go
quantities := lines.Try().Map(func(_ int, line QuantityLine) (int, error) {
	return strconv.Atoi(line.Units)
})
```

The result is a `TryLazySlice[int]`. When consumed, successful conversions yield
an integer and nil error; the invalid line yields a conversion error instead
of a usable integer. A source error bypasses the callback entirely.

## Stopping On Failure

Choose `Items` when a failure should stop this read:

```go
units, err := quantities.Items()
```

`units` contains only `[3]`, and `err` is the conversion error for `unknown`.
The third line is not processed. Decide whether your application can use that
prefix or must reject the import; it is not a complete result. `Eager` returns
an eager collection with the same first-error policy. A slice materializer
returns nil if no values succeed; map materializers return allocated maps.
Both own new outer storage with shallow element copies.

`Each` also stops on the first source or callback error. Work completed by an
earlier callback is not rolled back. If callbacks write application data, use
your application's transaction or idempotency policy rather than assuming
collection traversal is atomic.

## Continuing Explicitly

If independent import lines may succeed despite other invalid lines, choose
`ItemsAll` instead. This is an alternative traversal of the replayable
in-memory source above, not a second pass over an already consumed reader:

```go
units, err := quantities.ItemsAll()
```

`units` contains `[3, 5]`. `err` joins the encountered errors with `errors.Join`;
it is still non-nil even though later values succeeded. `EagerAll` collects
into an eager collection, and `EachAll` continues after both source and callback
errors. Use `errors.Is` or `errors.As` to inspect a joined cause when needed.

For a per-error decision, range over the fallible source. This loop deliberately
reports each conversion failure and skips its accompanying value; import `fmt`
for the output:

```go
for units, err := range quantities {
	if err != nil {
		fmt.Printf("Rejected quantity: %v\n", err)
		continue
	}
	fmt.Println(units)
}
```

The loop prints 3, reports the invalid quantity, then prints 5. Replacing
`continue` with `break` stops the source at that error. Continuation moves to
the next source result; it does not retry the failed line. A terminal read error
can end a source even when the consumer wants to continue.

## Limits And Indexes

`Take` and `Skip` count successful values or map entries, not errors. Errors
are forwarded in source order, including inside a skipped prefix. A positive
`Take` stops immediately after its last requested success. `Take(0)` and
negative limits consume nothing, so they cannot reveal any source error.
Duplicate lazy-map keys count as separate entries, even though materialization
keeps only the last successful value for a key.

`Nth`, `Chunk`, and `Sliding` likewise count successful slice values. An error
does not consume an offset, a step position, or space in a partial window.
Continuing after the error resumes that progress if the source has more
results; ignoring the error would conceal an incomplete or invalid input.
Sizes and steps must be positive where required, as described in
[Collection Methods](https://arkoren.dev/docs/guides/collections/available-methods.md#nth).

Streaming slice callbacks start at index zero. Source errors do not advance
the index, and callback errors leave the next callback at the same index.
For the three import lines, this records that difference:

```go
var indexes []int
parsed := lines.Try().Map(func(index int, line QuantityLine) (int, error) {
	indexes = append(indexes, index)
	return strconv.Atoi(line.Units)
})
units, err := parsed.ItemsAll()
```

`indexes` contains `[0, 1, 1]`, `units` contains `[3, 5]`, and `err` is non-nil.
A successful predicate still advances its input index when it rejects a value;
a successful `FlatMap` advances once even when it returns no items. Later steps
have their own input indexes. Carry `QuantityLine.Line` into application errors
or output records when you need the original line number.

`EachAll` deliberately differs: its index counts every successful source value,
including a value whose callback fails. Calling it directly on `lines.Try()`
therefore gives callback indexes zero, one, and two. Source errors still bypass
its callback and do not advance that index.

## Supplying Read Errors

For an existing application-owned `io.Reader` named `reader`, a scanner-backed
iterator must expose `Scanner.Err`, not silently treat a failed read as the end
of input. Import `bufio` along with `collection`:

```go
text := collection.NewTryLazySlice(func(yield func(string, error) bool) {
	scanner := bufio.NewScanner(reader)
	for scanner.Scan() {
		if !yield(scanner.Text(), nil) {
			return
		}
	}
	if err := scanner.Err(); err != nil {
		yield("", err)
	}
})
```

`text.Items()` returns scanned lines and any read or scanner-limit error.
The source returns immediately when `yield` is false. Its last error yield
needs no continuation check because the iterator returns immediately afterward.

The reader remains application-owned: wrapping it does not close it or cancel
a blocked read. If your iterator opens a resource, arrange its release inside
the iterator on every return, including early stop. If it captures an already
open resource, its existing owner arranges cleanup. Blocking sources must
observe their own context; the collection supplies no cancellation or timeout.
Use a non-nil iterator function, as explained under
[Supplying An Iterator](https://arkoren.dev/docs/guides/collections/lazy-collections.md#supplying-an-iterator).

This reader-backed source is single-use unless the reader itself supports
replay. Materialize once when you need to inspect the same values repeatedly.
Do not call `ItemsAll` after `Items` expecting it to resume with every original
line: each traversal invokes a new scanner over the reader's current position,
and buffering may already have read ahead.

## Native Producers

[Database cursors and batch traversal](https://arkoren.dev/docs/guides/database/batching-and-streaming.md),
[storage listings](https://arkoren.dev/docs/guides/file-storage/overview.md), and
[model streams](https://arkoren.dev/docs/guides/llm/streaming.md) return fallible lazy data. Their producers
own cleanup, cancellation, and repeat-traversal behavior. A storage listing's
page size is not a total-item limit; repeated traversal can issue fresh requests.
Repeating a model stream can repeat billed work, and stopping early interrupts
execution rather than completing a model turn.

These producers stop after terminal read or execution errors. `ItemsAll` can
continue after your transformation fails, but cannot reconnect a cursor, retry
a failed page fetch, or resume a failed model request. Check the returned error
before treating a collected prefix as complete.

Full-input transformations have their own failure shapes: `Sort` and `Reverse`
discard a failed materialization's prefix and yield only the error. `Partition`
consumes at the call and makes both returned iterators yield the error without
partial partitions. `GroupBy`, `KeyBy`, and `CountBy` return partial eager maps
with the first error. JSON encoding returns the first source error instead of
encoding a partial collection; materialization is not network streaming.

For exact declarations, see the [`collection` API reference](https://pkg.go.dev/arkoren.dev/foundation/collection).

---

Source: https://arkoren.dev/docs/guides/collections/available-methods.md
[HTML documentation](https://arkoren.dev/docs/guides/collections/available-methods)

# Collection Methods

Use this reference to choose an operation for an existing collection. Start
with the [overview](https://arkoren.dev/docs/guides/collections/overview.md) for eager usage, [Lazy Collections](https://arkoren.dev/docs/guides/collections/lazy-collections.md)
for evaluation timing, or [Fallible Collections](https://arkoren.dev/docs/guides/collections/fallible-collections.md) for
error policy.

Availability labels use **E** for eager `Slice` or `Map`, **L** for `LazySlice`
or `LazyMap`, and **T** for `TryLazySlice` or `TryLazyMap`. Streaming L and T
operations defer work until traversal. Terminal calls consume immediately;
buffering exceptions are stated beside the method. T callbacks that can fail
return an additional `error`, and terminal methods return an additional error.
Unless named `All`, terminals stop at the first error. Failed yielded values
must be ignored; callbacks do not receive source errors.

Import `arkoren.dev/foundation/collection`. Entries use these independent
application values and predicates:

```go
type Order struct {
	ID    string
	Zone  string
	Units int
	Ready bool
}

orders := collection.NewSlice([]Order{
	{ID: "ord-41", Zone: "north", Units: 3, Ready: true},
	{ID: "ord-42", Zone: "south", Units: 1, Ready: false},
	{ID: "ord-43", Zone: "north", Units: 2, Ready: true},
})
capacity := collection.NewMap(map[string]int{"north": 8, "south": 0})
ready := func(_ int, order Order) bool {
	return order.Ready
}
zone := func(_ int, order Order) string {
	return order.Zone
}
```

Examples are focused fragments, not consecutive declarations in one scope.
Map result comments describe entries, not a guaranteed display order. A slice
callback receives `(index, item)`; a map callback receives `(key, value)`.
Fallible callback indexes follow the [success rules](https://arkoren.dev/docs/guides/collections/fallible-collections.md#limits-and-indexes),
not stable source identities.

## Creating Collections

- [`NewSlice`](#newslice), [`NewMap`](#newmap): wrap in-memory containers.
- [`NewSliceAs`](#newsliceas), [`NewMapAs`](#newmapas): accept named containers.
- [`NewLazySlice`](#newlazyslice), [`NewLazyMap`](#newlazymap): wrap infallible iterators.
- [`NewTryLazySlice`](#newtrylazyslice), [`NewTryLazyMap`](#newtrylazymap), [`NewMapEntry`](#newmapentry): expose fallible sources.

### `NewSlice`

`NewSlice` wraps a `[]T` without copying: `collection.NewSlice([]Order{{ID: "ord-44"}})`.
The wrapper, original slice, and [`Items`](#items) result share storage.

### `NewMap`

`NewMap` wraps a `map[K]V` without copying:
`collection.NewMap(map[string]int{"north": 8})`. Keys must be comparable.
Nil eager maps support reads but still panic on indexed writes.

### `NewSliceAs`

`NewSliceAs` accepts a named slice type. With `type Orders []Order`, use
`collection.NewSliceAs(Orders{{ID: "ord-44"}})`. It returns `Slice[Order]`
without copying; [`ItemsAs`](#itemsas) converts back to a named type.

### `NewMapAs`

`NewMapAs` is the named-map counterpart of [`NewMap`](#newmap). With
`type Capacity map[string]int`, use `collection.NewMapAs(Capacity{"north": 8})`.

### `NewLazySlice`

`NewLazySlice` retains an `(index, item)` iterator. For a standard-library
`slices` import, `collection.NewLazySlice(slices.All(orders.Items()))` wraps
the same source data without consuming it.

Supply a non-nil function for all four iterator constructors. Nil is not an
empty iterator and panics on traversal. Sources must honor false yields and
own their [cleanup and cancellation](https://arkoren.dev/docs/guides/collections/lazy-collections.md#source-ownership).

### `NewLazyMap`

`NewLazyMap` retains an `iter.Seq2[K, V]`. With the standard-library `maps`
package, `collection.NewLazyMap(maps.All(capacity.Items()))` wraps a Go map
with unspecified order. It does not copy or traverse it.

### `NewTryLazySlice`

`NewTryLazySlice` retains an `(item, error)` iterator. A one-record source is:

```go
source := collection.NewTryLazySlice(func(yield func(Order, error) bool) {
	yield(Order{ID: "ord-44"}, nil)
})
```

No work runs yet. `source.Items()` returns that order and nil error. A source
that yields more records must stop after a false yield.

### `NewTryLazyMap`

`NewTryLazyMap` retains an iterator over `(MapEntry, error)` pairs:

```go
source := collection.NewTryLazyMap(func(yield func(collection.MapEntry[string, int], error) bool) {
	yield(collection.NewMapEntry("north", 8), nil)
})
```

`source.Items()` returns `north: 8` and nil error. Ignore entries paired with
non-nil errors.

### `NewMapEntry`

`NewMapEntry` constructs a typed pair: `collection.NewMapEntry("north", 8)`
has `Key == "north"` and `Value == 8`. The equivalent struct is
`collection.MapEntry[string, int]{Key: "north", Value: 8}`.

## Changing Evaluation

- [`Lazy`](#lazy), [`Try`](#try): defer later work or permit errors.
- [`Eager`](#eager), [`EagerAll`](#eagerall): collect reusable eager results.

### `Lazy`

*Slice, Map · E*

`Lazy` returns an iterator over shared source data. For example,
`orders.Lazy().Take(1).Items()` collects only `ord-41`. Conversion does not
copy elements, run callbacks, or cache a result.

### `Try`

*Slice, Map · E L*

`Try` exposes successful values through the fallible lazy form. For example,
`orders.Try().Items()` returns the orders and nil error. Later transformation
callbacks can return errors; conversion itself consumes nothing.

### `Eager`

*Slice, Map · L T*

`Eager` collects into an eager wrapper: `orders.Lazy().Take(1).Eager()` returns
a `Slice[Order]` containing `ord-41`. T returns `(collection, error)` and keeps
only the successful prefix before the first error. Outer result storage is new;
element copies remain shallow.

### `EagerAll`

*Slice, Map · T*

`EagerAll` is the continuing counterpart of [`Eager`](#eager):
`snapshot, err := orders.Try().EagerAll()`. It retains successes and joins
errors with `errors.Join`; it cannot retry or resume a terminal source.

## Selecting Records

- [`Filter`](#filter), [`Reject`](#reject), [`Unique`](#unique): select by values.
- [`Only`](#only), [`Except`](#except): select by map keys.

### `Filter`

*Slice, Map · E L T*

`Filter` retains matching values in source order: `orders.Filter(ready)` keeps
`ord-41` and `ord-43`. E allocates independent shallow result storage. On T,
source and predicate errors are forwarded, not filtered out.

### `Reject`

*Slice, Map · E L T*

`Reject` retains values for which the predicate is false:
`orders.Reject(ready)` keeps `ord-42`. It is the inverse selection of
[`Filter`](#filter), not an alias; T still forwards errors unchanged.

### `Unique`

*Slice · E L T; Map · L T*

`Unique` keeps the first slice item for each comparable callback key:
`orders.Unique(zone)` keeps `ord-41` and `ord-42`. Lazy forms retain a seen-key
set per traversal. On lazy maps, `capacity.Lazy().Unique()` takes no callback
and keeps the first entry per key. Materialization without `Unique` keeps the
last duplicate instead.

### `Only`

*Map · E L T*

`Only` retains listed keys: `capacity.Only("north")` contains `north: 8`.
T forwards all source errors, even if the associated key would be excluded.

### `Except`

*Map · E L T*

`Except` excludes listed keys: `capacity.Except("south")` contains `north: 8`.
Like [`Only`](#only), it does not hide source errors on T.

## Reshaping Values

- [`Map`](#map), [`FlatMap`](#flatmap), [`MapValues`](#mapvalues), [`MapKeys`](#mapkeys): change value or key types.
- [`Sort`](#sort), [`Reverse`](#reverse), [`Concat`](#concat), [`Merge`](#merge): order or combine collections.

### `Map`

*Slice · E L T*

`Map` transforms each item and can change its type:

```go
ids := orders.Map(func(_ int, order Order) string {
	return order.ID
})
```

`ids` contains `ord-41`, `ord-42`, and `ord-43`, in order. T callbacks return
`(value, error)` instead; a failed conversion yields an error rather than a value.

### `FlatMap`

*Slice · E L T*

`FlatMap` expands each item into zero or more values, preserving source and
expansion order:

```go
labels := orders.Take(1).FlatMap(func(_ int, order Order) []string {
	return []string{order.ID, order.Zone}
})
```

`labels` contains `ord-41` and `north`. A nil or empty expansion contributes
no values. T callbacks return an additional error.

### `MapValues`

*Map · E L T*

`MapValues` retains keys and transforms values, possibly to a new type:

```go
available := capacity.MapValues(func(_ string, slots int) bool {
	return slots > 0
})
```

`available` contains `north: true` and `south: false`.

### `MapKeys`

*Map · E L T*

`MapKeys` retains values and transforms keys. With `strings` imported:

```go
labels := capacity.MapKeys(func(name string, _ int) string {
	return strings.ToUpper(name)
})
```

`labels` contains `NORTH: 8` and `SOUTH: 0`. Colliding keys keep the last
materialized value. A Go map's unspecified order makes that winner unspecified;
lazy sources can retain duplicates until materialization.

### `Sort`

*Slice · E L T*

`Sort` sorts a shallow copy using a comparison function. With `cmp` imported:

```go
sorted := orders.Sort(func(left, right Order) int {
	return cmp.Compare(left.Units, right.Units)
})
```

`sorted` contains `ord-42`, `ord-43`, then `ord-41`; `orders` keeps its order.
The comparator must define a strict weak ordering. Sorting is not stable.
L and T read the whole source on each traversal before yielding; T yields only
the error if materialization fails, discarding the prefix.

### `Reverse`

*Slice · E L T*

`Reverse` reverses a shallow copy: `orders.Reverse()` begins with `ord-43`.
L and T buffer the whole source on each traversal; T discards partial results
if source materialization fails, as with [`Sort`](#sort).

### `Concat`

*Slice · E L T; Map · L T*

`Concat` appends another collection of the same form:
`orders.Take(1).Concat(orders.Skip(2))` contains `ord-41`, then `ord-43`.
E copies outer storage. Lazy forms traverse the first source before the second;
early stop can avoid the second source entirely. Lazy-map duplicate keys remain
until materialization, where later entries win. Eager maps use [`Merge`](#merge).

### `Merge`

*Map · E*

`Merge` gives the argument precedence in independent outer storage:
`capacity.Merge(collection.NewMap(map[string]int{"south": 4}))` contains
`north: 8` and `south: 4`, without changing either input. Nested references
remain shared. Nil inputs count as empty; a nil receiver merged with no entries
stays nil. Keep the return value rather than expecting receiver mutation.

## Windowing Records

- [`Take`](#take), [`Skip`](#skip), [`TakeWhile`](#takewhile), [`TakeUntil`](#takeuntil), [`SkipWhile`](#skipwhile), [`SkipUntil`](#skipuntil): select prefixes or suffixes.
- [`Nth`](#nth), [`ForPage`](#forpage), [`Chunk`](#chunk), [`Sliding`](#sliding), [`Partition`](#partition): select positions or groups.

### `Take`

*Slice · E L T; Map · L T*

`Take` keeps at most a count: `orders.Take(2)` contains the first two orders.
E shares the source array and clamps to bounds. Nonpositive limits produce no
items and do not invoke lazy sources, including no source errors on T.
Positive T limits count successes, not errors; duplicate map entries count
separately. Reaching the last requested success stops upstream immediately.

### `Skip`

*Slice · E L T; Map · L T*

`Skip` discards a prefix: `orders.Skip(2)` contains `ord-43`. Nonpositive
counts skip nothing; E clamps counts and shares the source array. T counts
successes while forwarding errors even inside skipped positions.

### `TakeWhile`

*Slice · E L T*

`TakeWhile` retains a prefix until the predicate is false:
`orders.TakeWhile(ready)` contains `ord-41`. It consumes but excludes the
first false item. E copies the selected prefix; T errors do not end selection
if the consumer continues.

### `TakeUntil`

*Slice · E L T*

`TakeUntil` retains a prefix until the predicate is true:
`orders.TakeUntil(ready)` is empty because the first order matches. The matching
item is excluded. It uses [`TakeWhile`](#takewhile) with the predicate inverted;
errors are forwarded, not inverted.

### `SkipWhile`

*Slice · E L T*

`SkipWhile` drops the leading matches: `orders.SkipWhile(ready)` contains
`ord-42` and `ord-43`. It includes the first false item and does not evaluate
the predicate on the rest. E shares the source array; T errors leave the
skipping phase unchanged.

### `SkipUntil`

*Slice · E L T*

`SkipUntil` drops items until the predicate is true:
`orders.SkipUntil(ready)` contains all three orders. It includes the matching
item and uses [`SkipWhile`](#skipwhile) with the predicate inverted. E shares
the source array.

### `Nth`

*Slice · E L T*

`Nth` samples every nth position: `orders.Nth(2)` contains `ord-41` and
`ord-43`; `orders.Nth(2, 1)` contains `ord-42`. The optional zero-based offset
defaults to zero; negative offsets clamp to zero. Offsets beyond the source
select nothing. The step must be positive or the call panics.

Every positive `int` step, including `math.MaxInt`, is valid without wrapping.
E allocates shallow result storage and returns nil for no selections. L counts
positions rather than source-supplied indexes and yields dense indexes. T
counts only successes and forwards errors even within skipped positions.
Continued traversal resumes selection progress; stopping avoids the next gap.

### `ForPage`

*Slice · E L T*

`ForPage` selects a one-based page: `orders.ForPage(2, 2)` contains `ord-43`.
It composes `Skip((page-1)*perPage)` and `Take(perPage)` without validating
arguments or overflow. Validate positive inputs and a representable product
before use. E shares storage; T forwards errors in skipped positions. This
selects collection positions, not [database result pages](https://arkoren.dev/docs/guides/queries/pagination.md).

### `Chunk`

*Slice · E L T*

`Chunk` forms consecutive groups: `orders.Chunk(2)` returns groups of two and
one orders. The size must be positive or the call panics. E returns
`[]Slice[T]` whose windows share the source array. L returns
`iter.Seq2[int, []T]`; T returns `iter.Seq2[[]T, error]`. Lazy chunks own
independent shallow storage and include the final short chunk. T errors leave
the partial chunk unchanged if traversal continues; stopping on an error does
not flush that prefix as a chunk.

### `Sliding`

*Slice · E L T*

`Sliding` forms complete windows: `orders.Sliding(2)` returns windows
`[ord-41, ord-42]` and `[ord-42, ord-43]`. An optional step defaults to one;
`orders.Sliding(2, 3)` returns only the first window. Incomplete final windows
are omitted. Nonpositive sizes or steps panic; positive steps through
`math.MaxInt` do not wrap the next start.

Return types match [`Chunk`](#chunk). E windows alias the source and each other
where they overlap; lazy windows own independent shallow storage. T counts
only successes, forwarding errors even in gaps without changing window or gap
progress. Stopping on a window or error stops consumption before the next gap.

### `Partition`

*Slice · E L T*

`Partition` splits matching and remaining items:
`selected, rest := orders.Partition(ready)` gives two ready orders and one
other order. E owns both shallow result slices. L and T consume at the call,
not on later traversal, and buffer replayable results. T returns two fallible
iterators: on a source or predicate error, both yield that error and no partial
partition values.

## Grouping Records

- [`GroupBy`](#groupby), [`KeyBy`](#keyby), [`CountBy`](#countby): retain groups, one item per key, or counts.

### `GroupBy`

*Slice · E L T*

`GroupBy` retains ordered groups: `orders.GroupBy(zone)["north"]` contains
`ord-41` and `ord-43`. Group slices and the outer map own shallow storage;
outer-map order is unspecified. L and T consume immediately. T returns
`(groups, error)` and preserves successful prefix groups on failure.

### `KeyBy`

*Slice · E L T*

`KeyBy` retains the last item for each key: `orders.KeyBy(zone)["north"]`
is `ord-43`. It allocates a new outer map. L and T consume immediately;
T returns its prefix map and the first error.

### `CountBy`

*Slice · E L T*

`CountBy` counts observed keys: `orders.CountBy(zone)` contains `north: 2`
and `south: 1`. Absent map keys read as zero. L and T consume immediately;
T returns prefix counts with the first error.

## Finding Answers

- [`First`](#first), [`FirstWhere`](#firstwhere), [`Last`](#last), [`Get`](#get): retrieve a value and an absence flag.
- [`Has`](#has), [`HasAny`](#hasany), [`Contains`](#contains), [`Every`](#every): test membership or conditions.
- [`Reduce`](#reduce), [`Len`](#len), [`IsEmpty`](#isempty), [`IsNotEmpty`](#isnotempty): summarize values or presence.

### `First`

*Slice · E L T*

`First` retrieves the initial item: `order, found := orders.First()` gives
`ord-41` and true. Empty input returns zero and false without panic. L consumes
at most one item; T returns `(item, found, error)` and stops even if the first
result is an error.

### `FirstWhere`

*Slice · E L T*

`FirstWhere` retrieves the first match:
`order, found := orders.FirstWhere(ready)` gives `ord-41` and true.
No match returns zero and false. Lazy traversal stops at the first match;
T also stops at the first source or predicate error.

### `Last`

*Slice · E L T*

`Last` retrieves the final item: `order, found := orders.Last()` gives
`ord-43` and true. Empty input returns zero and false. L reads the whole
source; T stops at an error and returns the last successful prefix item,
its flag, and that error.

### `Get`

*Map · E*

`Get` checks a key: `slots, found := capacity.Get("south")` gives zero and
true. An absent key gives zero and false, distinguishing absence from a stored zero.

### `Has`

*Map · E*

`Has` tests key presence: `capacity.Has("south")` is true even though its
stored value is zero. Lazy maps use [`HasAny`](#hasany) for key searches.

### `HasAny`

*Map · E L T*

`HasAny` tests whether any requested key exists:
`capacity.HasAny("west", "south")` is true. Lazy forms stop at a match;
T also stops at a source error. No requested keys returns false without
traversing the source or observing its errors.

### `Contains`

*Slice, Map · E L T*

`Contains` tests a predicate: `orders.Contains(ready)` is true. It stops at
the first match, or the first source/predicate error on T. Empty input is false.
It does not accept a plain value; use standard-library `slices.Contains` on
`Items()` for direct comparable-value membership.

### `Every`

*Slice, Map · E L T*

`Every` tests all items: `orders.Every(ready)` is false. It stops at the first
false result or, on T, error. Empty input returns true without invoking the predicate.

### `Reduce`

*Slice · E L T*

`Reduce` folds items into a value of any type, starting with the initial value:

```go
units := orders.Reduce(func(total, _ int, order Order) int {
	return total + order.Units
}, 0)
```

`units` is 6; empty input returns the initial zero. L and T consume at the
call. On T, source errors return the preceding accumulator; callback errors
return the accumulator supplied by that failing callback, with its error.

### `Len`

*Slice, Map · E L T*

`Len` counts items: `orders.Len()` is 3. Lazy forms traverse the entire source;
T returns the successful prefix count and first error. Lazy-map duplicate
entries count separately, even if materializing would collapse their keys.

### `IsEmpty`

*Slice, Map · E L T*

`IsEmpty` tests absence: `orders.IsEmpty()` is false. Lazy forms read at most
one result, not the whole source. An initial T error returns false and that
error; check the error before interpreting the boolean.

### `IsNotEmpty`

*Slice, Map · E L T*

`IsNotEmpty` negates [`IsEmpty`](#isempty): `orders.IsNotEmpty()` is true.
On T, an initial error returns true with that error, not proof of a successful
item. Always check its error first.

## Reading Map Parts

- [`Keys`](#keys), [`Values`](#values), [`LazyKeys`](#lazykeys), [`LazyValues`](#lazyvalues), [`Invert`](#invert): extract or reverse key/value roles.

### `Keys`

*Map · E · Package Function*

`Keys` collects keys into a `Slice`: `collection.Keys(capacity)` contains
`north` and `south`, in unspecified order. With `strings` imported,
`collection.Keys(capacity).Sort(strings.Compare)` gives repeatable order.

### `Values`

*Map · E · Package Function*

`Values` collects values into a `Slice`: `collection.Values(capacity)`
contains 8 and 0 in unspecified order. It does not establish an ordering
relationship with a separate [`Keys`](#keys) traversal.

### `LazyKeys`

*Map · L T*

`LazyKeys` projects keys without consuming now. On L,
`collection.LazyKeys(capacity.Lazy())` returns `LazySlice[string]`. On T,
`capacity.Try().LazyKeys()` is a method returning `iter.Seq2[string, error]`,
not a collection wrapper. Both preserve duplicates and source order.

### `LazyValues`

*Map · L T*

`LazyValues` is the value projection corresponding to [`LazyKeys`](#lazykeys):
`collection.LazyValues(capacity.Lazy())` returns `LazySlice[int]`;
`capacity.Try().LazyValues()` returns `iter.Seq2[int, error]`.
Ignore any key or value paired with a non-nil error.

### `Invert`

*Map · E · Package Function*

`Invert` swaps comparable keys and values: `collection.Invert(capacity)`
contains `8: north` and `0: south`. If values collide, the retained original
key is unspecified because Go-map iteration order is unspecified.

## Consuming Results

- [`Items`](#items), [`ItemsAs`](#itemsas), [`ItemsAll`](#itemsall), [`ItemsAsAll`](#itemsasall): extract or materialize containers.
- [`Iter`](#iter), [`Each`](#each), [`EachAll`](#eachall), [`TapEach`](#tapeach): consume or observe items.

### `Items`

*Slice, Map · E L T*

`Items` returns the native container: `orders.Items()` is a `[]Order` sharing
the eager source's storage. E preserves nil versus allocated-empty storage.
L and T materialize new shallow outer storage; T returns a prefix with the first
error. Slice materialization with no successes is nil; map materialization is
allocated even when empty. Repeated lazy calls repeat traversal, not cached data.

### `ItemsAs`

*Slice, Map · E L T*

`ItemsAs` is [`Items`](#items) with a named container result. With
`type Orders []Order`, use `orders.ItemsAs[Orders]()`. It has the same eager
aliasing, lazy allocation, and T first-error rules.

### `ItemsAll`

*Slice, Map · T*

`ItemsAll` retains successes and joins errors:
`items, err := orders.Try().ItemsAll()`. It reads the full finite source while
continuation is possible. It does not retry failed reads. Empty successful
slice output is nil; map output is allocated.

### `ItemsAsAll`

*Slice, Map · T*

`ItemsAsAll` is [`ItemsAll`](#itemsall) returning a named container. With
`type Orders []Order`, use `items, err := orders.Try().ItemsAsAll[Orders]()`.
Errors still accompany any collected successes.

### `Iter`

*Slice, Map · E L T*

`Iter` returns a standard `iter.Seq2` without consuming now:
`orders.Iter()` yields `(index, Order)`. Maps yield `(key, value)`.
T slices yield `(item, error)` and T maps `(MapEntry, error)`. Lazy collections
can also be ranged over directly. Iterator conversion changes no replay or
ownership guarantees.

### `Each`

*Slice, Map · E L T*

`Each` consumes items for side effects. With `fmt` imported:

```go
orders.Take(1).Each(func(_ int, order Order) {
	fmt.Println(order.ID)
})
```

This prints `ord-41`. T callbacks return an error; `Each` returns the first
source or callback error. Earlier side effects are not rolled back.

### `EachAll`

*Slice, Map · T*

`EachAll` continues after source or callback errors and joins them:

```go
err := orders.Try().EachAll(func(_ int, order Order) error {
	fmt.Println(order.ID)
	return nil
})
```

It prints all three IDs and returns nil. On fallible slices its callback index
advances for every successful source value, even if that callback fails.
Source errors bypass the callback and do not advance its index. Continuation
cannot make a terminal source recover.

### `TapEach`

*Slice, Map · E L T*

`TapEach` observes items while passing them along unchanged:

```go
observed := orders.Lazy().TapEach(func(_ int, order Order) {
	fmt.Println(order.ID)
})
```

Nothing prints until traversal. E instead calls the observer immediately and
returns the original collection, not a copy. On T, callback failures replace
their value with an error and do not advance that streaming callback's index.

## Encoding JSON

- [`JSONSchemaType`](#jsonschematype): declare eager wire shapes.
- [`MarshalJSON`](#marshaljson), [`MarshalJSONTo`](#marshaljsonto): encode containers.
- [`UnmarshalJSON`](#unmarshaljson), [`UnmarshalJSONFrom`](#unmarshaljsonfrom): decode containers.

Use `encoding/json/v2` for ordinary `json.Marshal` and `json.Unmarshal` calls.
The compatibility method names below still use JSON v2 behavior, not legacy
encoding defaults.

### `JSONSchemaType`

*Slice, Map · E*

`JSONSchemaType` declares the identical native container type as a
`reflect.Type`: `orders.JSONSchemaType()` is `reflect.TypeFor[[]Order]()`.
Schema tools use that declaration for [output descriptions](https://arkoren.dev/docs/guides/validation/json-schema.md#describe-output),
including nested element metadata and byte slices' base64 shape. It adds no
runtime rules; use the [named-container adapter](https://arkoren.dev/docs/guides/validation/collections.md#named-containers)
for logical validation. Lazy types do not declare this protocol.

### `MarshalJSON`

*Slice, Map · E L T*

`MarshalJSON` is the compatibility encoding method. Prefer ordinary encoding,
for example `data, err := json.Marshal(capacity.Only("north"))`, whose successful
result is `{"north":8}`. Default nil non-byte slices encode as `[]`, maps as
`{}`, and byte slices as base64 strings, with `""` for empty storage.

Lazy forms materialize the whole finite source; encoding is not progressive
item delivery. T returns the first source error instead of encoding a partial
collection. Encoding errors also remain visible.

### `MarshalJSONTo`

*Slice, Map · E L T*

`MarshalJSONTo` implements the JSON v2 streaming interface. With a configured
`*jsontext.Encoder` named `encoder`, call `err := orders.MarshalJSONTo(encoder)`.
Encoder options control formatting; encoding and writer errors are returned.
Lazy forms still materialize first, and T writes no collection data if source
materialization fails. For ordinary use, prefer [`MarshalJSON`](#marshaljson)'s
`json.Marshal` example.

### `UnmarshalJSON`

*Slice, Map · E L T · Pointer Receiver*

`UnmarshalJSON` is the compatibility decoding method. Prefer `json.Unmarshal`:

```go
var decoded collection.Map[string, int]
err := json.Unmarshal([]byte(`{"north":8}`), &decoded)
```

On success, `decoded["north"]` is 8. E follows JSON v2 merging rules, and a
decoding error can leave partial receiver updates. Lazy receivers replace their
iterator only after successful decoding, installing a replayable source over
the entire decoded container. T decoded values carry nil source errors. Lazy
decoding is not streaming input traversal.

### `UnmarshalJSONFrom`

*Slice, Map · E L T · Pointer Receiver*

`UnmarshalJSONFrom` implements JSON v2 decoding from a `*jsontext.Decoder`.
Given a decoder named `decoder` and the eager map `decoded` above, call
`err := decoded.UnmarshalJSONFrom(decoder)`. Receiver updates and failure rules
match [`UnmarshalJSON`](#unmarshaljson). Both streaming interface types come
from `encoding/json/jsontext`.

For exact declarations, see the [`collection` API reference](https://pkg.go.dev/arkoren.dev/foundation/collection).

---

Source: https://arkoren.dev/docs/guides/feature-flags/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/feature-flags/overview)

# Feature Flags Overview

A checkout rollout needs to select the new flow without removing the old one.
A **feature flag** supplies that decision, or a scalar setting, for a particular
subject. Your application reads flags through `*feature.Flags`; the selected
[driver](https://arkoren.dev/docs/guides/feature-flags/drivers.md) evaluates values managed in memory or in a provider's tools.
The portable contract does not create flags or change provider rules.

Flags are not authorization. Enforce [account permissions](https://arkoren.dev/docs/guides/authorization/overview.md)
regardless of which flow a flag selects, including when evaluation fails.

## Flag Decisions

Choose the safe behavior before declaring a flag. A `feature.Flag[T]` binds the
literal provider key, Go value type, and fallback in one immutable definition.
Keep the definition in the application package that owns the behavior and share
it with tests. Import `feature` from `arkoren.dev/core/service/feature`:

```go
var expressCheckout = feature.NewFlag("checkout-express", false)
```

This definition does not register a provider flag. Configure that exact key in
your provider; its schema is still checked at evaluation time. Empty names and
zero definitions are invalid: `Err()` reports `feature.ErrInvalidFlag` without
contacting a driver.

With an existing configured facade `flags`, operation context `ctx`, and stable
account ID `accountID`, read the decision:

```go
subject := feature.Subject{Key: accountID}
enabled, err := flags.Value(ctx, expressCheckout, subject)
```

Use `enabled` to select the checkout flow, after permission checks. Every non-nil
error comes with the declared fallback, so this flag selects the old flow on
failure. Focused snippets keep error results visible but omit routine propagation;
complete application code still handles its errors.

The same fallback can mean several different things:

| Evaluation | Value | Error |
| --- | --- | --- |
| Disabled flag | Declared fallback | Nil |
| No value applies to this subject | Declared fallback | Nil |
| Missing provider key | Declared fallback | `feature.ErrNotFound` |
| Incompatible value or numeric conversion | Declared fallback | `feature.ErrTypeMismatch` |
| Driver still loading | Declared fallback | `feature.ErrNotReady` |
| Invalid subject or definition | Declared fallback | `feature.ErrInvalidSubject` or `feature.ErrInvalidFlag` |

Errors wrap these identities; classify them with standard-library `errors.Is`.
Cancellation and other provider errors also return the fallback. A successful
provider result can itself equal the fallback, so the value alone does not prove
whether evaluation succeeded. Test your application's
[failed-evaluation policy](https://arkoren.dev/docs/guides/feature-flags/testing.md#failed-evaluations) explicitly.

### Scalar Settings

Use a value-bearing flag when a rollout needs more than a boolean. Definitions
support booleans, strings, signed integers, and floating-point values, including
named types:

```go
type CheckoutLayout string

var checkoutLayout = feature.NewFlag("checkout-layout", CheckoutLayout("standard"))

layout, err := flags.Value(ctx, checkoutLayout, subject)
```

`layout` retains the `CheckoutLayout` type. JSON objects and unsigned integers
are not supported. Integer values can be read as floats; float values can be read
as integers only when whole and representable. Reading `3.5` as an integer, or
`300` as `int8`, returns the fallback with `ErrTypeMismatch`, not a truncated value.

## Subject Targeting

A **subject** identifies who the decision is for and supplies facts that provider
rules can match. Build those facts from trusted application state rather than
letting request fields choose account identity or permissions. For an account
already loaded by your application, use a small typed input:

```go
type CheckoutAccount struct {
	ID    string
	Plan  string
	Seats int64
	Beta  bool
}
```

Convert an existing `account CheckoutAccount` into the contract's attribute map:

```go
subject := feature.Subject{
	Key: account.ID,
	Attributes: map[string]any{
		"plan":  account.Plan,
		"seats": account.Seats,
		"beta":  account.Beta,
	},
}
enabled, err := flags.Value(ctx, expressCheckout, subject)
```

The facade accepts only `string`, `bool`, `int64`, `float64`, `[]string`, and
`time.Time` attribute values. A plain `int` is invalid; convert it explicitly.
Invalid attributes fail before driver access, and validation errors name the
attribute rather than its value. Do not mutate the map or its slices during use.
Send only facts approved for the provider to receive.

Keep subject keys stable when your provider uses them for rollout bucketing.
Arkoren does not guarantee identical buckets across providers, unchanged
assignments after rule changes, exact percentages in a population, or atomic
rollout changes across replicas. A zero subject is valid and anonymous; provider
handling differs, especially for [LaunchDarkly](https://arkoren.dev/docs/guides/feature-flags/drivers.md#launchdarkly).

Segments and targeting rules remain provider-owned. A subject key identifies the
evaluated entity, not a segment; attributes supply facts, not portable segment
membership instructions. The memory driver supports key overrides but ignores
attributes. The subject is always an explicit argument, not inferred from `ctx`.

## Evaluation Details

When investigating an unexpected flow, read the value and its explanation in one
evaluation rather than evaluating again for diagnostics:

```go
detail, err := flags.ValueDetail(ctx, expressCheckout, subject)
```

`detail.Value` has the same fallback behavior as `Value`. `detail.Reason` describes
the driver's choice: static value, targeting match, split, default, disabled,
cached, or unknown. On failure it is `ReasonError`, the variant is empty, and the
error remains available separately. A no-value success preserves the driver's
reason and variant while substituting the fallback.

`detail.Variant` is optional provider metadata, not the flag key or subject ID.
OpenFeature passes through a provider's variant; LaunchDarkly reports a variation
index as text. Reason coverage differs too: not every percentage rollout reports
`ReasonSplit`. Use these fields for diagnostics, not authorization or a portable
business branch. [Drivers](https://arkoren.dev/docs/guides/feature-flags/drivers.md) explains each mapping.

## Flag Namespaces

If several flags share a key prefix and the same typed fallback, a namespace
keeps that naming policy together:

```go
checkoutFlags := feature.NewNamespace("checkout", false)
express := checkoutFlags.Flag("express")
```

`express.Name()` is `checkout:express`, and its fallback is `false`. Each child
inherits the exact fallback and Go type; use another namespace or standalone
definition when either differs. Create the assembled key in the provider.
Namespaces add neither provider isolation nor targeting or authorization.

If the provider uses dotted keys, derive a separate naming policy:

```go
dotted := checkoutFlags.Derive(func(names *feature.NamespaceBuilder) {
	names.Separator(".")
})
refunds := dotted.Subnamespace("payments").Flag("refunds")
```

The resulting key is `checkout.payments.refunds`; the original namespace is
unchanged. A subnamespace joins its own name using the parent's separator before
applying child callbacks to future joins. An empty separator concatenates names.
Names are literal: no trimming, escaping, or normalization prevents collisions
such as `a:b` plus `c` and `a` plus `b:c`.

Empty prefixes or child names and zero namespaces retain `ErrInvalidFlag` through
all descendants. Construction callbacks run once in order, including for invalid
definitions; nil callbacks panic, and retained drafts cannot change published
values. Namespace formatting and structured logging redact names and fallbacks;
explicit `Name()` and `Fallback()` accessors expose them.

Prefer stable flag keys with subject targeting over per-account keys. If dynamic
keys really exist in the provider, validate or encode components before joining
them and format numeric suffixes explicitly. Keys enter telemetry: personal IDs
and unbounded names create privacy and cardinality problems.

## Raw Requests

Application decisions should use the typed facade. A custom adapter or contract
test can call a driver directly with an explicit request:

```go
result, err := flags.Driver().Evaluate(ctx, feature.Request{
	Key:     expressCheckout.Name(),
	Kind:    feature.KindBool,
	Subject: subject,
})
```

This bypasses facade validation, fallback conversion, events, and telemetry.
`result.Value == nil` means no value was supplied; it does not install your
application fallback. Drivers first reject an already-canceled context, then
reject empty or unknown kinds with `ErrTypeMismatch` before readiness, lookup,
disabled/default shortcuts, or provider I/O. Only `KindBool`, `KindString`,
`KindInt`, and `KindFloat` are valid. Unknown kind strings can round-trip through
JSON and `String()` without becoming valid requests.

## Events

Fallbacks let requests continue, but repeated provider failures still need
attention. At facade construction, connect an existing `*event.Bus` named `bus`
from `arkoren.dev/core/event`:

```go
flags := feature.New(driver, func(client *feature.Builder) {
	client.Events(bus)
})
```

Each driver evaluation error publishes `feature.evaluation.failed` with the key
and error. Successful disabled/default results publish nothing. Invalid
definitions, invalid attributes, and conversion errors detected by the facade
also publish nothing; use telemetry to observe those failures.

The payload has no structured subject, attribute, or value fields, but provider
error text can contain sensitive information. Classify errors instead of
forwarding raw payloads to untrusted destinations. Count or sample hot-path
failures rather than alerting once per evaluation. See
[built-in events](https://arkoren.dev/docs/guides/events/built-in-events.md#feature-flags) for the event contract.

## Telemetry

Evaluations follow the [shared telemetry handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle)
unless construction explicitly supplies `Builder.Telemetry`. The process default
is resolved on each evaluation and is a no-op until configured. `Builder.Name`
sets a fixed provider label; its default is `default`.

When enabled, each evaluation records an internal `feature evaluate` span,
`arkoren.feature.evaluations` counter, and
`arkoren.feature.evaluation.duration` histogram. Spans and counters include the
key, provider label, and reason; the histogram includes key and provider label.
Failures add the error class as `error.type`, never the error text. Nonempty
variants appear on spans only.

Facade telemetry does not record subject keys, attributes, or flag values.
Keep keys and provider labels bounded and nonpersonal, and review variant names
too. Native SDK hooks and analytics have their own data policy; facade redaction
does not sanitize them. [Instrumented components](https://arkoren.dev/docs/guides/observability/instrumented-components.md)
lists the metric and attribute contracts.

For every function and option, see the
[`feature` API reference](https://pkg.go.dev/arkoren.dev/core/service/feature).

---

Source: https://arkoren.dev/docs/guides/feature-flags/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/feature-flags/drivers)

# Feature Flag Drivers

Choose memory for deterministic application tests or process-local settings.
Choose OpenFeature when your flag service has an OpenFeature Go provider or your
application already manages an OpenFeature client. Choose the native LaunchDarkly
driver when you want direct integration with LaunchDarkly's official server SDK.
All three expose the same [typed evaluation](https://arkoren.dev/docs/guides/feature-flags/overview.md#flag-decisions), but
targeting rules, update delivery, and outage behavior are not interchangeable.

## Compatibility Matrix

This comparison covers Arkoren's exposed evaluation contract, not provider flag
administration. Scalar flags are booleans, strings, signed integers, and floats.
Every driver accepts standalone and namespace-derived definitions; neither form
creates provider keys or enforces a provider schema.

| Driver | Scalar Flags | Key Targeting | Attribute Targeting | Object Flags | Native Client | Remote-Free Evaluation |
| --- | --- | --- | --- | --- | --- | --- |
| [Memory](#memory) | Yes | Yes — Explicit nonempty-key overrides. | No | No | No | Yes |
| [OpenFeature](#openfeature) | Yes | Partial — Depends on the configured provider. | Partial — Attributes pass through; rule support depends on the provider. | No | Yes | Partial — Depends on the provider's evaluation and refresh model. |
| [LaunchDarkly](#launchdarkly) | Yes | Yes — User contexts; anonymous subjects share one key. | Yes — Subject attributes become user-context attributes. | No | Yes | Yes — Evaluates locally after initial synchronization. |

No driver guarantees atomic rule changes across processes or identical rollout
assignments across providers. Memory has independent state in each instance.
OpenFeature's provider determines remote availability requirements. LaunchDarkly
can evaluate its last received data during a connection outage, but that does not
prove the data is current or let a new process start without initial data.

Run installation commands from your application's Go module. Examples use
`feature` from `arkoren.dev/core/service/feature`; `ctx` is an operation context.
The facade itself has no lifecycle. Register driver startup, cleanup, and
readiness with your existing [application lifecycle](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md),
and do not evaluate an owned provider until startup succeeds.

## Memory

`memoryfeature` is included in `arkoren.dev/core` and needs no external service.
Use it when your application controls the stored values and needs neither
provider targeting rules nor shared state between processes.

```go
import "arkoren.dev/core/service/feature/memoryfeature"

driver := memoryfeature.New()
flags := feature.New(driver)
```

Construction performs no I/O, has no options, and starts no background work.
There are no `Start`, `Stop`, or `Ping` hooks to register.

To supply the checkout decision, share the application's typed definition:

```go
expressCheckout := feature.NewFlag("checkout-express", false)
err := driver.Set(expressCheckout, true)
```

The next [facade evaluation](https://arkoren.dev/docs/guides/feature-flags/overview.md#flag-decisions) returns `true` unless
the flag is disabled or a subject override selects another value. Setup errors
must be handled before the value is used. `Set` retains existing overrides and
disabled state; it does not install the definition's fallback as a stored value.
[Testing](https://arkoren.dev/docs/guides/feature-flags/testing.md#memory-fixtures) covers overrides and state changes.

The driver is concurrent-safe and loses all state when discarded. It ignores
subject attributes, has no rollout engine, and never matches an override whose
key is empty. Separately constructed definitions with the same provider name
still address the same stored entry; they do not establish a schema.

See the [`memoryfeature` API](https://pkg.go.dev/arkoren.dev/core/service/feature/memoryfeature).

## OpenFeature

OpenFeature connects an evaluation API to a provider. Supply an
`openfeature.FeatureProvider` from your chosen vendor's Go integration, configured
with that vendor's endpoint and credentials. Provider documentation determines
targeting support, refresh intervals, network calls, caching, and outage recovery;
Arkoren does not add missing capabilities.

### Provider Setup

Install the Arkoren module and the Go provider module you selected:

```sh
go get arkoren.dev/driver/openfeature
```

With the configured provider named `provider`, construct an owned driver:

```go
import "arkoren.dev/driver/openfeature/openfeaturefeature"

driver := openfeaturefeature.New(provider)
flags := feature.New(driver)
```

`New` performs no I/O. `Start` registers and initializes the provider in a private
OpenFeature API instance; it neither uses nor replaces the global provider.
One provider belongs to one owned driver. Register `Stop` for cleanup, `Start`
for startup, and `Ping` for readiness. `Ping` performs no network call and
requires the SDK's provider state to be `READY`; a stale provider is not ready
even if it can still serve a cached evaluation.

An owned driver cannot restart after `Stop` or failed `Start`; construct a new
one to retry initialization. Stopping before startup also makes it terminal,
without initializing or shutting down the unused provider. Failed startup
attempts provider cleanup and leaves evaluation unavailable.

Owned lifecycle calls serialize, and waiting for another lifecycle call is not
bounded by `ctx`. Context-aware providers must cooperate with initialization and
cleanup cancellation. Legacy contextless shutdown is not bounded by the cleanup
context, and a legacy initializer can outlive canceled startup inside the SDK.
Do not call the same driver's lifecycle recursively from provider callbacks.

### Ambient Context

If every rule needs a deployment fact, supply it once at construction. Import
`openfeature` from `github.com/open-feature/go-sdk/openfeature`:

```go
driver := openfeaturefeature.New(provider, func(features *openfeaturefeature.Builder) {
	features.EvaluationContext(openfeature.NewTargetlessEvaluationContext(
		map[string]any{"environment": "production"},
	))
})
```

This is an alternative construction. Per-evaluation subject attributes replace
ambient attributes of the same name; a nonempty subject key replaces the ambient
targeting key. An empty key leaves the ambient key in effect. If neither provides
a key, providers that require one report `feature.ErrInvalidSubject`.

Construction copies outer attribute maps at input and publication, not nested
values. Retained drafts cannot change the published context; keep shared nested
values immutable. Construction callbacks run once in order; nil callbacks panic.
Configure credentials, polling, and network budgets on the provider, not on the
Arkoren builder.

### Evaluation Results

After startup, use the shared [flag decision](https://arkoren.dev/docs/guides/feature-flags/overview.md#flag-decisions).
The subject key becomes the targeting key, and accepted attributes pass through
unchanged. Each provider decides which types its rules support. Provider-side
segments retain that provider's own identifiers and rules.

Missing flags map to `ErrNotFound`; type and parse errors to `ErrTypeMismatch`;
not-ready providers to `ErrNotReady`; missing targeting keys and invalid contexts
to `ErrInvalidSubject`. Other failures retain the provider error code in the
error. A reported disabled result supplies no value, so the facade uses the
definition's fallback without an error.

Known reasons map to their portable equivalents; `STALE` maps to `ReasonCached`.
Empty or unrecognized reasons become `ReasonUnknown`. Variants pass through
unchanged. An OpenFeature default result can include a real provider value:
`ReasonDefault` alone does not mean the application fallback was used.

### Native Ownership

If your application already registers an OpenFeature provider and manages its
hooks, domains, and shutdown, borrow its initialized `*openfeature.Client`
named `client`:

```go
driver := openfeaturefeature.NewFrom(client)
flags := feature.New(driver)
```

`NewFrom` accepts no options. `Start` and `Stop` do nothing; the application
retains provider registration and cleanup. `Ping` still reports provider state.
Do not rebind or shut down a `New` driver's owned provider through native access.

To attach an existing `openfeature.Hook` named `hook` during setup, use the native
client before concurrent evaluation:

```go
driver.Client().AddHooks(hook)
```

Hooks can receive subject data and values omitted by facade telemetry. Review
their export policy before use. The owned native client is available before
startup, but direct evaluation then returns SDK no-op results rather than the
facade's `ErrNotReady`. Native calls bypass facade validation and telemetry.
Native access does not add portable object flags,
and Arkoren ships no OpenFeature flag-administration or webhook adapter.

### Local Provider

The SDK's memory provider exercises the OpenFeature path without a remote
service. Define the fixture separately from driver construction:

```go
import "github.com/open-feature/go-sdk/openfeature/memprovider"

provider := memprovider.NewInMemoryProvider(map[string]memprovider.InMemoryFlag{
	"checkout-express": {
		Key:            "checkout-express",
		State:          memprovider.Enabled,
		DefaultVariant: "express",
		Variants:       map[string]any{"express": true},
	},
})
```

Pass this provider to `New` and run the lifecycle described above. After startup,
evaluating `feature.NewFlag("checkout-express", false)` returns `true` with no
error. `Stop` releases the provider; no server needs cleanup. This proves SDK
wiring, not your production provider's targeting or refresh behavior.

See the [`openfeaturefeature` API](https://pkg.go.dev/arkoren.dev/driver/openfeature/openfeaturefeature).

## LaunchDarkly

Choose this driver for LaunchDarkly's official Go server SDK. The SDK receives
flag updates and evaluates locally. You need a LaunchDarkly environment, its
server-side SDK key, and the application's flags configured in that environment.
Load the key from approved secret configuration rather than embedding it in code.

### Provider Setup

```sh
go get arkoren.dev/driver/launchdarkly
```

With the environment's key already loaded as `sdkKey`, construct an owned driver:

```go
import "arkoren.dev/driver/launchdarkly/launchdarklyfeature"

driver, err := launchdarklyfeature.New(sdkKey)
```

`New` performs no I/O and rejects an empty key. Handle that error before
constructing the facade:

```go
flags := feature.New(driver)
```

Register `Start`, `Stop`, and `Ping`. Startup builds the SDK client and waits for
initial flag data; failure closes the acquired client instead of publishing it.
The default SDK receives updates from LaunchDarkly; an SDK-configured persistent
store can provide last-known data. Treat that data as potentially stale, not as
proof of a live remote connection.
`Ping` performs no I/O and checks that a client exists and reports initialized.
It is not a remote connection or freshness check. Before startup and after stop,
evaluation returns `ErrNotReady`.

Unlike OpenFeature, an owned LaunchDarkly driver can retry failed startup or
start again after stop. Starting an already running driver fails. `Stop` closes
the client and flushes pending analytics; the SDK's close is not bounded by the
cleanup context. Owned lifecycle calls serialize without a context-bounded wait.
Native construction hooks must not recursively call the driver's lifecycle.

### Startup Budget

Startup waits five seconds by default. If your deployment needs a longer initial
synchronization budget, configure it when constructing the driver. Import
standard-library `time`:

```go
driver, err := launchdarklyfeature.New(sdkKey, func(features *launchdarklyfeature.Builder) {
	features.StartWait(10 * time.Second)
})
```

Zero selects the five-second default; a negative wait returns immediately and
fails unless the source is already ready. Startup checks `ctx` before client
construction, but subsequent cancellation does not interrupt the SDK's wait.
`StartWait` is not an evaluation timeout.

For native endpoints, storage, event settings, or network budgets, supply an SDK
hook. This alternative construction disables SDK analytics, not facade telemetry:

```go
import (
	ld "github.com/launchdarkly/go-server-sdk/v7"
	"github.com/launchdarkly/go-server-sdk/v7/ldcomponents"
)

driver, err := launchdarklyfeature.New(sdkKey, func(features *launchdarklyfeature.Builder) {
	features.SDK(func(config *ld.Config) {
		config.Events = ldcomponents.NoEvents()
	})
})
```

The hook runs at `Start`, not at construction. Published settings are detached
from the construction draft, but native objects and captured state remain shared.
All construction callbacks run once in order; nil callbacks panic.

### Evaluation Results

Create `checkout-express` as a boolean flag in the selected environment, then use
the shared [flag decision](https://arkoren.dev/docs/guides/feature-flags/overview.md#flag-decisions) after startup.

> [!WARNING]
> An off flag or failed prerequisite returns the definition's fallback, not
> LaunchDarkly's configured off variation. Choose a safe application fallback.
> If a configured provider value must be served, keep the flag on with its
> prerequisites satisfied and select that value through targeting rules.

Each subject becomes a context of kind `user`. An empty key becomes the anonymous
key `anonymous`, so all anonymous subjects share one identity and rollout bucket.
Attributes named `key` or `kind` cannot replace this identity. String lists become
arrays, and `time.Time` becomes Unix milliseconds for date rules. The SDK can
reject reserved attributes with incompatible types as `ErrInvalidSubject`.

Target and rule matches report `ReasonTargetingMatch`; fallthrough reports
`ReasonDefault`. Experiment rule matches and fallthrough report `ReasonSplit`;
ordinary percentage rollouts need not. Variants are variation indexes formatted
as strings, such as `"1"`, not named segments or variation labels.

Missing keys, wrong types, uninitialized clients, and invalid contexts map to the
corresponding portable errors. Malformed flags and SDK exceptions retain their
error kind. Null variations supply no value and therefore use the fallback.
Fractional numbers requested as integers fail rather than truncate. Evaluation
is local and checks cancellation only before SDK evaluation.

### Native Ownership

If your application already owns a configured `*ld.LDClient` named `client`,
wrap it with `launchdarklyfeature.NewFrom(client)`. The wrapper accepts no
options, and its `Start` and `Stop` do nothing. You must close the client after
all users finish, including any client returned alongside an initialization
error. An owned driver's `Client()` is nil before successful startup and after
stop; do not close its client yourself.

For custom analytics, first configure the matching metric in LaunchDarkly. After
successful startup, track the metric's event name with the same stable account
identity used for evaluation:

```go
import "github.com/launchdarkly/go-sdk-common/v3/ldcontext"

err := driver.Client().TrackEvent("checkout-completed", ldcontext.New(accountID))
```

A nil error confirms SDK acceptance, not remote delivery. Analytics can transmit
context data outside facade telemetry's redaction policy; review permitted
attributes and event configuration before enabling it. Arkoren ships no native
flag-administration or webhook adapter.

### Local Data

The SDK's test data source exercises local evaluation without a LaunchDarkly
connection. Using the SDK imports above, define a checkout fixture:

```go
import "github.com/launchdarkly/go-server-sdk/v7/testhelpers/ldtestdata"

data := ldtestdata.DataSource()
data.Update(data.Flag("checkout-express").VariationForAll(true))
```

Build a client with remote analytics disabled. Retain cleanup even if client
initialization fails:

```go
client, err := ld.MakeCustomClient("local-checkout", ld.Config{
	DataSource: data,
	Events:     ldcomponents.NoEvents(),
}, 0)
if client != nil {
	defer client.Close()
}
```

Handle `err` before wrapping the client with `NewFrom` and constructing the
facade. Evaluating `feature.NewFlag("checkout-express", false)` then returns
`true` without a remote service. The deferred close is yours because the client
is borrowed; there is no server to remove. Use [memory fixtures](https://arkoren.dev/docs/guides/feature-flags/testing.md)
when only application behavior, rather than SDK integration, is under test.

See the [`launchdarklyfeature` API](https://pkg.go.dev/arkoren.dev/driver/launchdarkly/launchdarklyfeature).

---

Source: https://arkoren.dev/docs/guides/feature-flags/testing.md
[HTML documentation](https://arkoren.dev/docs/guides/feature-flags/testing)

# Feature Flag Testing

Test a rollout by controlling its inputs, not by waiting for a provider's rules
to change. The memory driver gives each test independent state while retaining
the real facade's type checks and fallback behavior. Pass that facade into your
application code to assert the selected behavior; use an error-returning fake
when the test needs a failure memory cannot produce.

## Memory Fixtures

Share the application's flag definition with its tests. Here `expressCheckout`
is the `feature.Flag[bool]` declared by the checkout package with key
`checkout-express` and fallback `false`. In your `_test.go` file, import `feature`
from `arkoren.dev/core/service/feature` and `memoryfeature` from its
`memoryfeature` subpackage.

With an existing `*testing.T` named `t`, create a fresh driver and install the
enabled value:

```go
driver := memoryfeature.New()
if err := driver.Set(expressCheckout, true); err != nil {
	t.Fatal(err)
}
flags := feature.New(driver)
```

Pass `flags` to the checkout service under test. Every subject initially receives
`true`; assert that the service selects the express flow. A second test can
install `false` and assert the standard flow. Keep fixtures local to each test
so parallel tests do not change one another's decisions.

`Set` creates the stored entry or replaces its default value, retaining existing
overrides and disabled state. Its value must have the definition's exact Go type,
including named types. Invalid definitions return `ErrInvalidFlag` without
mutation. The application's fallback is not installed as a provider default.

## Subject Overrides

If a rollout targets one account, arrange that identity explicitly rather than
trying to reproduce percentage bucketing. Using the memory `driver` and `flags`
above, override one stable account key:

```go
if err := driver.SetFor(expressCheckout, "account-river", false); err != nil {
	t.Fatal(err)
}
subject := feature.Subject{Key: "account-river"}
detail, err := flags.ValueDetail(t.Context(), expressCheckout, subject)
if err != nil || detail.Value || detail.Reason != feature.ReasonTargetingMatch {
	t.Fatalf("override = %+v, %v; want false, targeting match, nil", detail, err)
}
```

This account receives `false`; other accounts still receive the stored `true`.
`SetFor` creates an override-only flag if the key is absent. In that case an
unmatched subject receives the definition's fallback with `ReasonDefault` and
no error. An empty subject key never matches an override, even one stored under
the empty string.

Use the same [namespace-derived definition](https://arkoren.dev/docs/guides/feature-flags/overview.md#flag-namespaces) in
evaluation and fixture setup when the application uses one. `Set` and `SetFor`
address its assembled provider key without special namespace configuration.

Memory ignores subject attributes: these tests prove application behavior for
chosen identities, not provider attribute rules, segment membership, or rollout
distribution. Use the [local SDK fixtures](https://arkoren.dev/docs/guides/feature-flags/drivers.md#local-provider) to test
adapter wiring, and a disposable provider integration environment when the
provider's actual rules are the subject of the test.

## Disabled Flags

A disabled flag and a missing flag both select the fallback, but only the missing
flag is an error. To test that distinction, first install the shared checkout
definition in memory, then disable it:

```go
driver.Disable(expressCheckout.Name())
detail, err := flags.ValueDetail(t.Context(), expressCheckout, feature.Subject{})
if err != nil || detail.Value || detail.Reason != feature.ReasonDisabled {
	t.Fatalf("disabled = %+v, %v; want false, disabled, nil", detail, err)
}
```

Disabling retains the entry's default and overrides; `Enable` makes them visible
again. Disabling or enabling a missing key does nothing.

To exercise a missing definition in the provider instead, delete the entry.
Import standard-library `errors` for identity checks:

```go
driver.Delete(expressCheckout.Name())
enabled, err := flags.Value(t.Context(), expressCheckout, feature.Subject{})
if enabled || !errors.Is(err, feature.ErrNotFound) {
	t.Fatalf("missing = %v, %v; want false, ErrNotFound", enabled, err)
}
```

Deletion removes both the stored value and overrides and is harmless if repeated.
Use these distinct fixtures to verify whether your application tolerates a
missing flag or treats it as configuration failure.

## Failed Evaluations

The fallback is available even when evaluation fails; continuing the request is
an application policy, not an automatic retry or permission grant. For checkout,
a safe policy can keep the standard flow available while observing the failure.
Assert both the chosen flow and the error reporting your application requires.
Account permissions must remain enforced on both paths.

To force readiness or backend errors without a network dependency, implement a
small test-only driver. Import standard-library `context` as well as `errors`:

```go
type unavailableFlags struct{}

func (unavailableFlags) Evaluate(ctx context.Context, _ feature.Request) (feature.Result, error) {
	if err := ctx.Err(); err != nil {
		return feature.Result{}, err
	}
	return feature.Result{}, feature.ErrNotReady
}
```

This fake deliberately models one failure path; it is not a production driver
or a conforming implementation for arbitrary requests. Wrap it with the real
facade to check that your shared definition supplies the fallback:

```go
flags := feature.New(unavailableFlags{})
detail, err := flags.ValueDetail(t.Context(), expressCheckout, feature.Subject{})
if detail.Value || detail.Reason != feature.ReasonError ||
	!errors.Is(err, feature.ErrNotReady) {
	t.Fatalf("unavailable = %+v, %v; want false, error reason, ErrNotReady", detail, err)
}
```

Pass this facade into your application test to verify the standard flow and its
failure policy. Also test canceled contexts if your policy distinguishes request
cancellation from provider unavailability. Do not add sleeps to create failures.

For observability assertions, driver errors publish
[evaluation-failed events](https://arkoren.dev/docs/guides/feature-flags/overview.md#events) when a bus is configured. Invalid
definitions, invalid attributes, and facade conversion errors do not publish
that event. [Telemetry test helpers](https://arkoren.dev/docs/guides/testing/fakes.md) can inspect spans and
metrics, which cover those facade failures too. Prefer an explicit test telemetry
handle over changing the process default in parallel tests.

## Driver Conformance

If you [write a custom driver](https://arkoren.dev/docs/guides/drivers/writing-a-driver.md), use
`featuretest` from `arkoren.dev/core/service/feature/featuretest` to exercise the
portable contract. Its harness owns a ready driver and a way to install and
remove disposable backend fixtures; production `feature.Driver` stays read-only.
Never point a fixture harness at production flags.

Implement `featuretest.Harness` with `Driver() feature.Driver`,
`Set(*testing.T, featuretest.Flag)`, and `Delete(*testing.T, string)`. With an
existing factory `newHarness func(*testing.T) featuretest.Harness`, invoke it
from your test:

```go
featuretest.Run(t, newHarness)
```

Each subtest gets a fresh harness and unique keys, and subtests run in parallel.
The factory must register cleanup for any acquired resources. A harness returns
the same ready driver throughout its lifetime. Its fixture operations must:

- Replace the whole flag on `Set`, not merge with previous state. Install the
  value and overrides before applying `Disabled`.
- Accept `bool`, `string`, `int64`, and `float64` fixtures. A fixture has a default
  value or at least one override; it never has neither.
- Allow changes after evaluation and make `Delete` tolerate absent keys,
  including repeated cleanup deletion.
- Match an override independently of the subject's attributes. An unmatched
  default value reports static, default, or unknown; a matching override reports
  `ReasonTargetingMatch`, and a disabled flag reports `ReasonDisabled`.

`featuretest.Flag` is a raw backend fixture, not the application's typed
`feature.Flag[T]`. It separates stored and requested types so the suite can test
mismatches. A harness adapting typed memory setup must switch on fixture types
and construct corresponding typed definitions.

For asynchronously refreshed fixtures, give the suite an observation budget.
With standard-library `time` imported:

```go
featuretest.Run(t, newHarness, featuretest.WithPropagation(2*time.Second))
```

Reads after fixture installation or deletion retry every 100 milliseconds until
the expected state appears or that budget expires. This does not configure
provider retries. `WithoutUpdates()` skips cases that update or delete a served
flag when the backend cannot expose those changes in the test; report that
coverage limit rather than treating it as verified refresh behavior.

Raw invalid-kind checks run immediately without propagation retries. They cover
missing, default-only, disabled, matched, and unmatched fixtures, and require
cancellation to win over kind rejection. Because the harness supplies a ready
driver, separately test invalid kinds before startup and before provider calls
with a call recorder. Variants are not checked by the suite. Run remote-backed
conformance as an isolated [integration test](https://arkoren.dev/docs/guides/testing/integration-tests.md),
not as a prerequisite for deterministic application tests.

See the [`feature` API](https://pkg.go.dev/arkoren.dev/core/service/feature),
[`memoryfeature` API](https://pkg.go.dev/arkoren.dev/core/service/feature/memoryfeature),
and [`featuretest` API](https://pkg.go.dev/arkoren.dev/core/service/feature/featuretest).

---

Source: https://arkoren.dev/docs/guides/events/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/events/overview)

# Overview

When a report becomes available, several parts of your application might want to
observe it without the report service importing each observer. An **event bus**
connects a named notification to local listener functions. Arkoren calls those
functions directly; it does not store notifications or send them to another process.

Use the bus for short, best-effort observations. If work must survive a crash,
record a durable handoff where the application owns the change; adding a queue
call inside a listener does not protect the time before that call succeeds.

Focused snippets leave routine error handling to your application. Examples about
failure decisions retain the relevant checks.

## Choosing a Bus

With an existing application, use `bus := app.Events()`. It returns the same
nonnil `*event.Bus` throughout that application's lifetime. Pass this bus to
components that need it rather than passing the whole application.

For an isolated component or test, construct a separate bus:

```go
import "arkoren.dev/core/event"

bus := event.New()
```

Each bus has independent registrations. It needs no startup or shutdown and has
no resources to close. To choose where best-effort listener failures are logged,
use a construction callback; `slog` comes from `log/slog`:

```go
bus := event.New(func(events *event.Builder) {
	events.Logger(slog.Default())
})
```

This is an alternative construction. Settings are copied when construction ends;
retaining the builder does not reconfigure the bus. The logger remains shared.
Callbacks run once in order, later logger settings win, and nil selects
`slog.Default()` when a failure is logged. A supplied nil callback or callback
panic propagates. The application's own bus uses its application logger.

## Defining Topics

A **topic** associates an event name with its payload type. Define one topic per
name in the package that owns the fact. A report notification needs its stable
identifier, not its contents or a download credential:

```go
type ReportReadyEvent struct {
	ReportID string
}

var ReportReady = event.NewTopic[ReportReadyEvent]("reports.ready")
```

Typed calls check the payload at compile time, but routing uses the name. Topics
with the same name share listeners even if constructed separately. Give each
name exactly one payload type: a typed listener silently skips a nonnil payload
of another type.

Names contain dot-separated tokens of 1–64 lowercase letters, digits, or
hyphens, with no leading or trailing hyphen in a token and at most 255 bytes
overall. `reports.ready` is valid; `Reports.Ready` is not. Construction does not
validate the name. Registration and dispatch return `event.ErrInvalidName`.

If a notification needs no data, define a **signal** instead:

```go
var ReportsCleared = event.NewSignal("reports.cleared")
```

Publish it with `bus.Signal(ctx, ReportsCleared)`; listeners take `event.Void`.
These signals are unrelated to [workflow signals](https://arkoren.dev/docs/guides/workflows/signals-and-waiting.md).

## Registering Observers

Register before the operation can publish. With `ctx` from `context` and `logger`
an existing `*slog.Logger`, an observer can record only the report identifier:

```go
unsubscribe, err := bus.Listen(ReportReady, func(ctx context.Context, notice ReportReadyEvent) error {
	logger.InfoContext(ctx, "report ready", "report_id", notice.ReportID)

	return nil
})
```

Registration returns an error and a removal function. Check the error during
wiring before retaining or calling `unsubscribe`. Keep the registration for its
owner's lifetime; [Writing Listeners](https://arkoren.dev/docs/guides/events/listeners.md#removing-listeners) explains
request, test, and composition ownership. Registering after publication does not
replay an earlier event.

## Publishing Events

After the report is available, announce that fact from the application layer
that knows it happened:

```go
bus.Emit(ctx, ReportReady, ReportReadyEvent{ReportID: "report-2026-11"})
```

`Emit` returns no result. A listener error or panic is logged with the event name,
delivery stops, and later listeners are skipped. Panics include a diagnostic
stack in that log. The operation's result is unchanged. A nil bus makes `Emit`
and `Signal` no-ops.

If a callback must be allowed to fail the operation, use `Dispatch` and handle
its error at the boundary that can still reject or roll back the change:

```go
err := bus.Dispatch(ctx, ReportReady, ReportReadyEvent{ReportID: "report-2026-11"})
```

`Dispatch` returns the first listener error and does not recover listener panics.
It does not undo work that already happened. A direct function call is often
clearer for a mandatory dependency; [Event Auditing](https://arkoren.dev/docs/guides/events/auditing.md) shows where
transactional dispatch can enforce a required record.

## Delivery Boundaries

Both publishing methods run listeners synchronously on the publisher's goroutine.
Exact-name listeners run first in registration order, then matching pattern
listeners in their registration order. A slow listener delays the publisher;
there is no automatic retry, buffering, or independent execution.

Listeners receive the supplied context unchanged, including cancellation,
deadlines, and values. With a database transaction's callback context, compatible
database calls in a listener join that transaction. They do not observe a durable
commit yet. Use [after-commit hooks](https://arkoren.dev/docs/guides/database/transactions.md#after-commit)
for best-effort effects that must wait for commit, or persist pending work in the
transaction when losing it is unacceptable. Never keep using a completed
transaction context from a background goroutine.

The bus supports concurrent registration and publication. Different publishers
can call the same listener concurrently, so the listener owns synchronization
for its state. Registration changes affect later snapshots; removing a listener
does not cancel a callback already selected for delivery.

Use [queues](https://arkoren.dev/docs/guides/queues/overview.md) for worker delivery and
[broadcasting](https://arkoren.dev/docs/guides/broadcasting/overview.md) for messaging between processes.
Neither is supplied by this bus. Arkoren components publish on the bus only
when configured to do so, apart from the application's own lifecycle events;
[Built-in Events](https://arkoren.dev/docs/guides/events/built-in-events.md) is the discovery catalog.

For all functions and options, see the [`event` API reference](https://pkg.go.dev/arkoren.dev/core/event).

---

Source: https://arkoren.dev/docs/guides/events/listeners.md
[HTML documentation](https://arkoren.dev/docs/guides/events/listeners)

# Writing Listeners

A listener lets your application observe an operation without coupling the
publisher to its logging or monitoring code. It runs inside the publishing call,
not on a worker. Start with the [shared bus](https://arkoren.dev/docs/guides/events/overview.md#choosing-a-bus), register
before publication can begin, and keep the callback short.

## Typed Listeners

To observe uploaded object sizes, listen to `storage.ObjectWritten`. Import
`context`, `arkoren.dev/core/service/storage`, and use your existing bus and
`*slog.Logger`:

```go
unsubscribe, err := bus.Listen(storage.ObjectWritten,
	func(ctx context.Context, notice storage.ObjectWrittenEvent) error {
		logger.DebugContext(ctx, "object stored", "bytes", notice.Object.Size)

		return nil
	})
```

The callback receives `storage.ObjectWrittenEvent` without a type assertion.
`Listen` returns `event.ErrInvalidName` for an invalid topic name and
`event.ErrNilFunc` for a nil callback. Check registration errors during wiring;
do not call the returned removal function after failed registration.

Topics route by name, so a typed listener skips a nonnil payload with the wrong
type. Reuse the publisher's topic variable instead of redeclaring its name.
For a signal such as `application.Started`, the callback's second parameter is
`event.Void`; import those packages from `arkoren.dev/core/application` and
`arkoren.dev/core/event`.

## Name Matching

Use an exact string only when the name comes from configuration or another
boundary where the typed topic is unavailable. `ListenEvent` receives `any`;
inspect a payload's type before reading its fields. A name-only observer needs
no assertion:

```go
unsubscribe, err := bus.ListenEvent("storage.object.written",
	func(ctx context.Context, _ any) error {
		logger.DebugContext(ctx, "storage write observed")

		return nil
	})
```

It has the same name and nil-function validation as typed registration.
Prefer the typed form when your callback needs payload fields.

### Pattern Listeners

To count or log several related topics, use `ListenRegex` with `regexp` from the
standard library. This observer selects storage topics and forwards only names:

```go
unsubscribe, err := bus.ListenRegex(regexp.MustCompile(`^storage\.`),
	func(ctx context.Context, name string, _ any) error {
		logger.DebugContext(ctx, "storage observation", "event", name)

		return nil
	})
```

Patterns use `MatchString`, not whole-name matching. Without `^`, `storage` also
matches `app.storage-check`; add `$` when the end matters. A nil pattern or
callback returns `event.ErrNilFunc`.

Do not log whole payloads merely because a pattern matches. Some built-in events
carry raw errors, identifiers, or panic stacks. Select approved fields explicitly;
payload omissions do not redact the caller's context or returned error chains.

## Failure Ordering

Exact-name callbacks run in registration order before all matching pattern
callbacks, which also run in registration order. The first error stops delivery.
With `Emit`, the bus logs it and returns to the publisher; with `Dispatch`, the
error reaches the publisher. A panic similarly stops delivery: `Emit` recovers
and logs it, while `Dispatch` lets it propagate.

Returning an error is therefore not an isolated failure report: it prevents later
observers from receiving that event. If an optional observer can handle its own
failure, return nil afterward. Avoid retry loops or remote requests in that
callback, since the publisher waits for them. Keep dependent steps together
rather than relying on independently registered observers for correctness.

## Removing Listeners

Every successful registration returns an `event.Unsubscribe`. Call it when the
registration's owning scope ends. For a short-lived test or operation, use your
existing typed `handler` and topic:

```go
unsubscribe, err := bus.Listen(storage.ObjectWritten, handler)
if err != nil {
	return err
}
defer unsubscribe()
```

Repeated removal is safe and removes only that registration, even if the same
function was registered more than once. Delivery uses a snapshot: removal does
not stop a callback already selected, and new registrations do not join the
current snapshot.

For a composition-owned listener, give removal to the composer rather than
deferring it until the composition function returns:

```go
composer.MustDefer(bus.Listen(storage.ObjectWritten, handler))
```

This checks registration and removes the listener on rollback or shutdown, as
described in [subscription ownership](https://arkoren.dev/docs/guides/application/composition.md#own-subscriptions).
Process-owned registrations can remain for their bus's lifetime. Repeated
registration in a request handler without removal accumulates callbacks and
repeats each observation.

## Shared Execution

The same callback can run concurrently when different goroutines publish. A
listener that changes shared state must use its own synchronization. Payload
values are not deeply copied: copy slices, maps, or pointed-to data before
mutating or retaining them when their publisher's contract requires it.

Callbacks receive the publisher's context, including its deadline, cancellation,
and any transaction scope. Respect cancellation, but do not assume that passing
a context forcibly interrupts arbitrary callback code. Do not wait for a lock or
transaction the publisher is holding; the publisher cannot proceed until you return.

A listener using the callback context of a compatible database transaction joins
that transaction. Its writes commit or roll back with the caller's work, but
`Emit` cannot enforce rollback if that listener fails. Required records need
[transactional auditing](https://arkoren.dev/docs/guides/events/auditing.md#required-records). The committed-transaction
event itself runs after commit, outside the finished transaction.

For an external effect that must wait for commit, use
[after-commit hooks](https://arkoren.dev/docs/guides/database/transactions.md#after-commit). These hooks are
not durable. Queueing there still leaves a crash window before enqueueing;
persist pending work in the transaction if loss is unacceptable. Never retain
the transaction's callback context for use after it ends.

The bus holds no registration lock while callbacks run, so a listener can publish
or change registrations. Avoid publishing a topic recursively from its own
listener: that recurses synchronously rather than scheduling later work.

For all functions and options, see the [`event` API reference](https://pkg.go.dev/arkoren.dev/core/event).

---

Source: https://arkoren.dev/docs/guides/events/built-in-events.md
[HTML documentation](https://arkoren.dev/docs/guides/events/built-in-events)

# Built-in Events

Use this catalog to find the notification for an operation you want to observe.
Topic variables belong to the named package; payload names below are relative
to that package. A signal has no data and its listener takes `event.Void`.
[Writing Listeners](https://arkoren.dev/docs/guides/events/listeners.md) covers registration and ownership.

## Publishing Policies

Apart from application lifecycle and health events, components publish nothing
without an explicitly configured bus. Configure `Events` when constructing the
component. With `mail` imported from `arkoren.dev/core/service/mail`, an existing
mail driver, and the shared `bus`:

```go
mailer := mail.New(driver, func(mailer *mail.Builder) {
	mailer.Events(bus)
})
```

The setting enables notifications, not subscriptions. Components use best-effort
`Emit`: callbacks run synchronously, and their errors or panics are logged without
replacing the operation result. Failure stops delivery to later listeners. Slow
listeners delay the operation. Events stay in the publishing process, are not
retried, and can be lost on a crash—even if a listener would enqueue work.
Required records belong at an [application-owned durable boundary](https://arkoren.dev/docs/guides/events/auditing.md).

Payload omissions are not universal redaction. Some events retain raw errors,
identifiers, or diagnostic stacks; contexts retain caller values. Select safe
attributes instead of dumping a payload, context, or returned error chain.

## Application

The application always publishes on `app.Events()`. Register before `Start` or
during synchronous composition; a startup callback is too late for `Starting`.

| Topic | Name | Observation |
| --- | --- | --- |
| `application.Starting` | `app.starting` | Signal after composition and telemetry startup, before user startup callbacks. |
| `application.Started` | `app.started` | Signal after all runners launch; readiness can still be pending. |
| `application.Stopping` | `app.stopping` | Signal before runner cancellation and cleanup. |
| `application.Stopped` | `app.stopped` | Signal after runner waiting and cleanup, before telemetry finalization. |
| `application.Failed` | `app.failed` | `FailedEvent{Err}` for the original startup or runner failure. |
| `application.Ready` | `app.ready` | Signal for a successful first readiness pass or transition to ready. |
| `application.Unready` | `app.unready` | `UnreadyEvent{Failing}` for a failed first pass or transition to unready. |
| `application.CheckFailed` | `app.check.failed` | `CheckEvent{Name, Err}` for an initially failing check or transition from passing. |
| `application.CheckRecovered` | `app.check.recovered` | `CheckEvent{Name, Err: nil}` for a check recovering from failure. |

`Stopped` does not mean cleanup succeeded: inspect the lifecycle method's returned
error. The application is still stopping while that callback runs. `Failed`
excludes later cleanup and telemetry-finalization errors. Composition or telemetry
startup failure can publish `Failed` without `Starting`. `Failed` and `Stopped`
use contexts detached from cancellation.

`Failing` lists check names in registration order; do not mutate the shared slice.
Check errors and startup errors are raw diagnostics, not safe external text.
See [Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md) and
[Health and Readiness](https://arkoren.dev/docs/guides/application/health-and-readiness.md) for operating policy.

## HTTP

[HTTP Events](https://arkoren.dev/docs/guides/events/http-events.md) owns the exact publication conditions, payloads,
and privacy limits for server lifecycle, recovered panics, outgoing requests,
CSRF/rate-limit/content-type rejections, sessions, signed links, and webhook
receipts. Enable each component separately; setting a server's bus does not
implicitly enable every middleware's events.

## Authentication and Authorization

These notifications observe refusal, not successful identity establishment:

| Topic | Name | Enabling Policy | Observation |
| --- | --- | --- | --- |
| `auth.Rejected` | `http.auth.rejected` | `auth.Builder.Events` on `Require` | `RejectedEvent{Method}` when an anonymous request is rejected. |
| `gate.Denied` | `http.gate.denied` | `gate.Builder.Events` on `gate.New` | `DeniedEvent{Method, Rule}` when a rule returns false. |

`Rule` is the zero-based rule index in registration order. `Method` is empty
for `Gate.Authorize`, which has no request. Neither payload contains a path,
user, or cause. Rule errors, missing users, allowed calls, and the free
`gate.Authorize`/`gate.Require` functions publish no gate event. Identity
login/logout methods also have no bus and publish nothing themselves.

Each refusal publishes, even when a custom error response changes the default
status. Clients control the frequency: keep callbacks cheap and place
[rate limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) before these checks. Methods
are client-controlled too; keep secrets out of custom method names. See
[Authentication](https://arkoren.dev/docs/guides/authentication/overview.md) and
[denial monitoring](https://arkoren.dev/docs/guides/authorization/denials.md#denial-events).

### OAuth Outcomes

The client in `arkoren.dev/driver/oauth/oauthauth` uses `ClientBuilder.Events`:

| Topic | Name | Observation |
| --- | --- | --- |
| `oauthauth.LoginSucceeded` | `oauth.login.succeeded` | `LoginSucceededEvent{Subject, Issuer}` after local session login. |
| `oauthauth.LoginFailed` | `oauth.login.failed` | `LoginFailedEvent{ProviderError, Err}` for a rejected callback. |
| `oauthauth.LoggedOut` | `oauth.logged-out` | Signal after removing the user from the local session. |
| `oauthauth.TokenRejected` | `oauth.token.rejected` | `TokenRejectedEvent{Err}` for bearer rejection. |

Callback rejection covers invalid/replayed login cookies, provider errors, missing
codes, exchange/token-check failures, and local login failures. Configuration and
missing-session exits before cookie consumption publish nothing. Login/logout
events describe request-local mutation, not durable session persistence or provider logout.

`LoginFailedEvent.Err` is only `ErrInvalidCallback`, `ErrProviderUnavailable`, or
an opaque local-login classification with no exported sentinel or cause. Match
the two exported classes with `errors.Is`; do not classify opaque errors by text.
`TokenRejectedEvent.Err` is a fixed opaque error with no exported sentinel or
native cause. Count that outcome rather than trying to unwrap diagnostics.

`ProviderError` admits only `invalid_request`, `unauthorized_client`, `access_denied`,
`unsupported_response_type`, `invalid_scope`, `server_error`, or
`temporarily_unavailable`; absent or unknown hints become empty. It remains an
untrusted callback hint, not an authenticated diagnosis. `Callback` and `Verify`
return errors with original causes separately; those causes can expose credentials
or claims. `Subject` and `Issuer` can identify a person even without email or tokens.
See [OAuth Login](https://arkoren.dev/docs/guides/authentication/oauth.md#events) for configuration.

## Database

With `database.Builder.Events`, outermost transactions publish
`database.TransactionCommitted` (`database.transaction.committed`) or
`database.TransactionRolledBack` (`database.transaction.rolled-back`). Both carry
`TransactionEvent{System, Namespace, Err}`. Nested joined transactions publish
nothing independently.

A confirmed commit publishes nil `Err` before after-commit hooks; later cleanup
or hook errors can still return `database.ErrAfterCommit`. Before confirmed
commit, body/context/commit failures publish rollback. A body error bearing
`ErrAfterCommit` still requests rollback. Panics propagate without a transaction
event. Committed-event listeners run outside the finished transaction.
See [Transactions](https://arkoren.dev/docs/guides/database/transactions.md) and
[Query Monitoring](https://arkoren.dev/docs/guides/database/query-monitoring.md).

With `migrate.Builder.Events`, each handled migration carries
`MigrationEvent{ID, Group, Batch, Rollback, Duration, Err}`:

| Topic | Name | Observation |
| --- | --- | --- |
| `migrate.Applied` | `database.migration.applied` | Applied and recorded one migration. |
| `migrate.RolledBack` | `database.migration.rolled-back` | Reverted one migration and removed its record. |
| `migrate.Failed` | `database.migration.failed` | Work did not complete; the run stops. |

`Group` is empty for application migrations. A confirmed commit followed by
cleanup failure still publishes the completed outcome with nil payload `Err`;
the returned error stops the run. Raw database errors can contain sensitive
data. Migration procedures stay in [Migrations](https://arkoren.dev/docs/guides/migrations/overview.md).

## Feature Flags

`feature.Builder.Events` enables `feature.EvaluationFailed`
(`feature.evaluation.failed`), carrying `EvaluationFailedEvent{Key, Err}` when
driver evaluation fails and the caller receives its default plus the error.
The error retains the driver's cause, wrapped with the flag key. Subject,
attributes, and values are omitted.

Successful/off evaluations, invalid definitions or subjects, and facade type
conversion failures publish nothing. Every driver failure publishes, including
hot-path misses. See [Feature Flags](https://arkoren.dev/docs/guides/feature-flags/overview.md#events).

## Search

`search.Builder.Events` enables administrative `Sync`/`Drop` observations:
`search.Synced`, `search.Dropped`, and `search.Failed`. Queries and document
writes publish nothing. [Search outcomes](https://arkoren.dev/docs/guides/events/service-operations.md#search) give
their canonical names, bounded payload, and partial-effect limits.

## File Storage

With `storage.Builder.Events`, successful facade changes publish:

| Topic | Name | Observation |
| --- | --- | --- |
| `storage.ObjectWritten` | `storage.object.written` | `ObjectWrittenEvent{Store, Object}` after `Put` or a helper built on it. |
| `storage.ObjectDeleted` | `storage.object.deleted` | `ObjectDeletedEvent{Store, Key}` after successful idempotent deletion. |

`Store` is the configured name or `default`. Deleting an absent object also
succeeds, so the notification does not prove prior existence. Reads, listings,
failed writes, and direct signed-URL uploads publish nothing. Object metadata
and keys can identify resources. See
[storage monitoring](https://arkoren.dev/docs/guides/file-storage/events-and-telemetry.md#object-events).

## Secrets

`secret.Builder.Events` enables `secret.MutationSucceeded` and
`secret.MutationFailed` for facade `Put`, `Update`, and `Delete` results.
[Secret outcomes](https://arkoren.dev/docs/guides/events/service-operations.md#secrets) define their payload and names.
Reads and external/native writes publish nothing; these are not rotation or rollout records.

## Queues

Enable `queue.ConsumerBuilder.Events` on the consumer. Payloads share
`ID`, `Name`, `Queue`, and the driver's delivery count `Attempts` (zero if unavailable):

| Topic | Name | Additional Fields / Observation |
| --- | --- | --- |
| `queue.JobFailed` | `queue.job.failed` | `JobFailedEvent.Err`; accepted terminal failure settlement. |
| `queue.JobRetried` | `queue.job.retried` | `JobRetriedEvent{Delay, Err}`; accepted handoff for another attempt. |
| `queue.JobRejected` | `queue.job.rejected` | `JobRejectedEvent.Err`; accepted deliberate rejection, instead of `JobFailed`. |
| `queue.JobTimedOut` | `queue.job.timed-out` | `JobTimedOutEvent.Timeout`; attempt returned an error after its deadline, before settlement. |

Terminal failure covers permanent/exhausted errors, undecodable payloads, and
unknown names after their attempt limit. If settlement fails, no settlement
notification fires; the error returns through consumption and the broker outcome
remains unconfirmed. If the driver's `Consume` returns an error, `Consumer.Run`
cancels its other queue consumers, waits for them to stop, and returns the first
error. Accepted settlement does not prove durable failed-job storage.

`Delay` is the requested retry delay, not an exact execution time. Exhausted
retries, shutdown release, and driver-controlled unknown-name redelivery do not
publish `JobRetried`. A timeout can be followed by retry, failure, or rejection
for the same delivery; settlement failure can prevent that follow-up. Success
after the deadline publishes no timeout event. Errors retain diagnostics;
payload bytes are omitted. See [Consuming Jobs](https://arkoren.dev/docs/guides/queues/consuming-jobs.md).

## Broadcasting

`broadcast.ConsumerBuilder.Events` enables `broadcast.ListenerFailed`
(`broadcast.listener.failed`), carrying `ListenerFailedEvent{ID, Name, Err}`
once per listener that returns an error, fails decoding, or panics. Several
listeners can fail one delivery and publish in listener order. `ID` can be empty.
Success publishes nothing; transport settlement is not the observation. Payload
and metadata are omitted, but errors retain causes. See
[broadcast delivery](https://arkoren.dev/docs/guides/broadcasting/message-delivery.md#observing-failures).

## Mail

With `mail.Builder.Events`, `mail.Sent` (`mail.sent`) carries
`SentEvent{Receipt, Recipients, Attachments}` after driver acceptance;
`mail.Failed` (`mail.failed`) carries `FailedEvent{ErrorClass, Recipients, Attachments}`
after returned validation or driver failure. Counts omit addresses and contents;
the receipt is a provider submission identifier, not a delivery receipt.

Failure classes are `context.Canceled`, `context.DeadlineExceeded`,
`mail.ErrInvalidMessage`, then `error`, in that priority order. No native cause
or diagnostic text enters `FailedEvent`; `Send` returns the original error
separately. Nil-context rejection and driver panics publish neither outcome.
Acceptance is not recipient delivery, and failure does not rule out remote
acceptance. See [mail monitoring](https://arkoren.dev/docs/guides/mail/events-and-telemetry.md#events).

## Push Notifications

`push.Builder.Events` enables `push.Accepted` and `push.Failed` for returned
submission results. [Push outcomes](https://arkoren.dev/docs/guides/events/service-operations.md#push-notifications)
give their names and payload; neither confirms device delivery or display.

## Payments

`payment.Builder.Events` enables `payment.OperationCompleted` and
`payment.OperationFailed` for driver-returned facade calls.
[Payment outcomes](https://arkoren.dev/docs/guides/events/service-operations.md#payments) give their names and bounded
payload; neither is a settlement or fulfillment event.

## Scheduled Tasks

Enable `schedule.Builder.Events` for run outcomes and deferred-task requests:

| Topic | Name | Observation |
| --- | --- | --- |
| `schedule.RunFailed` | `schedule.run.failed` | `RunFailedEvent{Name, Task, Duration, Err}` for a returned run error other than `ErrSkipped`. |
| `schedule.RunSkipped` | `schedule.run.skipped` | `RunSkippedEvent{Name, Task, Duration}` when execution returns `ErrSkipped`. |
| `schedule.TaskDeferred` | `schedule.task.deferred` | `TaskDeferredEvent{Name, ID, At}` after successful `At`/`After` submission. |
| `schedule.TaskCanceled` | `schedule.task.canceled` | `TaskCanceledEvent{ID}` after successful cancellation request. |

Driver-started runs and `RunNow` both publish run outcomes. `Task` distinguishes
deferred tasks from entries; task payloads are omitted. A skip is not a failure,
and a driver's overlap guard preventing invocation is not an execution skip.
Failure errors can include recovered panics and timeout errors; returning nil
after a deadline still succeeds. Driver retry policy is separate.

`TaskDeferred` reports acceptance, not task completion. Canceling an unknown or
finished ID can succeed and publish too; it does not prove removal or that a
running handler stopped. See [failure events](https://arkoren.dev/docs/guides/scheduled-tasks/error-handling.md#failure-events)
and [deferred tasks](https://arkoren.dev/docs/guides/scheduled-tasks/deferred-tasks.md).

## Workflows

Enable `workflow.ClientBuilder.Events` on the client:

| Topic | Name | Observation |
| --- | --- | --- |
| `workflow.RunStarted` | `workflow.run.started` | `RunEvent{Workflow, RunID}` after accepted start. |
| `workflow.RunSignaled` | `workflow.run.signaled` | `RunSignaledEvent{Workflow, RunID, Signal}` after accepted signal delivery. |
| `workflow.RunCanceled` | `workflow.run.canceled` | `RunEvent{Workflow, RunID}` after accepted cancellation. |

Failures, including already-started runs, publish nothing. Canceling a finished
run succeeds and publishes. These are local client actions, not subscriptions to
engine completion or proof a signal was consumed. Input, output, signal values,
and metadata are omitted; run IDs remain potentially identifying data. See
[workflow observations](https://arkoren.dev/docs/guides/workflows/running-workflows.md#events-and-telemetry).

## Language Models

Use `llm.ModelBuilder.Events` for model/agent/tool events and
`llm.EmbedderBuilder.Events` for embeddings:

| Topics | Names | Observation |
| --- | --- | --- |
| `llm.ResponseGenerated`, `llm.ResponseFailed` | `llm.response.generated`, `llm.response.failed` | Standalone `Generate` or exhausted `Stream`. |
| `llm.EmbeddingsGenerated`, `llm.EmbeddingsFailed` | `llm.embeddings.generated`, `llm.embeddings.failed` | Returned `Embed` outcome. |
| `llm.AgentCompleted`, `llm.AgentFailed` | `llm.agent.completed`, `llm.agent.failed` | Finished `Run` or `RunStream`. |
| `llm.ToolInvoked`, `llm.ToolFailed` | `llm.tool.invoked`, `llm.tool.failed` | Tool outcome, including argument or availability rejection before a handler runs. |

Each topic has its correspondingly named `…Event` type. Payloads describe
model/agent/tool names, usage or counts where relevant, and duration—not prompts,
replies, arguments, vectors, or raw errors. `Steps` counts model turns, not tool
calls. Agent turns do not also publish standalone response events. Abandoning
a stream, even by breaking at standalone `Done`, suppresses its final outcome.

Response, embeddings, and agent failure payloads carry bounded `ErrorType`:
context cancellation/deadline names first, then known `llm` sentinel names,
otherwise `unknown`. Policy failures use `llm.ErrToolAvailability` ahead of
`ErrPermanent` unless a context error matches. Tool failure carries no error
field. Original returned/yielded errors retain diagnostic causes and need a
separate privacy review. See [observing calls](https://arkoren.dev/docs/guides/llm/overview.md#observing-calls).

For registration and dispatch, see the [`event` API reference](https://pkg.go.dev/arkoren.dev/core/event).

---

Source: https://arkoren.dev/docs/guides/events/http-events.md
[HTML documentation](https://arkoren.dev/docs/guides/events/http-events)

# HTTP Events

HTTP notifications help you distinguish a refused request, an exhausted outgoing
call, and a failure during response-time persistence. They observe local decisions;
they do not prove that a client received a response. Enable the relevant component
with the policy below and register on the same bus.

These notifications use synchronous, best-effort `Emit`. A slow listener delays
the operation; a listener error or panic is logged and skips later listeners
without changing the result. No bus means no publication. Review payloads and
context values before retaining them: selected schemas omit sensitive fields,
but raw errors and caller context values are not universally redacted.

## Topics

| Topic | Name | Enabling Policy |
| --- | --- | --- |
| `server.Listening` | `http.server.listening` | `server.Builder.Events` |
| `server.ShutdownTimedOut` | `http.server.shutdown.timed-out` | `server.Builder.Events` |
| `router.PanicRecovered` | `http.panic.recovered` | `router.RecoverBuilder.Events` |
| `client.RequestFailed` | `http.client.request.failed` | `client.Builder.Events` |
| `csrf.Blocked` | `http.csrf.blocked` | `csrf.Builder.Events` |
| `ratelimit.Exceeded` | `http.ratelimit.exceeded` | `ratelimit.Builder.Events` |
| `contenttype.Rejected` | `http.contenttype.rejected` | `contenttype.Builder.Events` |
| `session.Expired` | `session.expired` | `session.Builder.Events` |
| `session.Destroyed` | `session.destroyed` | `session.Builder.Events` |
| `session.PersistFailed` | `session.persist.failed` | `session.Builder.Events` |
| `link.Rejected` | `link.rejected` | `link.Builder.Events` on the signer |
| `link.Consumed` | `link.consumed` | `link.Builder.Events` on the signer |
| `webhook.Rejected` | `http.webhook.rejected` | `webhook.Builder.Events` |
| `webhook.Failed` | `http.webhook.failed` | `webhook.Builder.Events` |
| `webhook.Ignored` | `http.webhook.ignored` | `webhook.Builder.Events` |

Authentication, gate, and OAuth notifications have their own
[catalog entries](https://arkoren.dev/docs/guides/events/built-in-events.md#authentication-and-authorization).

## Server Lifecycle

`ListeningEvent{Addr}` fires when `Start` binds its listener. With port zero,
`Addr` includes the chosen port. It is a bound address, not a public URL or proof
that serving or TLS negotiation has begun.

`ShutdownTimedOutEvent{Addr, Timeout}` fires only when `Serve` exceeds its native
HTTP drain budget and force-closes tracked connections. `Timeout` is the
configured budget, not total elapsed time. A clean drain, another shutdown
failure, and `Stop` publish nothing. Handler work can still remain, and hijacked
connections are outside native draining: their owners must close them and await
associated work. See [server shutdown](https://arkoren.dev/docs/guides/http-networking/server.md#shutting-down).

## Recovered Panics

`PanicRecoveredEvent{Method, Pattern, Err, Stack}` fires when `Recover` converts
a handler panic into an error. `Pattern` is the registered route pattern,
including its method when present, or empty if no route matched. It is not the
expanded path. `http.ErrAbortHandler` is re-raised and publishes nothing.

`Err`, its causes, and the goroutine `Stack` are sensitive diagnostics. A monitoring
listener can avoid them and select only registered route metadata. With `context`
and `arkoren.dev/core/http/router` imported, and your existing bus and logger:

```go
_, err := bus.Listen(router.PanicRecovered,
	func(ctx context.Context, notice router.PanicRecoveredEvent) error {
		logger.ErrorContext(ctx, "handler panic", "route", notice.Pattern)

		return nil
	})
```

Keep secrets out of route literals too. Send full diagnostics only to approved,
restricted sinks, never directly to clients. The event stack is detached from
the returned error's diagnostic storage, and that error's `Stack()` accessor
returns an owned copy. Listeners still share the event's slice: copy before
mutation. Ownership copies are not redaction.

The event does not promise a replacement response after bytes are committed.
See [panic handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md#panics),
[response commitment](https://arkoren.dev/docs/guides/http-fundamentals/responses.md#response-commitment), and
[streaming responses](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md).

## Outgoing Requests

`RequestFailedEvent{Method, Host, Attempts, StatusCode, Err}` describes the final
outcome of one transport round trip. It fires for a returned transport error,
or a response `RetryIf` would retry after attempts run out or `Retry-After`
exceeds the allowed delay. Arbitrary 4xx/5xx responses do not automatically qualify.

`Attempts` includes the initial attempt; redirects have separate round trips.
`Host` is the request URL's hostname without port, path, or query. Exactly one
of nonnil `Err` and nonzero `StatusCode` describes the outcome. Native transport
errors retain causes and can contain sensitive text.

If the parent request context is done, publication is suppressed. A per-attempt
timeout can still publish while the parent context remains active. Later response
body-read failures are not transport outcome events. Retry and body ownership
belong in [HTTP Client](https://arkoren.dev/docs/guides/http-networking/http-client.md).

## Request Rejections

`csrf.BlockedEvent{Method, Path, Origin, Err}` observes rejection by the configured
cross-origin policy. `Path` is the request path without query, not a route template.
`Origin` is the untrusted header, empty when absent; `Err` is the native diagnostic.
Paths can contain identifiers, and an origin is not an authenticated identity.
See [CSRF Protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md).

`ratelimit.ExceededEvent{Key, Method, Path, Limit, Window}` describes a denied
request. `Key` is the client part of the counter key—client IP by default—not
the internal namespace/window suffix. `Path` is the decoded request path without
query. `Limit` and `Window` contain normalized positive policy values; `Window`
is the sliding-window duration, not total counter retention.

A cache marker normally limits publication to the first denial per key and
fixed window. If marker insertion fails, the limiter still publishes and later
denials can duplicate it. This is best-effort deduplication, not exactly-once
notification. The marker failure does not allow the denied request. Treat keys
and paths as potentially identifying data; see
[Rate Limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md).

`contenttype.RejectedEvent{Method, Path, MediaType}` fires on every content-type
rejection, including with a custom response status. `MediaType` is the lower-case
base type without parameters; it is empty for missing or unparseable base types.
Malformed parameters can still leave a parsed base type. Paths and media types
are client-controlled. Keep these callbacks cheap and rate-limit hostile request
traffic before it reaches them. See [Content Type](https://arkoren.dev/docs/guides/http-fundamentals/content-type.md).

## Session Changes

`ExpiredEvent{CreatedAt, ExpiresAt}` fires when middleware encounters an expired
session, including maximum-lifetime expiry, and replaces it. This happens on a
request after expiry, not on a timer. No session identifier or data is included.

`DestroyedEvent{Method, Path}` fires after middleware successfully writes the
tombstone for a session ended with `Session.Destroy`, before expiring its cookie.
If retirement fails, it publishes `PersistFailed` instead: the old identifier
can still be usable. Expiry is a different event; regeneration alone publishes
no destruction event. Persistence runs at response-header writing and is skipped
for server-error responses.

`PersistFailedEvent{Operation, Method, Path}` describes the failed persistence step:
`session.PersistSave` (`save`) writes the record, `session.PersistVerify` (`verify`)
checks the original record is live, and `session.PersistRetire` (`retire`) writes
a tombstone. The payload omits the storage error. `session.Builder.ErrorHandler`
still receives that diagnostic with its cause; sanitize it separately.

For destruction and persistence failure, `Path` is the registered route pattern
with its method prefix removed, or empty without a pattern. Expanded parameters
and queries never replace it; host-specific patterns retain their host prefix.
Avoid secrets in route literals.

Destruction uses a persistence context detached from request cancellation, with
the persistence timeout. Failure events use a context detached from cancellation
and deadlines, so an already-exhausted persistence budget does not cancel them.
This does not forcibly bound arbitrary listeners or callbacks. Best-effort
lease cleanup has a separate ten-second deadline and requires backend context
cooperation. Session persistence and cookie behavior belong in
[Sessions](https://arkoren.dev/docs/guides/http-capabilities/sessions.md).

## Signed Links

`link.RejectedEvent{Method}` observes middleware rejection of invalid link form
or signature, including binding failure. Expiry of a valid link publishes no
rejection. The payload contains no path, URL, binding, or error; binding causes
remain in the middleware's returned problem for `errors.Is`/`errors.As`.

`link.ConsumedEvent{Method}` fires only after `Consume`, `ConsumeURL`, or
`ConsumeRequest` records single use. The direct URL operations leave `Method`
empty. Failed consumption, including an already-spent link, publishes nothing.
`Verify`, `Inspect`, and middleware do not consume and publish no consumption event.

Methods are client-controlled, and contexts and returned errors remain outside
payload redaction. See [Secure Links](https://arkoren.dev/docs/guides/http-capabilities/secure-links.md).

## Webhook Receipts

Webhook notifications describe locally selected receipt outcomes:

| Topic | Payload | Outcome |
| --- | --- | --- |
| `webhook.Rejected` | `RejectedEvent{Label, Reason, Status}` | Selected 4xx rejection, including native 413 body-limit handling. |
| `webhook.Failed` | `FailedEvent{Label, Stage}` | Processing failure answered with 503. |
| `webhook.Ignored` | `IgnoredEvent{Label, Reason}` | Authenticated, unbound type acknowledged with 200. |

`RejectedEvent.Reason` is `webhook.RejectionReason`: compare
`RejectionReasonBodyTooLarge` (`body_too_large`), `RejectionReasonUnreadableBody`
(`unreadable_body`), or `RejectionReasonVerify` (`verify`). Interrupted body reads
from cancellation/timeouts are failures, not malformed-body rejections.

`FailedEvent.Stage` is `webhook.FailureStage`: `FailureStageRead` (`read`),
`FailureStageVerify` (`verify`), `FailureStageDecode` (`decode`), or
`FailureStageAccept` (`accept`). Decoder and `Event.Map` errors share decode;
every acceptance error, including `ErrSkip`, is an accept failure. Acceptance
can already have committed work, so this classification does not imply rollback
or safe replay.

`IgnoredEvent.Reason` is `webhook.IgnoreReason`, currently only
`IgnoreReasonUnbound` (`unbound`). Authenticated decoder/map skips and duplicates
publish none of these topics. Empty zero categories remain unspecified (`""`),
not defaults. Compare typed constants; convert explicitly for string-only APIs.

Payloads omit causes, bodies, headers, provider IDs, and event types. `Label`
remains application-controlled and unvalidated; keep it free of credentials and
personal data. Context values are not sanitized. Failure responses are committed
locally, including native 413 before authentication; provider retry behavior is
separate. See [Receiving Webhooks](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md),
[receiver diagnostics](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md#receiver-diagnostics), and
[acknowledgement outcomes](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md#acknowledgement-outcomes).

For declarations, see the API references for
[`server`](https://pkg.go.dev/arkoren.dev/core/http/server),
[`router`](https://pkg.go.dev/arkoren.dev/core/http/router),
[`client`](https://pkg.go.dev/arkoren.dev/core/http/client),
[`csrf`](https://pkg.go.dev/arkoren.dev/core/http/middleware/csrf),
[`ratelimit`](https://pkg.go.dev/arkoren.dev/core/http/middleware/ratelimit),
[`contenttype`](https://pkg.go.dev/arkoren.dev/core/http/middleware/contenttype),
[`session`](https://pkg.go.dev/arkoren.dev/core/http/session),
[`link`](https://pkg.go.dev/arkoren.dev/core/http/link), and
[`webhook`](https://pkg.go.dev/arkoren.dev/core/http/webhook).

---

Source: https://arkoren.dev/docs/guides/events/service-operations.md
[HTML documentation](https://arkoren.dev/docs/guides/events/service-operations)

# Service Events

A successful API call is not necessarily a completed business action. A payment
can remain pending, an accepted push can remain undelivered, and a secret update
can leave consumers using an older credential. These events report the local
operation result at the facade's driver boundary, not those later outcomes.

Enable the service's construction-time `Events` policy and listen on that same
bus. Omitted or nil buses publish nothing. Publication is synchronous and
best effort: listener failures cannot replace returned results, but delay or
stop later observations. Other processes receive nothing automatically.
Do not drive required fulfillment, auditing, or reconciliation solely from these
notifications; use an [application-owned durable record](https://arkoren.dev/docs/guides/events/auditing.md).

## Payments

`payment.Builder.Events` enables two topics, both carrying
`payment.OperationEvent{Operation, ErrorType}`:

| Topic | Name | Driver Result |
| --- | --- | --- |
| `payment.OperationCompleted` | `payment.operation.completed` | Nil error. |
| `payment.OperationFailed` | `payment.operation.failed` | Nonnil error. |

Each validated facade call that reaches the driver and returns publishes one
observation after its telemetry operation ends. `Operation` is the fixed
snake_case facade operation, such as `get_payment` or `refund_payment`.
Validation failures, missing drivers, pre-canceled calls, and provider panics
publish nothing. Cancellation returned after driver dispatch can still publish.

Completion includes reads, pending results, and idempotent replays. It proves
neither authorization nor settlement and must not be treated as fulfillment
eligibility. The payload omits resource IDs, amounts, keys, references, URLs,
handoff secrets, and native errors.

`ErrorType` is empty on completion. On failure it is a payment sentinel name,
`context.Canceled`, `context.DeadlineExceeded`, or `unknown`.
`payment.ErrOutcomeUnknown` outranks all joined causes: the provider might have
applied a mutation. Other context classifications precede other payment sentinels.
The returned error retains original causes separately.

To observe uncertainty without exporting provider diagnostics, use `context` and
`payment` from `arkoren.dev/core/service/payment`, with your existing bus and logger:

```go
_, err := bus.Listen(payment.OperationFailed,
	func(ctx context.Context, notice payment.OperationEvent) error {
		logger.WarnContext(ctx, "payment call failed",
			"operation", notice.Operation,
			"error_type", notice.ErrorType)

		return nil
	})
```

This records the classification, not whether funds moved. Never infer that retry
with a new key is safe. Follow [payment recovery](https://arkoren.dev/docs/guides/payments/recovery.md) for
uncertain outcomes; operation behavior belongs in
[hosted checkout](https://arkoren.dev/docs/guides/payments/hosted-checkout.md#starting-checkout),
[explicit completion](https://arkoren.dev/docs/guides/payments/hosted-checkout.md#explicit-completion),
[refunds](https://arkoren.dev/docs/guides/payments/refunds.md), and [subscriptions](https://arkoren.dev/docs/guides/payments/subscriptions.md).
See the [`payment` API reference](https://pkg.go.dev/arkoren.dev/core/service/payment).

## Push Notifications

`push.Builder.Events` enables `push.Accepted` (`push.submission.accepted`) and
`push.Failed` (`push.submission.failed`). Both carry
`SubmissionEvent{Duration, ErrorClass}` for the returned driver call from `Send`.
Nil error selects acceptance; nonnil error selects failure. Preflight rejection,
missing drivers, and provider panics publish neither.

`Duration` measures driver work, excluding listener execution and telemetry
completion, not device latency. `ErrorClass` is empty on acceptance; otherwise
it is `outcome_unknown`, `canceled`, `deadline_exceeded`, a push sentinel name,
or `error`, in that precedence. Unknown acceptance outranks cancellation because
the provider might already have accepted the submission.

Payloads omit recipients, endpoints, tokens, contents, receipts, and native
errors. Acceptance is not delivery or display; failure is not proof that delivery
cannot occur. Native returned errors remain available to the caller and require
separate redaction. Retry and registration cleanup stay application-owned; see
[Push Notifications](https://arkoren.dev/docs/guides/push/overview.md) and the
[`push` API reference](https://pkg.go.dev/arkoren.dev/core/service/push).

## Search

`search.Builder.Events` enables administrative outcomes only. All three topics
carry `MutationEvent{Operation, Duration, ErrorClass}`:

| Topic | Name | Observation |
| --- | --- | --- |
| `search.Synced` | `search.index.synced` | `Sync` returned nil error. |
| `search.Dropped` | `search.index.dropped` | `Drop` returned nil error. |
| `search.Failed` | `search.index.failed` | Either administrative call returned an error. |

`Operation` is `sync` or `drop`. `Duration` measures driver work, excluding listener
execution. `ErrorClass` is empty on success; otherwise context classes `canceled`
and `deadline_exceeded` precede known search sentinel names, with `error` as the
fallback. Native errors, index names, documents, queries, and document IDs are absent.

Sync can succeed without changes, and dropping an absent index can succeed.
Failure can follow partial changes, particularly across multiple indexes; neither
atomicity nor rollback is implied. `Check`, queries, document writes/deletes,
invalid registrations, and provider panics publish nothing. Deployment procedures
stay in [Deploying Search](https://arkoren.dev/docs/guides/search/deploying.md); see the
[`search` API reference](https://pkg.go.dev/arkoren.dev/core/service/search).

## Secrets

`secret.Builder.Events` enables `secret.MutationSucceeded`
(`secret.mutation.succeeded`) and `secret.MutationFailed`
(`secret.mutation.failed`). Both carry `MutationEvent{Operation, ErrorType}` after
facade `Put`, `Update`, or `Delete` returns from the driver. Nil error selects
success; nonnil error selects failure, including a missing secret.

`Operation` is `put`, `update`, or `delete`. `ErrorType` is empty on success;
otherwise it is `secret.ErrNotFound`, `context.Canceled`,
`context.DeadlineExceeded`, or `other`. Names, versions, values, sizes, and native
diagnostics are omitted. Context cancellation and deadline classifications take
precedence over `secret.ErrNotFound`. Returned driver errors remain unchanged.

Reads, direct driver/native calls, provider-console updates, and other-process
writes are outside these observations. Success does not prove credential rollout,
global rotation, or permanent erasure; failure does not establish that the
provider made no change. See [Managing Secrets](https://arkoren.dev/docs/guides/secrets/managing.md) and the
[`secret` API reference](https://pkg.go.dev/arkoren.dev/core/service/secret).

---

Source: https://arkoren.dev/docs/guides/events/auditing.md
[HTML documentation](https://arkoren.dev/docs/guides/events/auditing)

# Event Auditing

An audit record explains which actor changed which resource and when. Arkoren's
event bus can connect that application fact to a recording function, but Arkoren
does not provide an audit store or make notifications durable. Your application
owns the record format, access policy, retention, and persistence boundary.

Choose the requirement before choosing how to publish. An optional diagnostic
trail can tolerate a lost record and use `Emit`. If every committed change must
have a record, write that record in the same database transaction as the change.
A direct insert is often the clearest dependency; `Dispatch` is an alternative
when your application deliberately expresses that dependency through listeners.

## Defining Facts

For an invoice void, capture the identifiers and occurrence time where the
application authorizes and performs the action. In your invoice package, import
`time` and `arkoren.dev/core/event`:

```go
type VoidedEvent struct {
	InvoiceID string
	ActorID   string
	At        time.Time
}

var Voided = event.NewTopic[VoidedEvent]("invoice.voided")
```

An occurrence time in the payload remains meaningful if processing happens later.
Set `ActorID` from the authenticated, authorized application identity—not an
arbitrary request field. Background work needs an explicit trusted actor too;
it cannot assume the original request's identity remains in a worker context.

Keep secrets and unnecessary personal data out of the payload. Stable identifiers
can still identify people, so restrict audit access and retention. An event name
or typed struct does not establish authorization or sanitize its fields.

## Optional Observations

For an optional local trail, register a callback that selects only approved
fields. Here `invoice` is your package above, `bus` is the shared bus, `context`
is imported, and `logger` is your existing `*slog.Logger`:

```go
_, err := bus.Listen(invoice.Voided, func(ctx context.Context, fact invoice.VoidedEvent) error {
	logger.InfoContext(ctx, "invoice void observed",
		"invoice_id", fact.InvoiceID,
		"actor_id", fact.ActorID,
		"at", fact.At)

	return nil
})
```

After the application establishes that the invoice was voided, publish its fact:

```go
bus.Emit(ctx, invoice.Voided, invoice.VoidedEvent{
	InvoiceID: "invoice-2048",
	ActorID:   actor.ID,
	At:        time.Now(),
})
```

Here `actor.ID` is the already-authorized string identifier. The observer runs
before `Emit` returns. Its error or panic is logged rather than returned, and
later listeners are skipped. A process crash can lose the observation. A log
line also does not prove that an external collector durably stored it.

If publication occurs inside a transaction, the fact is still provisional:
rollback remains possible. Use [after-commit hooks](https://arkoren.dev/docs/guides/database/transactions.md#after-commit)
for optional observations that must not describe rolled-back changes, or use the
required-record pattern below. Do not infer commit merely from the event's name.

## Required Records

A required row must commit or roll back with the invoice change. Assume your
application has an `invoice_audit` table, a configured `*database.Database` named
`db`, and an application-owned `audit.Insert(ctx, fact)` function that inserts
the fact into that table using this database and the supplied context. It must
return any insert failure; it must not use a separate connection or detached context.

Register that insert as the required callback:

```go
_, err := bus.Listen(invoice.Voided, func(ctx context.Context, fact invoice.VoidedEvent) error {
	return audit.Insert(ctx, fact)
})
```

Check registration before accepting writes. Your application-owned
`invoices.Void(ctx, id)` below authorizes and changes the invoice using the same
database; it returns an error if the change cannot be performed. Keep the
failure branches that determine rollback:

```go
err := db.WithTransaction(ctx, func(ctx context.Context) error {
	if err := invoices.Void(ctx, "invoice-2048"); err != nil {
		return err
	}

	return bus.Dispatch(ctx, invoice.Voided, invoice.VoidedEvent{
		InvoiceID: "invoice-2048",
		ActorID:   actor.ID,
		At:        time.Now(),
	})
})
```

The listener receives the transaction's callback context, so its compatible
database insert joins the transaction. A failed insert returns through
`Dispatch` and the transaction body, preventing commit. A listener panic
propagates rather than being swallowed. `Emit` cannot enforce this requirement:
it hides callback errors from the publisher. Nor does `Dispatch` verify that a
listener exists—without the required registration it succeeds without a record.
Keep wiring and tests responsible for that invariant, or call the insert directly.

Do not perform remote calls inside the transaction listener: the publisher waits
while holding transaction resources and locks. Do not use its callback context
after the transaction ends. The database's committed event runs outside the
finished transaction and is too late to add an atomic audit row.

The enclosing transaction can return `database.ErrAfterCommit` after a confirmed
commit if cleanup or an after-commit hook fails. Do not retry the transaction on
that result: the change and audit row can already be committed. Transaction
ownership and recovery stay in [Transactions](https://arkoren.dev/docs/guides/database/transactions.md).

## Durable Forwarding

If an audit destination is remote, write a pending record in the same transaction
and have an application-owned worker forward it. This durable pending record is
often called an **outbox**. The worker needs an explicit identifier, occurrence
time, and approved actor/resource fields; it cannot rely on the former request's
context. Make forwarding retry-safe and define deduplication at the destination.

A queue call from an ordinary event listener is only a best-effort handoff. An
after-commit queue call avoids sending rolled-back changes, but a crash between
commit and dispatch can still lose it. Neither gives exactly-once delivery.
[Queues](https://arkoren.dev/docs/guides/queues/overview.md) provide worker transport; atomic record creation,
forwarding policy, and retention remain your application's responsibility.

## Built-in Observations

[Built-in Events](https://arkoren.dev/docs/guides/events/built-in-events.md) can supplement an operational trail but use
best-effort publication and cannot enforce your required audit row. Match the
observation to the claim: payment call completion is not settlement, mail/push
acceptance is not delivery, and secret notifications do not cover console or
other-process changes. Session login changes request-local state before response-time
persistence; it is not evidence that a cookie or stored session reached the client.

Some built-in payloads contain raw diagnostics or identifying data. A broad
[pattern listener](https://arkoren.dev/docs/guides/events/listeners.md#pattern-listeners) should not forward whole
payloads or contexts. Select fields under your privacy policy, and use the
[shared telemetry handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle) for correlation
without treating a trace as a required business record.

## Checking Guarantees

Use an isolated `event.New()` bus to test publication and listener behavior.
Delivery is synchronous, so a sequential test can assert collected payloads
immediately after the operation returns; no polling is needed. Concurrent tests
must synchronize shared collectors.

For required records, test the application boundary, not only callback execution:
an insert failure must roll back the invoice change, successful commit must retain
both rows, and missing registration must fail your wiring check. Test forwarding
retries and destination deduplication separately. A bus test proves local callback
behavior, not database durability or external delivery.

For dispatch behavior, see the [`event` API reference](https://pkg.go.dev/arkoren.dev/core/event).

---

Source: https://arkoren.dev/docs/guides/broadcasting/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/broadcasting/overview)

# Broadcasting Overview

When an order is created, a search index and a dashboard can both need to hear
about it. A **broadcast** announces that fact without making the publisher know
which services listen. The publisher sends once; the transport determines which
subscribers receive a copy.

Arkoren separates publishing from receiving. A `broadcast.Topic[T]` names the
fact and fixes its payload type. A `Publisher` JSON-encodes that payload; a
`Consumer` routes received messages to registered listeners. A
[driver](https://arkoren.dev/docs/guides/broadcasting/drivers.md) connects these facades to an in-process broker or an external
service. Constructing a facade neither starts nor takes ownership of its driver.

## Choosing Broadcasts

Use broadcasts for notifications whose recipients can recover current state,
such as refreshing an order index. Most broadcast drivers do not retain messages
for offline subscribers. A successful publish confirms transport acceptance,
not that any recipient handled the notification.

Use a [queue](https://arkoren.dev/docs/guides/queues/overview.md) to dispatch work to competing workers with
a retry policy, such as generating an invoice. Use an
[event](https://arkoren.dev/docs/guides/events/overview.md) to notify listeners within the same process.
Broadcasting is not a substitute for either a durable work queue or a direct
in-process call.

Independent subscribers ordinarily each receive a copy. Replicas do not
automatically share that copy: use a [NATS queue group](https://arkoren.dev/docs/guides/broadcasting/drivers.md#nats), or
an [SNS subscription with one SQS queue per service](https://arkoren.dev/docs/guides/broadcasting/drivers.md#sns). Replicas
of a service then compete on its queue; independent services retain separate
subscriptions.

## Defining Topics

Publish an order identifier rather than a complete private order record when
recipients can load the authorized data themselves. Share the topic declaration
and JSON contract between publisher and listener:

```go
import "arkoren.dev/core/service/broadcast"

type OrderCreated struct {
	OrderID string `json:"order_id"`
}

var OrderCreatedTopic = broadcast.NewTopic[OrderCreated]("order.created")
```

A name contains dot-separated tokens, each 1–64 lowercase letters, digits, or
hyphens, without a leading or trailing hyphen. The complete name is at most
255 bytes. Topic construction retains the name without validation; publication
and registration return an error matching `broadcast.ErrInvalidName` for invalid
names. They do not normalize names.

Topics sharing a name must agree on JSON payload semantics, even when services
declare them separately. [AsyncAPI](https://arkoren.dev/docs/guides/asyncapi/overview.md) can describe this
wire contract; generating a document does not configure subscriptions.

## Publishing Facts

With an existing `driver` from [Drivers](https://arkoren.dev/docs/guides/broadcasting/drivers.md) and operation context `ctx`,
wrap the publishing side and announce the committed order:

```go
publisher := broadcast.NewPublisher(driver)

err := publisher.Publish(ctx, OrderCreatedTopic, OrderCreated{
	OrderID: "o-1042",
})
```

The envelope carries name `order.created` and JSON payload
`{"order_id":"o-1042"}`. A nil error means acceptance by the driver, not listener
completion or retention. Encoding errors occur before transport access; driver
errors retain their causes. Focused snippets leave routine error results visible;
your application must handle them.

With non-retaining drivers, publishing before subscribers are active loses the
notification. Driver setup and listener registration alone do not establish an
active subscription.

Commit the order before publishing it. The commit and publish are separate
operations: a crash between them can lose the notification. If that loss is
unacceptable, persist the intent to publish alongside the order and arrange
recovery. Retrying an uncertain publication can duplicate a notification.

## Listening

In a receiving service, use a driver connected to the publisher's scope, such as
the same Redis prefix or RabbitMQ exchange. SNS instead uses a subscribed SQS
queue. With your application's `search` indexer, register a typed listener:

```go
consumer := broadcast.NewConsumer(driver)

unsubscribe, err := consumer.Listen(OrderCreatedTopic,
	func(ctx context.Context, order OrderCreated) error {
		return search.Index(ctx, order.OrderID)
	})
if err != nil {
	return err
}
defer unsubscribe()
```

An accepted delivery decodes into `OrderCreated`, then calls the indexer with
`o-1042`. Keep the registration alive for as long as you need it; the deferred
unsubscribe belongs to a scope that outlives consumption, not a short-lived
setup function.

Registration does not subscribe to the broker. Run the consumer in your
application's supervised scope:

```go
err := consumer.Run(ctx)
```

`Run` blocks until cancellation or a driver error. Starting a goroutine or
registering an application runner is not a subscription-ready signal. For
non-retaining drivers, establish active consumption before publishing a
notification you need to observe. [Drivers](https://arkoren.dev/docs/guides/broadcasting/drivers.md#driver-lifecycle) owns
connection and cleanup setup.

Listener errors do not reach the publisher, and they do not by themselves end
`Run`. The driver decides whether to retry. A retry can invoke successful
listeners again, so make repeated notifications safe. See
[Message Delivery](https://arkoren.dev/docs/guides/broadcasting/message-delivery.md) for cancellation, ordering, and failure
handling.

## Matching Names

To observe several order facts, register a pattern listener. It receives the
complete envelope rather than a decoded application value. With an existing
`logger`, record only its routing name:

```go
unsubscribe, err := consumer.ListenPattern("order.#",
	func(ctx context.Context, message broadcast.Message) error {
		logger.InfoContext(ctx, "order notification", "name", message.Name)
		return nil
	})
if err != nil {
	return err
}
defer unsubscribe()
```

`order.#` matches `order`, `order.created`, and `order.created.eu`. A final `#`
matches zero or more tokens; `*` matches exactly one. Thus `order.*` matches only
the middle example. Wildcards must fill whole tokens, and `#` must be last:
`order.cre*` is invalid.

Every matching listener runs, including an exact listener and a pattern listener
on the same consumer. Raw payload bytes and metadata are shared between these
listeners; do not mutate them. Avoid logging their contents. The
[monitoring guide](https://arkoren.dev/docs/guides/broadcasting/metadata-and-telemetry.md) explains metadata and privacy.

## Testing Publications

To check that application code announces an order without running a broker, use
the recording fake. In a test with `t *testing.T`, assume `CreateOrder` accepts
the publisher and publishes the topic above after saving the order:

```go
import "arkoren.dev/core/service/broadcast/broadcasttest"

fake := broadcasttest.New()
publisher := broadcast.NewPublisher(fake)

if err := CreateOrder(t.Context(), publisher, "o-1042"); err != nil {
	t.Fatal(err)
}
fake.AssertPublished(t, OrderCreatedTopic, func(order OrderCreated) bool {
	return order.OrderID == "o-1042"
})
```

This proves the publication request and payload, not broker delivery. To exercise
registered listeners, `fake.Drain(t.Context(), consumer)` routes recorded
messages synchronously and returns joined listener errors without redelivery.
The [testing guide](https://arkoren.dev/docs/guides/testing/fakes.md#separate-execution) explains why requesting
work and executing its effects need separate assertions.

For every function and option, see the
[`broadcast` API reference](https://pkg.go.dev/arkoren.dev/core/service/broadcast).

---

Source: https://arkoren.dev/docs/guides/broadcasting/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/broadcasting/drivers)

# Broadcast Drivers

Choose a transport by what recipients can miss and how service replicas share
notifications. Memory stays within one process. Redis Pub/Sub and Core NATS suit
live notifications; NATS queue groups can share them among replicas. RabbitMQ
uses temporary AMQP subscriptions. MQTT suits MQTT 5 deployments, but its owned
sessions are not persistent. SNS with subscribed SQS queues retains notifications
for offline services, subject to queue policies.

None of these drivers confirms recipient handling to the publisher. A broker
connection is not an active subscription, and protocol acknowledgement is not
proof of a successful application effect. See [Message Delivery](https://arkoren.dev/docs/guides/broadcasting/message-delivery.md)
for the consequences of retries and cancellation.

## Compatibility Matrix

This compares Arkoren's exposed behavior, not every feature of each broker.
Replica sharing means competing for one service's notifications, not receiving
independent fan-out copies. Listener retry means redelivery caused by the
consumer's returned error, not native connection retries.

| Driver | Offline Retention | Replica Sharing | Listener Retry |
| --- | --- | --- | --- |
| [Memory](#memory) | No | No | No |
| [Redis](#redis) | No | No | No |
| [NATS](#nats) | No | Partial — Set the same queue group on replicas. | No |
| [RabbitMQ](#rabbitmq) | No | No | No |
| [MQTT](#mqtt) | No | No | No |
| [SNS with SQS](#sns) | Partial — Subscribed queues retain messages under their retention and redrive policies. | Yes | Partial — Nonpermanent failures remain for native redelivery. |
| [Direct SQS](#direct-sqs) | Partial — A provisioned queue retains messages under its policies. | Yes | Partial — Nonpermanent failures remain for native redelivery. |

The MQTT row describes `New`'s clean session. A caller-owned Paho manager keeps
its native session policy; that does not add a portable persistence guarantee.
With SNS, give each independent service its own queue, then share that queue
among the service's replicas. Direct SQS alone is point-to-point, not fan-out.

## Driver Lifecycle

Import `arkoren.dev/core/service/broadcast` for the facades. Install external
driver modules from your application's module directory. These focused fragments
assume an operation context `ctx`; handle constructor errors before registering
or using a returned driver.

For Redis, NATS, RabbitMQ, and MQTT, an existing application composer can own
connection setup, readiness, and the consumer runner:

```go
composer.Cleanup(driver.Stop)
composer.Startup(driver.Start)
composer.Check("broadcast", driver.Ping)
composer.Run(consumer.Run)
```

Register cleanup before startup so acquired resources are released even after
startup failure. Construct the consumer and register its listeners before the
application runs. `Start` establishes connectivity; `consumer.Run` owns the
subscription. For non-retaining transports, do not publish a required test
notification until consumption is active. There is no shared subscription-ready
method. See [Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md)
for application supervision.

For Memory, register only `Stop` and `consumer.Run`: `Start` is a no-op and there
is no `Ping`. SNS/SQS has no driver lifecycle hooks; register only the consumer
runner. A publish-only service needs no consumer runner.

Cancel and join consumers and release raw subscriptions before resource cleanup.
Driver shutdown cannot forcibly stop arbitrary application callbacks. Borrowed
native resources stay caller-owned; keep them alive until all users finish.

The local examples use dedicated project names and loopback ports. Run each in
a new directory with the shown Compose file. If its name or port is already in
use, choose another disposable name or port; do not stop an existing deployment
to make room. Never reuse these cleanup commands for production resources.

## Memory

Use the bundled memory driver for asynchronous notifications within one process.
Use the [recording fake](https://arkoren.dev/docs/guides/broadcasting/overview.md#testing-publications) when a test only needs
to assert publication. No broker or additional module is required:

```go
import "arkoren.dev/core/service/broadcast/memorybroadcast"

driver := memorybroadcast.New()
publisher := broadcast.NewPublisher(driver)
consumer := broadcast.NewConsumer(driver)
```

Construction starts no work. Register the [order listener](https://arkoren.dev/docs/guides/broadcasting/overview.md#listening),
run the consumer, then [publish an order](https://arkoren.dev/docs/guides/broadcasting/overview.md#publishing-facts) through
the same driver instance. Only already-running consumers receive the message;
each `Consume` gets its own payload bytes and metadata map. Restarting the
process loses all broker state.

### Delivery Limits

If listeners need a lower concurrency bound, change the shared handler admission
limit at construction:

```go
driver := memorybroadcast.New(func(broker *memorybroadcast.Builder) {
	broker.MaxConcurrentDeliveries(64)
})
```

The default is 1024 simultaneous deliveries across consumers; nonpositive values
restore that default. `Logger` defaults to `slog.Default()` and records delivery
failures without retrying. A full consumer buffer blocks publication until space
is available or the publishing context ends. A publication can fail after some
consumers accepted their copies; fan-out is not an atomic transaction.

Cancellation and `Stop` discard buffered messages and interrupt messages waiting
for a handler slot. Admitted handlers preserve consumer context values but lose
its cancellation and deadline; they can outlive `Consume`. Give each handler its
own work deadline. `Stop` closes admission even with an expired context, then
waits for admitted handlers only until its context ends. It neither cancels
those handlers nor waits for `Consume` calls to return. A later `Stop` can wait
for the same handlers again.

After shutdown, valid operations with live contexts return
`memorybroadcast.ErrBrokerClosed`. An already-canceled context takes precedence.

See the [`memorybroadcast` API](https://pkg.go.dev/arkoren.dev/core/service/broadcast/memorybroadcast).

## Redis

Choose Redis Pub/Sub if you already run Redis and recipients can reload state
after missed notifications. Connected consumers each receive a copy; this driver
does not use Streams, retention, consumer groups, or listener retries.

```sh
go get arkoren.dev/driver/redis
```

### Connecting Redis

`New` constructs an owned client without network I/O. `Start` verifies
connectivity; `Stop` closes that client and waits for subscription goroutines:

```go
import "arkoren.dev/driver/redis/redisbroadcast"

driver := redisbroadcast.New()
publisher := broadcast.NewPublisher(driver)
consumer := broadcast.NewConsumer(driver)
```

The defaults are TCP at `localhost:6379`, no credentials or TLS, and literal
prefix `arkoren.`. Apply the [lifecycle hooks](#driver-lifecycle), then use the
[typed order listener](https://arkoren.dev/docs/guides/broadcasting/overview.md#listening) and
[publisher](https://arkoren.dev/docs/guides/broadcasting/overview.md#publishing-facts).

On a shared server, choose a distinct namespace. Redis Pub/Sub ignores database
numbers, so `DB` cannot isolate applications:

```go
driver := redisbroadcast.New(func(broker *redisbroadcast.Builder) {
	broker.Addr("localhost:6379")
	broker.Prefix("orders:")
})
```

Prefix bytes are literal, including Redis glob metacharacters.
Set `Username`, `Password`, and `TLS` from your deployment configuration for
authenticated, encrypted connections. Prefixes route messages; they do not
authorize publishers. Use Redis channel ACLs or separate instances for untrusted
applications. Overlapping broad subscriptions can transfer foreign bytes into
the process before envelope rejection. For raw subscribers, use prefixes neither
of which starts with the other, such as `orders:` and `billing:`.

### Native Clients

To reuse an existing `*redis.Client` from `github.com/redis/go-redis/v9`, borrow it
with `NewFrom`:

```go
driver := redisbroadcast.NewFrom(client, func(broker *redisbroadcast.Builder) {
	broker.Prefix("orders:")
})
```

Only `Prefix` and `Logger` apply. Configure native `ContextTimeoutEnabled` before
wrapping the client if operation deadlines must bound command socket I/O; `New`
enables it for its owned client. `Client()` exposes the same pointer without
ownership transfer. End consumers and raw subscriptions before `Stop`, then let
the owner close a borrowed client. Native calls bypass facade encoding and
telemetry. Config formatting redacts `Password`, not direct field access or
native configuration.

Owned `Stop` closes its client even with an expired context, then waits for
subscription goroutines only until that context ends.

### Local Redis

Use Docker in a new disposable directory, with this `compose.yaml`. A named
project avoids operating an existing Compose deployment:

```yaml
services:
  redis:
    image: redis:8-alpine
    ports:
      - "127.0.0.1:16379:6379"
```

Start and verify only this project:

```sh
docker compose -p arkoren-broadcast-redis-demo up -d
docker compose -p arkoren-broadcast-redis-demo exec redis redis-cli ping
```

`PONG` confirms connectivity. Use `broker.Addr("localhost:16379")` in the
constructor, start consumption, then publish `o-1042` and check the indexer's
effect. These are separate checks: connectivity does not prove delivery.

```sh
docker compose -p arkoren-broadcast-redis-demo down --volumes
```

This removes only the disposable project's server and data. See the
[`redisbroadcast` API](https://pkg.go.dev/arkoren.dev/driver/redis/redisbroadcast).

## NATS

Use Core NATS for live cross-process notifications. Set a queue group when replicas
of one service should share notifications. This driver does not use JetStream;
queue groups provide load sharing, not offline retention.

```sh
go get arkoren.dev/driver/nats
```

### Connecting NATS

`New` validates settings without network or file I/O. `Start` connects, and `Stop`
drains and closes the owned connection:

```go
import "arkoren.dev/driver/nats/natsbroadcast"

driver, err := natsbroadcast.New()
```

The default URL is `nats://127.0.0.1:4222` and the subject prefix is `arkoren`.
Handle construction errors, apply [lifecycle hooks](#driver-lifecycle), and wrap
the driver in `broadcast.NewPublisher` and `broadcast.NewConsumer` for the
[order notification](https://arkoren.dev/docs/guides/broadcasting/overview.md#listening).

To share notifications among indexer replicas, use the same prefix and group:

```go
driver, err := natsbroadcast.New(func(broker *natsbroadcast.Builder) {
	broker.Prefix("orders")
	broker.QueueGroup("indexers")
})
```

An independent group receives its own copy. An empty group gives every consumer
a copy. Prefixes must be literal NATS subjects without wildcards, whitespace,
or empty tokens; queue groups cannot contain spaces, tabs, or line breaks.
Enforce broker subject permissions: a prefix is not an authorization boundary.

### Connection Policy

`URLs` accepts multiple failover servers, schemeless hosts, or `nats`, `tls`,
`ws`, and `wss` URLs. Empty lists select the default; empty entries, missing hosts,
and unsupported schemes fail validation without echoing the address. Use TLS and
load credentials from configuration rather than hard-coding them in URLs.

`CredentialsFile` is a JWT/seed file path read by the SDK during connection,
authentication, and potentially reconnect. `NKeySeed` instead takes encoded
user seed material, never a path. The seed string stays in memory for signing;
wiping temporary signing keys does not erase it. Basic authentication applies
only when both `Username` and `Password` are nonempty; `Token` is another option.

Reconnect attempts default to unlimited; `MaxReconnects(0)` also selects that
default. Set a positive limit or `NoReconnect(true)` to change it.
`ReconnectWait(0)` restores two seconds. `Timeout(0)` selects the native
two-second connection timeout, not an operation deadline. Negative wait and
timeout values are invalid. `Logger(nil)` selects `slog.Default()`.

Config's `String`, `GoString`, `LogValue`, and dispatched formatting hide Password,
Token, NKeySeed, and URL userinfo; Username remains visible. Unparseable URLs and
entries without an authority hostname are hidden wholesale. This does not redact
direct fields, serialization, native objects, or arbitrary errors. Go bypasses
formatting methods for invalid `%p` on a Config value and invalid `%w` on values
or pointers. For an externally chosen format string, format `record.String()`,
not the record itself, to avoid exposing raw credentials.

### Native Connections

With an existing nonnil `*nats.Conn` from `github.com/nats-io/nats.go`, borrow it
instead of creating an owned connection:

```go
driver, err := natsbroadcast.NewFrom(connection, func(broker *natsbroadcast.Builder) {
	broker.Prefix("orders")
	broker.QueueGroup("indexers")
})
```

Nil connections and invalid routing match `natsbroadcast.ErrInvalidConfig`.
Only Prefix, QueueGroup, and Logger apply; `Start` does not reconfigure or dial
the supplied connection. Borrowed `Stop` does nothing. Cancel and join consumers,
release raw subscriptions, then drain or close the connection yourself.
`Connection()` returns the borrowed pointer immediately, or nil before owned
startup, without transferring ownership. Native operations bypass envelope checks
and facade telemetry.

Owned `Stop` waits for draining and closure. Its context ending forces close and
returns the context error; native drain timeout remains identifiable as
`nats.ErrDrainTimeout`. Forced close cannot stop an application callback.
Connection closure does not end `Consume`'s context wait: cancel and join consumers
separately. Canceling `Start` abandons dialing and closes any connection obtained
later; native dialing can finish in the background.

Already-canceled Publish, Consume, and Subscribe contexts are rejected before
native access. Later cancellation is not a guarantee that a native publish or
subscription setup is interrupted. Portable `#` patterns become exact and `>`
subscriptions where needed because native NATS `>` excludes zero tokens.

### Local NATS

In a new disposable directory, save this `compose.yaml`:

```yaml
services:
  nats:
    image: nats:2-alpine
    command: ["-m", "8222"]
    ports:
      - "127.0.0.1:14222:4222"
      - "127.0.0.1:18222:8222"
```

Start the isolated project and check its local health endpoint:

```sh
docker compose -p arkoren-broadcast-nats-demo up -d
curl --fail http://localhost:18222/healthz
```

A successful response confirms readiness. Configure
`broker.URLs("nats://localhost:14222")`, start the consumer, then publish `o-1042`
and check the indexer. Remove the disposable server afterward:

```sh
docker compose -p arkoren-broadcast-nats-demo down --volumes
```

See the [`natsbroadcast` API](https://pkg.go.dev/arkoren.dev/driver/nats/natsbroadcast).

## RabbitMQ

Choose RabbitMQ if you already use AMQP topic routing. Each consuming process
gets an exclusive, auto-delete queue with automatic acknowledgement. Offline
subscribers miss notifications; for durable work use the
[RabbitMQ queue driver](https://arkoren.dev/docs/guides/queues/drivers.md#rabbitmq).

```sh
go get arkoren.dev/driver/rabbitmq
```

### Connecting RabbitMQ

Load the credential-bearing URL from configuration; use `amqps://` outside
local development. `New` rejects blank URLs without network I/O:

```go
import (
	"os"

	"arkoren.dev/driver/rabbitmq/rabbitmqbroadcast"
)

driver, err := rabbitmqbroadcast.New(os.Getenv("AMQP_URL"))
```

`Start` connects and declares the topic exchange. Register the
[lifecycle hooks](#driver-lifecycle) and wrap the driver in the facades to use
the [order listener](https://arkoren.dev/docs/guides/broadcasting/overview.md#listening). Owned negotiation has a 30-second
budget or the startup context's earlier end. Serialize `Start` and `Stop`.

`Exchange` defaults to `arkoren:events`; `Logger` defaults to `slog.Default()`.
Use separate exchanges or virtual hosts to isolate applications. Shared exchanges
also need topic permissions to restrict publication keys. Envelope routing checks
do not authenticate a publisher or payload.

Publication waits for broker confirmation, retrying once after reopening the
publisher channel on a non-cancellation failure. Losing the first confirmation
can therefore duplicate an accepted publication. Automatic acknowledgement does
not make the entire path at-most-once, and confirmation does not establish
retention or processing.

### Connection Ownership

On an owned connection, canceling a blocking SDK setup or write can close the
socket, interrupting every channel sharing it. Keep unrelated native work on a
separate connection. Confirmation waiting is separately context-aware: its
cancellation does not always close the socket or prove nondelivery.

For an existing `*amqp091.Connection` from
`github.com/rabbitmq/amqp091-go`, use `rabbitmqbroadcast.NewFrom(connection)`.
It returns a driver and error; `Start` still opens a publisher channel and declares
the exchange. `Stop` closes that channel but never the caller's connection.
Borrowed SDK channel-open, topology, write, and close RPCs cannot be forcibly
bounded by the driver's context. A responsive peer, native socket deadlines, or
caller-managed connection close must unblock stuck operations.

`Connection()` exposes the shared pointer without transferring ownership: nil
before owned startup, immediately available when borrowed, and still available
after shutdown. Do not close an owned connection outside `Stop`. Native calls
bypass envelope and lifecycle protections and facade telemetry.

Owned `Stop` is terminal and releases the connection before waiting on channel
handshakes or confirmations. Graceful close has at most five seconds or the
earlier end of its context; even expired contexts force socket release. It does
not wait for application handlers. Consumer channel cleanup has its own five-second
budget and can abort an unresponsive owned socket. A closed subscription makes
`Run` return `rabbitmqbroadcast.ErrSubscriptionClosed` unless cancellation supplies
the exit reason.

### Local RabbitMQ

In a new disposable directory, save this `compose.yaml`. These plaintext
credentials are only for this loopback-bound development server:

```yaml
services:
  rabbitmq:
    image: rabbitmq:4-alpine
    environment:
      RABBITMQ_DEFAULT_USER: local
      RABBITMQ_DEFAULT_PASS: local-only
    ports:
      - "127.0.0.1:15672:5672"
```

Start the isolated project and repeat the diagnostic until successful:

```sh
docker compose -p arkoren-broadcast-rabbitmq-demo up -d
docker compose -p arkoren-broadcast-rabbitmq-demo exec rabbitmq rabbitmq-diagnostics -q ping
```

Set `AMQP_URL=amqp://local:local-only@localhost:15672/` for your local application.
After consumption is active, publish `o-1042` and check the indexer. Remove only
the disposable project's resources:

```sh
docker compose -p arkoren-broadcast-rabbitmq-demo down --volumes
```

See the [`rabbitmqbroadcast` API](https://pkg.go.dev/arkoren.dev/driver/rabbitmq/rabbitmqbroadcast).

## MQTT

Use the experimental MQTT 5 driver with brokers such as Mosquitto, EMQX, or
HiveMQ. `New` owns a clean, nonpersistent session: offline subscribers lose
notifications at every QoS. QoS 2 is a protocol guarantee, not exactly-once
application effects.

```sh
go get arkoren.dev/driver/mqtt
```

### Connecting MQTT

`New` validates broker URLs and settings without network I/O. `Start` creates the
owned Paho manager and waits for its first connection:

```go
import "arkoren.dev/driver/mqtt/mqttbroadcast"

driver, err := mqttbroadcast.New([]string{"mqtt://localhost:1883"})
```

Handle the error, apply [lifecycle hooks](#driver-lifecycle), then use the facades
for [typed notifications](https://arkoren.dev/docs/guides/broadcasting/overview.md#publishing-facts). Multiple URLs allow
failover. Supported schemes are `mqtt`, `tcp`, `mqtts`, `ssl`, `tls`, `mqtt+ssl`,
`tcps`, `ws`, and `wss`. All require a host. TCP/TLS URLs require ports 1–65535
and reject paths and queries; WebSocket URLs preserve paths and queries and can
omit the port. Fragments and URL userinfo are rejected without echoing URL
credentials. Use `mqtts://` and builder `Username`/`Password` outside local
development; passwords apply only with a nonempty username.

### Protocol Settings

The default QoS is 1. `QoS(0)` explicitly selects at-most-once protocol delivery;
1 requests at-least-once and 2 requests exactly-once at the protocol level.
Other values fail construction. QoS 1 and 2 publication waits for protocol
completion and treats negative reason codes as errors, even if Paho returns no
error. A timeout does not prove nondelivery. Listener errors do not request
redelivery or prevent normal acknowledgement.

Already-canceled Publish, Consume, and Subscribe contexts install or send
nothing. Later cancellation of raw setup does not remove its subscription;
call the returned unsubscribe function.

With automatic acknowledgement, QoS 1 acknowledgement follows all matching
callbacks, including failures. A borrowed manager retains any manual policy.
Up to 1024 delivery callbacks run concurrently; saturation blocks admission and
backpressures Paho's receive loop rather than dropping work.

If your broker requires a different keepalive interval, use a duration:

```go
import "time"

driver, err := mqttbroadcast.New([]string{"mqtt://localhost:1883"},
	func(broker *mqttbroadcast.Builder) {
		broker.KeepAlive(45 * time.Second)
	})
```

Zero restores 30 seconds; negative disables keepalive. Positive values must be
whole seconds from 1 through 65535. The broker can override the interval in its
CONNACK. Numeric `KeepAlive(30)` means 30 nanoseconds, not 30 seconds, and is invalid.
`Logger(nil)` uses `slog.Default()` for panics and failed subscription restoration,
not Paho's native loggers. Config formatting redacts Password, not direct fields
or native data.

Names map dots to slashes, so `order.created` uses MQTT topic `order/created`.
Pattern `*` maps to `+`, and final `#` remains `#`. Enforce broker topic ACLs or
separate namespaces for untrusted publishers. Routing checks do not authorize
publishers.

### Native Managers

To customize TLS, native logging, or session policy, borrow a Paho manager. The
driver's filters belong to the whole native client; do not share those filters
with unrelated subscriptions because cleanup unsubscribes them for that client.
Only QoS and Logger apply to `NewFrom`; invalid QoS is reported at use because
that constructor has no error result.

With your existing `autopaho.ClientConfig` named `config`, install both driver
hooks before native construction:

```go
import (
	"github.com/eclipse/paho.golang/autopaho"
	"github.com/eclipse/paho.golang/paho"
)

driver := mqttbroadcast.NewFrom(nil)
config.ClientConfig.OnPublishReceived = append(config.ClientConfig.OnPublishReceived, driver.HandlePublish)
previous := config.OnConnectionUp
config.OnConnectionUp = func(manager *autopaho.ConnectionManager, reply *paho.Connack) {
	if previous != nil {
		previous(manager, reply)
	}
	driver.HandleConnectionUp(manager, reply)
}
```

Callbacks must return promptly. `HandleConnectionUp` schedules filter restoration
without waiting in the connection callback; failed restoration is logged once,
not retried on the same connection.

Create the manager under a separately owned, long-lived `sessionCtx`. With an
owning application `composer`, register cleanup before waiting for connection:

```go
import (
	"context"
	"errors"
)

manager, err := autopaho.NewConnection(sessionCtx, config)
if err != nil {
	return err
}
driver.SetClient(manager)
composer.Cleanup(func(cleanup context.Context) error {
	return errors.Join(driver.Stop(cleanup), manager.Disconnect(cleanup))
})
err = manager.AwaitConnection(ctx)
```

The native owner must disconnect the manager even if `AwaitConnection` fails.
The cleanup stops the driver before disconnecting, and preserves both errors.
Do not make a request context own a shared connection. `SetClient` attaches once,
installs no hooks, and transfers no ownership. `Client()` exposes the pointer;
native operations bypass facade encoding and telemetry. With a borrowed manager,
`Start` does nothing.

Cancel and join consumers, then call `driver.Stop(cleanup)` before
`manager.Disconnect(cleanup)`, using a fresh bounded cleanup context and handling
both errors. `Stop` is terminal, drains admitted callbacks and reconnect work
until its context ends, clears local registrations, and attempts filter cleanup
even after a drain timeout. It disconnects owned managers only. Borrowed filter
cleanup uses a separate ten-second acknowledgement timeout; native writes and
arbitrary callbacks can outlive context deadlines. A failed unsubscribe may
require the native owner to disconnect the session. `Stop` does not end
`Consume`'s context wait.

### Local MQTT

In a new disposable directory, save this `compose.yaml`. The anonymous plaintext
server is intentionally bound only to loopback:

```yaml
services:
  mqtt:
    image: eclipse-mosquitto:2
    command: ["mosquitto", "-c", "/mosquitto-no-auth.conf"]
    ports:
      - "127.0.0.1:11883:1883"
```

Start only this project:

```sh
docker compose -p arkoren-broadcast-mqtt-demo up -d
```

Connect with `mqttbroadcast.New([]string{"mqtt://localhost:11883"})`.
Successful `Start(ctx)` confirms connectivity. Run the consumer, publish
`o-1042`, and check the indexer to verify envelope delivery, not just connection.

```sh
docker compose -p arkoren-broadcast-mqtt-demo down --volumes
```

See the [`mqttbroadcast` API](https://pkg.go.dev/arkoren.dev/driver/mqtt/mqttbroadcast).

## SNS

Use SNS with SQS when independent services must receive notifications published
while they are offline. Provision an SNS topic, one SQS queue per service, and
subscriptions first; the drivers create none of these resources. Retention,
visibility, and redrive remain queue policies, not facade guarantees.

```sh
go get arkoren.dev/driver/aws
```

### Connecting AWS

The SNS driver only publishes. Map each logical broadcast name to an existing
topic ARN; the default AWS region and credential chain supplies configuration:

```go
import "arkoren.dev/driver/aws/awsbroadcast"

outbox, err := awsbroadcast.NewSNS(ctx, map[string]string{
	"order.created": "arn:aws:sns:eu-west-1:123456789012:orders-created",
})
```

Wrap `outbox` with `broadcast.NewPublisher`. The SQS driver receives from one
service's subscribed queue:

```go
inbox, err := awsbroadcast.NewSQS(ctx, func(broker *awsbroadcast.SQSBuilder) {
	broker.QueueURL("https://sqs.eu-west-1.amazonaws.com/123456789012/order-indexer")
})
consumer := broadcast.NewConsumer(inbox)
```

Register the [order listener](https://arkoren.dev/docs/guides/broadcasting/overview.md#listening) and consumer runner; neither
AWS driver has Start, Stop, or Ping hooks. Construction does not verify resource
existence, subscriptions, or permissions. Grant the publisher `sns:Publish`, the
consumer `sqs:ReceiveMessage` and `sqs:DeleteMessage`, and the SNS topic permission
to send to the queue through its queue policy. Use IAM and queue policies for
authorization; local listener patterns are not access controls.

### Queue Policies

The SNS map must be nonempty, with valid broadcast names and nonblank ARNs.
SQS requires a consuming `QueueURL`, publishing `Queues`, or both. Invalid
settings wrap `awsbroadcast.ErrInvalidConfig`; unmapped publication names match
`awsbroadcast.ErrUnknownName`. Constructors copy routing maps. Nonblank ARN and
URL syntax and actual permissions are verified by AWS when used.

SQS polls for 20 seconds and receives one message by default.
`WaitTimeSeconds` accepts 1–20, `MaxMessages` accepts 1–10, and zero restores
their defaults. `Logger(nil)` uses `slog.Default()` for invalid deliveries,
handler errors, and deletion failures. It processes a batch sequentially and does
not extend visibility while listeners run. Set the queue's visibility timeout to
cover the entire batch, or another replica can process a delivery still being
handled.

Successful messages and malformed envelopes are deleted; so are deliveries whose
listener failures are all permanent. Other failures remain for native redelivery
to every matching listener. Even successful handling can repeat if deletion
fails. Make effects idempotent and configure queue redrive for messages that keep
failing. Permanent failures are deleted here, not moved to a dead-letter queue by
the driver. Deletion uses a detached ten-second budget so cancellation after
handling does not by itself prevent acknowledgement; failed deletion is logged.

SNS supplies the message attribute `name`, so a subscription filter policy such
as `{"name":["order.created"]}` can select names before queue delivery. This is
broker configuration, separate from consumer patterns. Full broadcast envelopes
arrive either directly or inside an SNS notification wrapper. Their payload bytes
use the envelope's base64 JSON representation, not a bare order object.

### Direct SQS

If only one service consumes a notification, publish directly to its provisioned
queue instead of adding an SNS topic. With its URL named `queueURL`:

```go
outbox, err := awsbroadcast.NewSQS(ctx, func(broker *awsbroadcast.SQSBuilder) {
	broker.Queues(map[string]string{"order.created": queueURL})
})
publisher := broadcast.NewPublisher(outbox)
```

Replicas on that queue compete for messages. Use SNS subscriptions instead for
independent fan-out copies.

SNS ARNs and SQS queue URLs ending in `.fifo` cause publications to group by
broadcast name. Native deduplication uses `Message.ID` if supplied directly to
the driver, otherwise a fresh random ID. Facade publications supply no ID:
repeating `publisher.Publish` does not deduplicate an application event, even on
FIFO destinations.

### Native Clients

Use `NewSNSFrom` or `NewSQSFrom` with existing clients from the AWS SDK v2
`service/sns` or `service/sqs` packages to customize region, endpoints, credentials,
and native policy. Nil clients are rejected. `Client()` exposes the pointer
without ownership transfer. The drivers have no client-close hooks; native
configuration and transport cleanup remain caller-owned. Native calls bypass
facade encoding and telemetry.

### Local SNS

Use Docker, AWS CLI, and Python 3 in a new disposable directory. Save this
`compose.yaml`; only SNS/SQS run, bound to loopback:

```yaml
services:
  aws:
    image: localstack/localstack:4
    environment:
      SERVICES: sns,sqs
    ports:
      - "127.0.0.1:14568:4566"
```

Start the isolated project and check its health:

```sh
docker compose -p arkoren-broadcast-aws-demo up -d
curl --fail http://localhost:14568/_localstack/health
```

Wait for SNS and SQS to be available. In a disposable shell, disable AWS profiles
and use dummy local credentials. Every provisioning command below explicitly
targets the emulator, never a production account:

```sh
unset AWS_PROFILE AWS_DEFAULT_PROFILE AWS_SESSION_TOKEN
export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=eu-west-1 AWS_EC2_METADATA_DISABLED=true
queueURL=$(aws --endpoint-url http://localhost:14568 sqs create-queue --queue-name order-indexer --query QueueUrl --output text)
topicARN=$(aws --endpoint-url http://localhost:14568 sns create-topic --name orders-created --query TopicArn --output text)
```

These commands return routing values for the local clients. The following policy
replacement is only for this newly created disposable queue. Retrieve its ARN,
then authorize this topic:

```sh
queueARN=$(aws --endpoint-url http://localhost:14568 sqs get-queue-attributes --queue-url "$queueURL" --attribute-names QueueArn --query Attributes.QueueArn --output text)
policy=$(python3 -c 'import json,sys; print(json.dumps({"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sns.amazonaws.com"},"Action":"sqs:SendMessage","Resource":sys.argv[1],"Condition":{"ArnEquals":{"aws:SourceArn":sys.argv[2]}}}]}))' "$queueARN" "$topicARN")
attributes=$(python3 -c 'import json,sys; print(json.dumps({"Policy":sys.argv[1]}))' "$policy")
aws --endpoint-url http://localhost:14568 sqs set-queue-attributes --queue-url "$queueURL" --attributes "$attributes"
```

Subscribe the queue so SNS can route publications into it:

```sh
aws --endpoint-url http://localhost:14568 sns subscribe --topic-arn "$topicARN" --protocol sqs --notification-endpoint "$queueARN"
```

For local Go code, configure the native SDK with static dummy credentials and
explicit endpoints, rather than relying on your normal AWS profile:

```go
import (
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/sns"
	"github.com/aws/aws-sdk-go-v2/service/sqs"
)

cfg := aws.Config{
	Region:      "eu-west-1",
	Credentials: credentials.NewStaticCredentialsProvider("test", "test", ""),
}
client := sns.NewFromConfig(cfg, func(options *sns.Options) {
	options.BaseEndpoint = aws.String("http://localhost:14568")
})
queue := sqs.NewFromConfig(cfg, func(options *sqs.Options) {
	options.BaseEndpoint = aws.String("http://localhost:14568")
})
```

Supply the CLI results as Go strings `topicARN` and `queueURL`, then construct
the drivers without provisioning anything further:

```go
outbox, err := awsbroadcast.NewSNSFrom(client, map[string]string{
	"order.created": topicARN,
})
inbox, err := awsbroadcast.NewSQSFrom(queue, func(broker *awsbroadcast.SQSBuilder) {
	broker.QueueURL(queueURL)
})
```

Use the facades to publish `o-1042` and check the indexer's effect. A successful
SNS publication alone does not verify subscription delivery. Remove only this
disposable project's emulator, topics, subscriptions, and queues afterward:

```sh
docker compose -p arkoren-broadcast-aws-demo down --volumes
```

See the [`awsbroadcast` API](https://pkg.go.dev/arkoren.dev/driver/aws/awsbroadcast).

## Raw Subscriptions

Redis, NATS, RabbitMQ, and MQTT also expose `Subscribe` for native producers that
do not send Arkoren envelopes. With one of these drivers and
`handleBytes func([]byte)`, own the returned resource explicitly:

```go
unsubscribe, err := driver.Subscribe(ctx, "order.#", handleBytes)
if err != nil {
	return err
}
err = unsubscribe()
```

Keep the subscription alive for the receiving scope; this final call illustrates
its cleanup. The return type is `func() error`, unlike the facade's
`broadcast.Unsubscribe`. Handle cleanup errors. Raw subscriptions bypass envelope
validation, typed listener routing, and facade telemetry. Publications from these
drivers arrive as full JSON `broadcast.Message` envelopes, not bare application
payloads: payload bytes use the envelope's base64 JSON representation and metadata
stays in the body, not protocol headers.

The context governs raw setup; canceling it later is not a substitute for calling
unsubscribe. NATS and MQTT raw callback bytes are borrowed/read-only: copy them
before retaining or mutating. RabbitMQ raw unsubscribe waits for its running
callback and channel-close RPC; callbacks must return, and a borrowed connection
needs owner-managed interruption if the RPC stalls.

MQTT unsubscribe is idempotent, returns its original cleanup result, and removes
only its registration. The broker filter remains until the last matching local
registration leaves. Subscription changes and reconnect restoration serialize so
old cleanup cannot remove a replacement registration.

Normal `Consume` validates envelope/routing identity before invoking facade
listeners. Redis checks the literal channel suffix, NATS the prefixed subject,
RabbitMQ the routing key, and MQTT the mapped topic. Malformed or mismatched
envelopes are dropped before facade processing telemetry. RabbitMQ drops remain
auto-acknowledged; MQTT follows its native acknowledgement path. These checks
prevent routing confusion, not unauthorized publication.

For the portable contracts, see the
[`broadcast` API](https://pkg.go.dev/arkoren.dev/core/service/broadcast).

---

Source: https://arkoren.dev/docs/guides/broadcasting/message-delivery.md
[HTML documentation](https://arkoren.dev/docs/guides/broadcasting/message-delivery)

# Message Delivery

A registered listener and a broker subscription have different lifetimes.
The consumer owns local listener routing; its driver owns receiving, retention,
and settlement. Understanding that boundary lets you stop a service without
mistaking cancellation for completed work, or a callback error for a retry.

This page assumes a configured [driver](https://arkoren.dev/docs/guides/broadcasting/drivers.md) and the
[`OrderCreated` topic](https://arkoren.dev/docs/guides/broadcasting/overview.md#defining-topics). Publication acceptance remains
separate from recipient handling: the publisher never receives listener errors.

## Owning Registrations

Use the unsubscribe function when a component no longer needs notifications.
With an existing consumer and application `search` indexer, retain the function
only after successful registration:

```go
unsubscribe, err := consumer.Listen(OrderCreatedTopic,
	func(ctx context.Context, order OrderCreated) error {
		return search.Index(ctx, order.OrderID)
	})
if err != nil {
	return err
}
```

The registration now belongs to your component. On its shutdown, remove it:

```go
unsubscribe()
unsubscribe()
```

Both calls are safe; only the first removes the registration. Removal affects
future delivery snapshots. A delivery that already captured the listener can
still invoke it, so unsubscribe does not join or cancel an in-flight callback.
Do not release callback dependencies until their active users have finished.

Registration and removal are concurrent-safe, and callbacks run outside the
registration lock. Each registration is independent, even if two use the same
topic or function. Removing one does not remove the others or close the driver's
subscription. Conversely, closing a transport does not remove the consumer's
local registrations.

## Stopping Consumers

Prefer an [application runner](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md) for a
service-wide consumer. If a smaller component directly owns consumption, give it
a cancelable context and retain the result so its owner can join the runner:

```go
import "context"

runCtx, cancel := context.WithCancel(ctx)
result := make(chan error, 1)
go func() {
	result <- consumer.Run(runCtx)
}()
```

This starts consumption, not a portable readiness handshake. Do not use goroutine
startup as proof that a non-retaining subscription can receive messages.
When the component ends, cancel and wait:

```go
import "errors"

cancel()
err := <-result
if !errors.Is(err, context.Canceled) {
	return err
}
```

The driver returns the context error when cancellation ends consumption;
`Run` preserves driver errors. An earlier transport failure can therefore be the
result instead. Listener failures are delivery outcomes and do not by themselves
end this loop.

Joining `Run` does not establish that every callback has finished. In particular,
memory deliveries already admitted retain context values without the consumer's
cancellation or deadline. NATS cancellation unsubscribes without waiting for a
running callback. Bound work inside the callback when its dependencies need
predictable cleanup:

```go
import "time"

unsubscribe, err := consumer.Listen(OrderCreatedTopic,
	func(ctx context.Context, order OrderCreated) error {
		work, cancel := context.WithTimeout(ctx, 5*time.Second)
		defer cancel()
		return search.Index(work, order.OrderID)
	})
```

The timeout only bounds operations that honor their context. If resource release
requires proof that callbacks ended, track and join those callbacks in the owning
component too. Driver `Stop` has provider-specific draining limits; follow
[driver ownership](https://arkoren.dev/docs/guides/broadcasting/drivers.md#driver-lifecycle) rather than assuming all drivers
have the same shutdown contract. NATS and MQTT connection shutdown does not end
their `Consume` context waits: cancel consumers separately.

## Listener Ordering

For one delivery, the facade takes a snapshot and invokes every matching listener
sequentially in registration order. An exact listener and a wildcard listener
can both match. A failure does not skip later listeners. A delivery with no
matching listeners succeeds; on SQS that permits deletion of the message.

Typed listeners decode the payload independently. A malformed payload can fail
one listener's decoding without preventing a different listener from running.
Topics sharing a name must still agree on their intended JSON contract; separate
decoding is not schema negotiation.

The facade joins all listener failures and returns the delivery result to the
driver. Listener panics are recovered as errors, including a stack trace.
One delivery's ordering does not order different messages: transports can invoke
several deliveries concurrently. Synchronize shared application state and do not
assume publication order is a portable callback-order guarantee.

Pattern listeners share the envelope's payload bytes and metadata map. Treat
them as read-only; copy data that needs mutation or independent ownership.

## Permanent Failures

If retrying the same input cannot help, mark that failure permanent. For example,
reject an order notification without its required identifier before calling the
indexer:

```go
import "errors"

unsubscribe, err := consumer.Listen(OrderCreatedTopic,
	func(ctx context.Context, order OrderCreated) error {
		if order.OrderID == "" {
			return broadcast.Permanent(errors.New("missing order identifier"))
		}
		return search.Index(ctx, order.OrderID)
	})
```

The returned error matches `broadcast.ErrPermanent` under `errors.Is` and retains
its original cause. `Permanent(nil)` returns nil. Invalid delivery names and
typed payload-decoding failures are permanent automatically.

A joined delivery failure is permanent only if every failed listener is
permanent. Successful listeners do not change that decision. If one failure is
transient and another permanent, the delivery remains retryable. A redelivery
runs every matching listener again, including listeners that previously succeeded.
Use stable application identifiers and idempotent effects; a transport-generated
ID is not a universal application deduplication key.

Permanent does not create retention, a retry policy, or a dead-letter destination.
Non-retaining drivers do not retry either error class. The SQS broadcast driver
deletes permanent failures rather than sending them to a dead-letter queue.

## Observing Failures

To observe callback failures independently of transport policy, give the consumer
your existing `*event.Bus` named `bus`:

```go
consumer := broadcast.NewConsumer(driver, func(receiver *broadcast.ConsumerBuilder) {
	receiver.Events(bus)
})
```

Attach an observer on that bus. With an application `logger`, record the routing
name without exporting the original error:

```go
unsubscribe, err := bus.Listen(broadcast.ListenerFailed,
	func(ctx context.Context, failure broadcast.ListenerFailedEvent) error {
		logger.WarnContext(ctx, "broadcast listener failed", "name", failure.Name)
		return nil
	})
if err != nil {
	return err
}
defer unsubscribe()
```

The topic is `broadcast.listener.failed`. Each failed listener emits one
`ListenerFailedEvent` in listener order, with `ID`, `Name`, and `Err`. ID is empty
when the transport supplies none. Decode failures and recovered panics also
emit events; successful deliveries emit none. Without `Events`, no bus
notification is published. Publishers and drivers do not emit this event.

The event omits message payload and metadata, but `Err` retains application
causes and recovered panic details. Treat it as sensitive diagnostic data, not
safe external output. Observing the event cannot change delivery settlement,
and it is not evidence of broker retention or acknowledgement. See
[built-in service events](https://arkoren.dev/docs/guides/events/built-in-events.md#broadcasting) and
[monitoring privacy](https://arkoren.dev/docs/guides/broadcasting/metadata-and-telemetry.md#protecting-data).

## Transport Outcomes

Memory, Redis Pub/Sub, and Core NATS do not retry returned listener errors or
retain offline notifications. Memory and NATS log handler failures; Redis does
not turn those failures into retries. Subscribers must recover missed state
another way, including after a process restart.

RabbitMQ uses auto-acknowledged, ephemeral consumer queues. Listener failure does
not requeue a notification. Publication retries after a lost broker confirmation
can nevertheless duplicate it. MQTT's owned clean sessions likewise have no
offline retention; callback errors do not request retries or change native
acknowledgement. Protocol QoS does not guarantee exactly-once effects.

SQS deletes successfully handled messages, malformed envelopes, and deliveries
whose failures are all permanent. Other failures remain for redelivery after
visibility expires, subject to queue retention and redrive policies. Deletion
failure can repeat an already successful effect. The driver does not renew
visibility; size the timeout for the entire sequential batch. SNS acceptance
does not prove delivery to every subscribed queue.

For complete transport configuration and cancellation limits, use the
[driver comparison](https://arkoren.dev/docs/guides/broadcasting/drivers.md#compatibility-matrix). For every facade function,
see the [`broadcast` API](https://pkg.go.dev/arkoren.dev/core/service/broadcast).

---

Source: https://arkoren.dev/docs/guides/broadcasting/metadata-and-telemetry.md
[HTML documentation](https://arkoren.dev/docs/guides/broadcasting/metadata-and-telemetry)

# Monitoring

An order request can publish one notification that several services handle
later. Trace propagation connects those deliveries to the publication; broadcast
spans and metrics show transport calls and listener execution. They do not prove
that every intended recipient received the notification.

Broadcasting carries trace context in **metadata**, a string map beside the
application payload. Telemetry can record routing names and error classes
without recording payloads, metadata values, or error text. Broker readers can
still read the message body, including that metadata.

## Connecting Telemetry

By default, publishers and consumers resolve the
[shared telemetry handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle) at each
operation. Its initial no-op records and propagates nothing. If your application
configures that shared handle, facade construction needs no telemetry override.

For an explicit application handle, configure both sides. With an existing
`app` and configured broadcast `driver`:

```go
import "arkoren.dev/core/service/broadcast"

publisher := broadcast.NewPublisher(driver, func(sender *broadcast.PublisherBuilder) {
	sender.Telemetry(app.Telemetry())
})
consumer := broadcast.NewConsumer(driver, func(receiver *broadcast.ConsumerBuilder) {
	receiver.Telemetry(app.Telemetry())
})
```

The publisher injects context using the telemetry driver's propagator; the
consumer extracts it before running listeners. Both sides need compatible
propagation. Export configuration belongs in the
[telemetry guide](https://arkoren.dev/docs/guides/observability/telemetry.md). Instrumentation does not add
retention, retries, or different listener routing.

## Message Metadata

The typed `Publisher.Publish` operation has no metadata parameter. It allocates
a fresh propagation map when enabled telemetry injects values, such as W3C
`traceparent` and `tracestate`. Typed listeners receive only their decoded
payload; pattern listeners can inspect the whole envelope.

If you integrate directly with a driver, you can supply application metadata
yourself. This intentionally bypasses facade JSON encoding and producer telemetry.
With a configured `driver`, `ctx`, and `broadcast` imported, send valid payload
JSON plus a nonsecret correlation identifier:

```go
message := broadcast.Message{
	Name:    "order.created",
	Payload: []byte(`{"order_id":"o-1042"}`),
	Metadata: map[string]string{
		"correlation-id": "order-import-7",
	},
}
err := driver.Publish(ctx, message)
```

A receiving pattern listener sees `message.Metadata["correlation-id"]` as
`order-import-7`. Metadata does not select listeners or authorize the message.
To observe that field with an existing application `logger`, allowlist a
nonsecret value and treat values from external publishers as untrusted:

```go
unsubscribe, err := consumer.ListenPattern("order.#",
	func(ctx context.Context, message broadcast.Message) error {
		logger.InfoContext(ctx, "order notification",
			"correlation", message.Metadata["correlation-id"])
		return nil
	})
if err != nil {
	return err
}
defer unsubscribe()
```

Missing metadata permits map lookup and yields an empty string. Do not log a
field merely because it is metadata. Keep payload bytes and the metadata map
read-only because matching listeners share them.

Drivers preserve metadata within this portable round-trip guarantee:

| Limit | Value |
| --- | --- |
| Entries | At most 16. |
| Key | 1–64 bytes from `A-Za-z0-9._-`, case-sensitive. |
| Value | Valid UTF-8. |
| Combined Size | At most 4096 bytes of keys and values together. |

The constants are `MaxMetadataEntries`, `MaxMetadataKeyLength`, and
`MaxMetadataSize` in `broadcast`. Beyond these limits a driver may reject a
message. Nil and empty metadata are equivalent. A message published without
metadata arrives with a nil map; existing messages without metadata remain
decodable.

External drivers carry metadata in the JSON envelope body, not MQTT user
properties, AMQP headers, or SNS propagation attributes. Those protocols' header
limits therefore do not define this map's limits; broker message-size limits
still apply.

## Reading Spans

A publish that reaches the driver records producer span `send order.created`.
A valid message routed by the consumer records `process order.created` around
all matching listeners. When propagation is present and extracted, the process
span is a child of the send span in the same trace. Listener contexts carry that
span, allowing context-aware logs and downstream calls to join the trace.

Three independent recipients can therefore produce three child process spans.
Redelivery adds more process spans; a process span is not an exactly-once
receipt. A valid unmatched delivery also records processing even though no
listener effect occurs.

| Attribute | Meaning |
| --- | --- |
| `messaging.system` | Driver system, or an explicit override. |
| `messaging.operation.name` | `send` or `process`. |
| `messaging.operation.type` | `send` or `process`; spans only. |
| `messaging.destination.name` | Logical broadcast name. |
| `messaging.message.id` | Nonempty delivery ID; spans only. |
| `error.type` | Error class on failure. |

Driver systems are `memory`, `redis`, `nats`, `rabbitmq`, `mqtt`, `aws_sns`, and
`aws_sqs`. SNS publication followed by SQS consumption correctly shows different
systems on the two sides. `PublisherBuilder.System` and `ConsumerBuilder.System`
override the driver name. If neither supplies one, the value is `_OTHER`.
Broadcast spans do not include queue job or attempt attributes.

Invalid publication names and payload-encoding errors occur before send
instrumentation. Invalid delivery names, or broker-driver rejection of malformed
or routing-mismatched envelopes, occur before facade process instrumentation.
They produce no corresponding facade span or metric. Use provider diagnostics
for transport drops; absence of a process span alone does not identify the cause.

## Reading Metrics

All four instruments use scope `arkoren.dev/core/service/broadcast`, distinct
from queues even though their metric names are shared:

| Metric | What It Measures |
| --- | --- |
| `messaging.client.sent.messages` | Publish attempts that reach the driver, including failures. |
| `messaging.client.consumed.messages` | Valid routed deliveries, including failures and repeats. |
| `messaging.client.operation.duration` | Publish-call duration, in seconds. |
| `messaging.process.duration` | All matching listeners' processing duration per delivery, in seconds. |

Metric attributes are `messaging.system`, `messaging.operation.name`,
`messaging.destination.name`, and `error.type` on failure. Counts are neither
unique application events nor successful recipient effects. One facade call
with an internal RabbitMQ publication retry still produces one send observation.

A failed publish or listener delivery marks its span as erroneous and records
an error class, never the error text. Cancellation and deadline errors use
`context.Canceled` and `context.DeadlineExceeded`. Otherwise classification follows
the single-cause unwrap chain; a multi-cause error has no single root and uses
its aggregate type. [Listener failure events](https://arkoren.dev/docs/guides/broadcasting/message-delivery.md#observing-failures)
provide original errors to your application when you need sensitive diagnostics.

## Protecting Data

Recorded broadcast names, systems, durations, error classes, and optional message
IDs are still operational data. Keep names stable and nonsecret: do not embed
customer identifiers or credentials in routing names. Many distinct names also
create metric cardinality; use
[bounded metric retention](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md#prometheus) when
publishers can generate many destinations.

The facade does not record payloads, metadata values, error messages, or panic
stack traces into its telemetry. That boundary does not sanitize your logs,
event observers, native SDK errors, or custom spans. Nor does it encrypt trace
context inside broker messages. Restrict broker access and use transport
encryption; never put credentials in metadata. See
[Telemetry Privacy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md) for deployment-wide
controls.

## Testing Signals

Use `telemetrytest` with the broadcast recording fake to check instrumentation
without a broker. In a test with `t *testing.T`, import the two fake packages
and configure both facades with the same recorder:

```go
import (
	"arkoren.dev/core/service/broadcast/broadcasttest"
	"arkoren.dev/core/service/telemetry/telemetrytest"
)

signals := telemetrytest.New()
fake := broadcasttest.New()
recorder := signals.Telemetry(t)
publisher := broadcast.NewPublisher(fake, func(sender *broadcast.PublisherBuilder) {
	sender.Telemetry(recorder)
})
consumer := broadcast.NewConsumer(fake, func(receiver *broadcast.ConsumerBuilder) {
	receiver.Telemetry(recorder)
})
```

With the [order topic](https://arkoren.dev/docs/guides/broadcasting/overview.md#defining-topics), publish and route one
notification, then assert the two observations:

```go
if err := publisher.Publish(t.Context(), OrderCreatedTopic, OrderCreated{OrderID: "o-1042"}); err != nil {
	t.Fatal(err)
}
if err := fake.Drain(t.Context(), consumer); err != nil {
	t.Fatal(err)
}
signals.AssertSpan(t, "send order.created")
signals.AssertSpan(t, "process order.created")
```

No listener is needed to record a valid routed delivery; register one separately
when checking its effect. The fake has no system name, so its spans report
`_OTHER` unless overridden. These assertions prove instrumentation ran, not
network propagation, parentage, or recipient success. See
[Capture Telemetry](https://arkoren.dev/docs/guides/testing/fakes.md#capture-telemetry) for inspecting recorded
spans and metrics more precisely.

For every function and option, see the
[`broadcast` API](https://pkg.go.dev/arkoren.dev/core/service/broadcast) and
[`telemetry` API](https://pkg.go.dev/arkoren.dev/core/service/telemetry).

---

Source: https://arkoren.dev/docs/guides/mail/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/mail/overview)

# Mail Overview

A welcome email or receipt starts with application data and ends with a request
to submit mail. Arkoren separates those responsibilities: an immutable
`mail.Message` describes the email, and a `*mail.Mailer` applies sender defaults,
validates it, and passes it to a driver.

The mailer sends one message synchronously per call. It does not render templates,
queue work, retry, or track inbox delivery. Choose and construct your backend on
[Drivers](https://arkoren.dev/docs/guides/mail/drivers.md); use a [queue job](https://arkoren.dev/docs/guides/queues/producing-jobs.md) when submission
should happen outside the request that triggered it.

## Sending Mail

Assume a configured `*mail.Mailer` named `mailer` with a
[default sender](https://arkoren.dev/docs/guides/mail/configuration.md), and a nonnil request or job context named
`ctx`. Import `arkoren.dev/core/service/mail`. For a newly created account,
use the recipient data your application has already validated and authorized:

```go
type WelcomeEmail struct {
	Name  string
	Email string
}

welcome := WelcomeEmail{Name: "Ada", Email: "ada@example.com"}
```

Construct the recipient list and body together, then add the optional subject:

```go
message := mail.NewMessage(mail.Recipients{
	To: []mail.Address{{Name: welcome.Name, Email: welcome.Email}},
}, mail.Body{Text: "Your account is ready."}, func(letter *mail.MessageBuilder) {
	letter.Subject("Welcome to Acme")
})
```

At least one address across `To`, `CC`, and `BCC`, and at least one nonempty text
or HTML body, are required. The omitted sender comes from the mailer. The zero
`mail.Message` is invalid; use `NewMessage` rather than a struct literal.

Submit that value through the facade:

```go
receipt, err := mailer.Send(ctx, message)
```

On success, `receipt` describes driver acceptance. `Send` fills missing sender
and reply-to defaults on a local copy; your original message stays unchanged.
Focused examples leave error results visible and omit routine propagation;
handle those errors in your application before using successful results.
For HTML, recipient groups, and files, continue with
[Composing Messages](https://arkoren.dev/docs/guides/mail/composing-messages.md).

## Submission Outcomes

`receipt.ID` is an opaque submission identifier, not a delivery status or retry
idempotency key. SMTP and Log return generated message IDs; API drivers return
provider identifiers. The portable contract permits an empty ID. Store an
available identifier with your application's mail record when you need support
correlation, under an appropriate access and retention policy.

A sending server can accept mail that later bounces, is deferred, or lands in
spam. The Log driver simulates acceptance without sending anything. Optional
[provider notifications](https://arkoren.dev/docs/guides/mail/drivers.md#provider-notifications) can report later
outcomes, but constructing an outbound mailer does not register a receiver.
Even a delivery notification is not proof that a person read the message.

## Handling Failures

Invalid messages fail before driver submission with an error wrapping
`mail.ErrInvalidMessage`. A driver's preflight also rejects an already canceled
or expired context before I/O. Native failures retain their wrapped causes;
use `errors.Is` or `errors.As`, not diagnostic-text matching. Returned errors
can contain recipients or provider diagnostics: redact them before logging or
forwarding them.

With standard-library imports `context` and `errors`, distinguish malformed
input from an interrupted submission:

```go
_, err := mailer.Send(ctx, message)
invalid := errors.Is(err, mail.ErrInvalidMessage)
interrupted := errors.Is(err, context.Canceled) ||
	errors.Is(err, context.DeadlineExceeded)
```

If `invalid` is true, correct the message or sender configuration rather than
retrying unchanged input. If `interrupted` is true, stop this attempt; after I/O
starts, acceptance can remain unknown. Other failures need provider-specific
classification, including credential and recipient rejections.

Neither the facade nor its drivers add a retry loop, although native SDK or
transport policies can retry. A timeout can follow remote acceptance, so another
attempt can send duplicate mail. There is no portable idempotency key or
exactly-once guarantee; an application record alone cannot atomically commit
both a remote submission and a local update.

A driver panic propagates unchanged without a `mail.Sent` or `mail.Failed`
event. Absence of an outcome event does not prove nonacceptance. See
[Events and Telemetry](https://arkoren.dev/docs/guides/mail/events-and-telemetry.md) for the signals that remain.
To assert application behavior without sending mail, use the
[mail fake](https://arkoren.dev/docs/guides/testing/fakes.md#assert-mail) and
[failure injection](https://arkoren.dev/docs/guides/testing/fakes.md#inject-failures).

For every function and option, see the
[`mail` API reference](https://pkg.go.dev/arkoren.dev/core/service/mail).

---

Source: https://arkoren.dev/docs/guides/mail/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/mail/drivers)

# Mail Drivers

Choose Log when development mail must not leave the process, SMTP for a relay or
local inbox, or an API driver for an existing Resend, Mailgun, Postmark, or
Amazon SES account. Provider credentials, verified senders, and account sending
limits remain provider-specific even though your application sends the same
immutable `mail.Message`.

## Compatibility Matrix

This compares Arkoren's shipped mappings and companion adapters, not the whole
vendor product or verified inbox delivery. Every driver accepts text-only,
HTML-only, alternative bodies, and To/CC/BCC groups, including BCC-only mail.
Log records those fields without rendering or sending email.

| Driver | Outbound Submission | Regular Attachments | Inline Attachments | Notification Webhooks | Inbound Email |
| --- | --- | --- | --- | --- | --- |
| [Log](#log) | No | Partial — Logs metadata, never attachment bytes. | No | No | No |
| [SMTP](#smtp) | Yes | Yes — MIME preserves media-type parameters. | Yes | No | No |
| [Resend](#resend) | Yes | Yes | Yes | Yes | Partial — Metadata only; fetch content separately. |
| [Mailgun](#mailgun) | Yes | Partial — SDK selects media types. | Partial — Content ID must equal filename; SDK selects media types. | Yes | No |
| [Postmark](#postmark) | Yes | Yes | Yes — Adds cid: only in the native API payload. | Yes | Partial — Typed bodies; attachments need a custom decoder. |
| [SES](#amazon-ses) | Yes | Yes — MIME preserves media-type parameters. | Yes | No | No |

SMTP and SES use the shared [MIME encoder](https://arkoren.dev/docs/guides/mail/composing-messages.md#mime-encoding).
Resend and Postmark forward attachment media types to their APIs; Mailgun's SDK
selects them instead. Provider size limits still apply to encoded submissions.

## Connecting Mailers

Run each `go get` command from your application's module directory. Load secrets
at startup from your environment or [secret store](https://arkoren.dev/docs/guides/secrets/overview.md), not
committed source. Examples use `os.Getenv` from the standard-library `os` package.
Check a driver's construction error before wrapping it; construction does not
verify remote sending permissions.

With import `arkoren.dev/core/service/mail`, connect your successfully constructed
driver to a sender verified with the chosen provider:

```go
mailer := mail.New(driver, func(sender *mail.Builder) {
	sender.From(mail.Address{Name: "Acme", Email: "notifications@acme.test"})
})
```

Use the [welcome message](https://arkoren.dev/docs/guides/mail/overview.md#sending-mail) with this mailer and a nonnil
send context. `mail.New` returns no error; invalid sender defaults are detected
when sending. The facade and these drivers have no application lifecycle hooks.
SMTP connects per send; API requests happen at submission. SES construction loads
AWS configuration but does not call SES. Configure shared clients before concurrent
use and leave native resource cleanup with their owner.

Set a send-context deadline and a deliberate retry policy. Arkoren adds no retry
loop, but native SDKs or transports can retry, including AWS SDK defaults.
Interrupted submissions can have unknown acceptance; retrying can duplicate mail.
Direct `driver.Send` bypasses facade defaults, events, and telemetry, so the
message must already contain a valid sender.

## Provider Notifications

Outbound drivers do not register webhook receivers. The optional companion
examples below assume import `arkoren.dev/core/http/webhook`, an existing
`*router.Router` named `r` from `arkoren.dev/core/http/router`, separately loaded
receiver credentials, and typed application-owned acceptance functions. The
fragments run in a function returning `error`; both construction errors must be
checked before mounting a handler.

An acceptance function receives `context.Context` and `webhook.Delivery[T]` and
returns `error`; store or queue the verified notification according to your
application's durability requirements. Configure the mounted route's public
HTTPS URL in the provider dashboard and enable the events you bind. Receiver
construction alone does not configure the provider.

Use the canonical [provider adapter guide](https://arkoren.dev/docs/guides/webhooks/provider-adapters.md)
for verification boundaries and [Delivery Policies](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md)
for receipt identity, retention, queued acceptance, body limits, and responses.
By default, successful acceptance answers 200, rejected authentication 400, and
ordinary authenticated envelope/decode/acceptance failures 503. Unknown
authenticated event types answer 200 without event-specific decoding or a
callback. These acknowledgements concern your receiver, not delivery of the
original email; they neither guarantee provider retries nor exactly-once effects.

## Log

Log is bundled in `arkoren.dev/core/service/mail/slogmail`. It simulates
acceptance with a generated message ID and a log record, not external delivery.
For assertions rather than inspection, use the [mail fake](https://arkoren.dev/docs/guides/testing/fakes.md#assert-mail).

Even default summary logs disclose sender, To/CC/BCC, reply-to, subject, body
sizes, and attachment names, media types, and sizes. Use synthetic local data;
BCC privacy in delivery headers does not hide BCC from these logs.

```go
import "arkoren.dev/core/service/mail/slogmail"

driver := slogmail.New()
```

Wrap it as shown in [Connecting Mailers](#connecting-mailers). It uses
`slog.Default()` at info level. A successful send can produce no visible record
if the logger filters that level. Attachment bytes are never logged.

### Log Detail

If local debugging requires body inspection, pass an existing `*slog.Logger`
named `logger` and import `log/slog`. Body logging can expose reset links and
other credentials; enable it only for safe fixture messages:

```go
driver := slogmail.New(func(output *slogmail.Builder) {
	output.Logger(logger)
	output.Level(slog.LevelDebug)
	output.Detail(slogmail.DetailBody)
})
```

This alternative construction adds text and HTML bodies to the summary.
Omitting `Detail`, or selecting `DetailSummary`, leaves bodies out.

See the [`slogmail` API](https://pkg.go.dev/arkoren.dev/core/service/mail/slogmail).

## SMTP

Choose SMTP when you have relay access and know the server's host, port,
encryption mode, and any credentials. The standard-library transport renders
MIME and keeps BCC destinations in the envelope, not visible headers.

```sh
go get arkoren.dev/driver/smtp
```

### Relay Connection

Port 587 commonly uses STARTTLS, the required default. Construction validates
settings without connecting:

```go
import "arkoren.dev/driver/smtp/smtpmail"

driver, err := smtpmail.New("smtp.example.com", 587, func(transport *smtpmail.Builder) {
	transport.Username(os.Getenv("SMTP_USER"))
	transport.Password(os.Getenv("SMTP_PASSWORD"))
})
```

An empty host, zero port, unknown TLS mode, incomplete credential pair, or
credentials with plaintext transport fails construction. Empty username and
password together mean no authentication. Wrap the checked driver and send the
[welcome message](https://arkoren.dev/docs/guides/mail/overview.md#sending-mail).

### Transport Policy

If the relay requires TLS from the first byte, commonly on port 465, use a
separate implicit-TLS construction. Import `time` for a shorter exchange bound:

```go
driver, err := smtpmail.New("smtp.example.com", 465, func(transport *smtpmail.Builder) {
	transport.TLSMode(smtpmail.TLSModeImplicit)
	transport.Username(os.Getenv("SMTP_USER"))
	transport.Password(os.Getenv("SMTP_PASSWORD"))
	transport.Timeout(10 * time.Second)
})
```

Certificates are verified against the host with TLS 1.2 minimum. Credentials use
PLAIN authentication over encryption. Required STARTTLS fails if no upgrade is
offered. `TLSModeNone` is an explicit plaintext choice only for trusted local,
unauthenticated servers; the constructor rejects credentials in that mode.

The default timeout is 30 seconds for dialing and the entire SMTP exchange.
Zero restores it; negative disables that bound, not context cancellation or an
earlier context deadline. Cancellation interrupts greeting, TLS negotiation,
authentication, envelope commands, and DATA reads or writes.

### Submission Limits

MIME encoding and its line/metadata checks happen before connection. If a relay
rejects a recipient, the error can name that address; the driver submits no DATA
for the other recipients. A successful receipt is the generated Message-ID,
not a relay-assigned delivery ID; correlate it with server logs.

Each concurrent send owns a separate connection and handshake. Cancellation
while waiting for the final DATA acknowledgement returns an empty receipt and
an error matching the context error, but acceptance can already have happened.
After DATA acknowledgement, a failed or canceled QUIT does not revoke success.
Respect relay rate limits rather than retrying every failure.

SMTP driver values print as fixed type labels through `fmt`, including copied
values. Its `Config` formatting redacts a nonempty password, but leaves username
and other fields visible. Neither protection sanitizes returned errors, messages,
or arbitrary serialization.

### Local Inbox

With Docker installed, run a disposable Mailpit inbox bound only to loopback.
It is an unauthenticated relay; do not expose these ports publicly:

```sh
docker run --detach --rm --name acme-mailpit --publish 127.0.0.1:1025:1025 --publish 127.0.0.1:8025:8025 axllent/mailpit:latest
```

Connect without credentials or TLS:

```go
driver, err := smtpmail.New("localhost", 1025, func(transport *smtpmail.Builder) {
	transport.TLSMode(smtpmail.TLSModeNone)
})
```

Check the error, wrap it, and send the shared welcome message. Open
`http://localhost:8025` to check its subject and recipient. This proves local
capture, not internet delivery. `docker stop acme-mailpit` stops the disposable
container and removes its captured messages.

See the [`smtpmail` API](https://pkg.go.dev/arkoren.dev/driver/smtp/smtpmail).

## Resend

Choose Resend with an account, sending API key, and verified sending domain.
Its SDK maps bodies, recipient groups, reply-to, and regular or inline attachments.

```sh
go get arkoren.dev/driver/resend
```

### API Connection

The key is the only driver setting; an empty key fails construction:

```go
import "arkoren.dev/driver/resend/resendmail"

driver, err := resendmail.New(os.Getenv("RESEND_API_KEY"))
```

Wrap the checked driver and send the [welcome message](https://arkoren.dev/docs/guides/mail/overview.md#sending-mail).
The receipt ID is Resend's email identifier. The driver adds no timeout of its
own; bound sends with your context.

### Native Client

If you need HTTP transport or timeout control, configure the SDK first. With
imports `net/http`, `time`, and `github.com/resend/resend-go/v3`, and your loaded
`apiKey`:

```go
client := resend.NewCustomClient(&http.Client{Timeout: 10 * time.Second}, apiKey)
driver := resendmail.NewFrom(client)
```

`NewFrom` performs no validation; supply a nonnil initialized client. It borrows
the client and transport without copying, closing, or reconfiguring them.
`driver.Client()` returns that live SDK client. Configure it before concurrent
use; native sends bypass facade observations and validation.

If you hold only the facade, `mailer.DriverAs[*resendmail.Driver]()` returns
the first matching driver through wrappers, plus a boolean indicating a match.
Check that boolean before using `Client()`; inspection neither removes wrappers
nor transfers ownership.

### Receiving Resend Notifications

Use the optional `resendwebhook` companion from the same module for delivery,
bounce, complaint, sent, or received-email events. Follow the shared
[receiver setup](#provider-notifications). The endpoint's `whsec_` signing secret
is not the sending API key. Its suffix must be nonempty unbroken standard base64;
malformed or line-broken input fails construction with `webhook.ErrInvalid`
without exposing or repairing its bytes.

Import `arkoren.dev/driver/resend/resendwebhook`. With application-owned
`accept` of type `webhook.AcceptFunc[resendwebhook.Email]`, create the source:

```go
source, err := resendwebhook.New(signingSecret)
if err != nil {
	return err
}
```

Bind delivery events, check the receiver, then mount it on the existing router `r`:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[resendwebhook.Envelope]) {
	receiver.Handle(resendwebhook.EmailDelivered, accept)
})
if err != nil {
	return err
}
r.Post("/webhooks/resend", handler)
```

Svix v1 signatures cover the exact raw body and require an attempt timestamp
within five minutes in either direction. Synchronize clocks. Missing, invalid,
stale, future, or duplicate authentication header lines fail closed; signature
errors match both `resendwebhook.ErrInvalidSignature` and `webhook.ErrRejected`.
The delivery identity is `svix-id`, not the email ID. Replays within the window
still require [receipt policy](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md#duplicate-receipts)
and idempotent effects.

For inbound metadata, configure a receiving domain and enable `email.received`
in Resend, then bind `resendwebhook.EmailReceived` to a compatible acceptance
function instead. It provides metadata, not body or attachment content; fetch
those separately through the provider API. Sender claims and fetched content
remain untrusted. Bounce and SMTP diagnostics can contain personal data.

See the [Resend mail API](https://pkg.go.dev/arkoren.dev/driver/resend/resendmail)
and [Resend webhook API](https://pkg.go.dev/arkoren.dev/driver/resend/resendwebhook).

## Mailgun

Choose Mailgun with a verified sending domain and private API key. Its SDK uses
the Messages API; account region determines the endpoint.

```sh
go get arkoren.dev/driver/mailgun
```

### API Connection

US-region accounts use the default endpoint:

```go
import "arkoren.dev/driver/mailgun/mailgunmail"

driver, err := mailgunmail.New("mg.acme.test", os.Getenv("MAILGUN_API_KEY"))
```

Both domain and key are required. For an EU account, use this alternative
construction before wrapping the driver:

```go
driver, err := mailgunmail.New("mg.acme.test", os.Getenv("MAILGUN_API_KEY"),
	func(transport *mailgunmail.Builder) {
		transport.Endpoint("https://api.eu.mailgun.net")
	})
```

Supply a trusted absolute API base URL without a version suffix such as `/v3`.
The SDK rejects version-path segments at construction but does not fully validate
URL syntax; other malformed URLs can fail at send time.

For a preconfigured `*mailgun.Client` from `github.com/mailgun/mailgun-go/v5`,
use `driver, err := mailgunmail.NewFrom(client, domain)`. The client carries key
and endpoint but not the sending domain. Nil client or empty domain fails
construction. The driver borrows the client without copying or closing it;
`driver.Client()` exposes it. Configure HTTP and retry policies before sharing
it, and bound sends with a context deadline.

### Attachment Mapping

Use the [welcome send](https://arkoren.dev/docs/guides/mail/overview.md#sending-mail) or the
[inline image example](https://arkoren.dev/docs/guides/mail/composing-messages.md#inline-images). Mailgun requires
`ContentID` to equal `Filename`, such as `logo.png`, with HTML `cid:logo.png`.
A mismatch fails before submission. The SDK chooses attachment media types;
it does not preserve `Attachment.ContentType`. The receipt is Mailgun's response ID.

### Receiving Mailgun Notifications

Use the same module's `mailgunwebhook` for event webhooks, not inbound Routes
forms or Alerts. Prepare the account's HTTP webhook signing key, not the sending
API key. Empty keys, whitespace, or control characters fail with
`webhook.ErrInvalid`; keys are not trimmed, repaired, or quoted in errors.

Mailgun signs only `timestamp + token`, **not event data**. Serve HTTPS and apply
provider IP restrictions at a trusted edge where possible. A captured valid token
can accompany altered event data within the freshness window; the HMAC is not
proof of payload integrity.

Import `arkoren.dev/driver/mailgun/mailgunwebhook` and
`github.com/mailgun/mailgun-go/v5/events`. With `accept` of type
`webhook.AcceptFunc[events.Delivered]`, create the source:

```go
source, err := mailgunwebhook.New(signingKey)
if err != nil {
	return err
}
```

Bind it using the [shared receiver prerequisites](#provider-notifications):

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[mailgunwebhook.Envelope]) {
	receiver.Handle(mailgunwebhook.Delivered, accept)
})
if err != nil {
	return err
}
r.Post("/webhooks/mailgun", handler)
```

The companion covers accepted, delivered, failed, opened, clicked, unsubscribed,
and complained events. For `Failed`, inspect native `Severity`; temporary and
permanent failures both use key `failed`, not dashboard name `permanent_fail`.
Delivery IDs combine the event's UTC date and ID because IDs are unique only
within a day.

Signature freshness defaults to thirty minutes in either direction, separately
from event age. If delayed attempts need a wider window, construct the source
with a callback calling `signature.Tolerance(time.Hour)` on
`*mailgunwebhook.Builder` (`time` is the standard-library package). Zero restores
thirty minutes; negative fails construction. Subaccounts need their own signing
key; `parent-signature` is not substituted. There is no token cache.

See the [Mailgun mail API](https://pkg.go.dev/arkoren.dev/driver/mailgun/mailgunmail)
and [Mailgun webhook API](https://pkg.go.dev/arkoren.dev/driver/mailgun/mailgunwebhook).

## Postmark

Choose Postmark with a server token and confirmed sender signature or domain.
The driver calls its HTTP API directly, without an official Go SDK.

```sh
go get arkoren.dev/driver/postmark
```

### API Connection

An empty server token fails construction; remote permissions are checked only
when submitting:

```go
import "arkoren.dev/driver/postmark/postmarkmail"

driver, err := postmarkmail.New(os.Getenv("POSTMARK_TOKEN"))
```

Wrap it and use the [welcome send](https://arkoren.dev/docs/guides/mail/overview.md#sending-mail). The receipt contains
Postmark's `MessageID`. For [inline images](https://arkoren.dev/docs/guides/mail/composing-messages.md#inline-images),
keep the portable `ContentID` bare. The driver adds `cid:` only in its API
payload; it does not change the immutable message or prove rendering.

### HTTP Policy

The default HTTP client has a 30-second timeout. To supply a shorter bound or
an instrumented transport, import `net/http` and `time` and use the loaded `token`:

```go
client := &http.Client{Timeout: 10 * time.Second}
driver, err := postmarkmail.New(token, func(transport *postmarkmail.Builder) {
	transport.HTTPClient(client)
})
```

The supplied client is borrowed. Each send shallow-copies its current settings
and replaces only the copy's redirect policy: every redirect is rejected,
including same-origin redirects, without invoking your `CheckRedirect` callback.
The original client is not changed or closed; its transport and cookie jar remain
shared and caller-owned. Configure them before concurrent use and trust the
transport with the token and message. Nil `HTTPClient` restores a fresh default
client. The driver has no `NewFrom` or client accessor.

`Builder.Endpoint` can replace `https://api.postmarkapp.com/email`; empty restores
it. Only use a trusted final URL: the token and message go directly there, and
redirect rejection does not make arbitrary endpoints safe. Malformed URL errors
are deferred until sending. Use synthetic credentials and messages for local
HTTP fixtures.

Responses with status 300 or higher return an empty receipt and an error with
the status and up to 4096 response-body bytes. Redirects fail without forwarding
the token or replaying the message through HTTP redirection. Redact returned
diagnostics before logging. Driver values format as fixed type labels through
`fmt`; that does not sanitize clients, errors, messages, or serialization.

### Receiving Postmark Notifications

Configure separate strong Basic Auth credentials on the Postmark webhook URL,
not the sending token. Use HTTPS, protect any proxy-to-app hop, and apply provider
IP restrictions at a trusted edge where possible. Postmark does not sign these
payloads; anyone holding the credentials can forge or replay notifications.

Import `arkoren.dev/driver/postmark/postmarkwebhook`. With `accept` of type
`webhook.AcceptFunc[postmarkwebhook.Delivery]`, create the source:

```go
source, err := postmarkwebhook.New(username, password)
if err != nil {
	return err
}
```

Bind delivery notifications and mount the receiver:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[postmarkwebhook.Envelope]) {
	receiver.Handle(postmarkwebhook.Delivered, accept)
})
if err != nil {
	return err
}
r.Post("/webhooks/postmark", handler)
```

The companion also covers bounce, spam complaint, open, click, and subscription
changes. Delivery reports receiving-server acceptance, not inbox placement;
opens and clicks can come from automated clients. Inspect `Bounce.Inactive`
before treating a recipient as deactivated and `Subscription.SuppressSending`
for the resulting suppression state.

Outbound identity uses `X-PM-Webhook-Trace-Id`. Without it the ID is empty and
deduplication is disabled, rather than merging distinct events sharing a message
ID. Both source constructors reject empty credentials, control characters, or
a colon in the username with `webhook.ErrInvalid`. Valid credential bytes are
retained without trimming and excluded from construction errors.

### Receiving Inbound Email

Configure Postmark's inbound stream/domain to POST to a separate HTTPS endpoint
with separate Basic Auth credentials. With `acceptEmail` of type
`webhook.AcceptFunc[postmarkwebhook.Email]`, create its source:

```go
source, err := postmarkwebhook.NewInbound(inboundUser, inboundPassword)
if err != nil {
	return err
}
```

Bind the parsed inbound message. This example raises the body allowance to
10 MiB; choose a limit within your own resource budget:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[postmarkwebhook.Envelope]) {
	receiver.Handle(postmarkwebhook.Inbound, acceptEmail)
	receiver.MaxBodySize(10 << 20)
})
if err != nil {
	return err
}
r.Post("/webhooks/postmark/inbound", handler)
```

Inbound JSON has no `RecordType`; the adapter assigns local key `Inbound` and
uses the received `MessageID` as delivery identity. Scope receipt storage
separately per endpoint. The typed payload includes text, HTML, extracted reply
text, and the sender's variable-format `Date` string. Omitted or nullable fields
retain zero values; validate business-required fields yourself. For attachments
or full address structures, define a custom event decoder over `Envelope.Body`.

Endpoint authentication does not authenticate the claimed sender as an
application user. Sanitize HTML before rendering and scan attachments before
use. Align the [body limit](https://arkoren.dev/docs/guides/webhooks/delivery-policies.md#body-limits) with
ingress and queue limits; raising this setting cannot override a smaller outer
HTTP limit.

See the [Postmark mail API](https://pkg.go.dev/arkoren.dev/driver/postmark/postmarkmail)
and [Postmark webhook API](https://pkg.go.dev/arkoren.dev/driver/postmark/postmarkwebhook).

## Amazon SES

Choose SES with an AWS account, region, verified SES identity, and IAM permission
`ses:SendEmail` for that identity. The driver sends raw MIME through SES v2.
Sandbox accounts can send only to verified recipients; request production access
before relying on customer addresses.

```sh
go get arkoren.dev/driver/aws
```

### AWS Connection

Use an existing startup context `ctx` while loading the standard AWS
configuration and credential chain:

```go
import "arkoren.dev/driver/aws/awsmail"

driver, err := awsmail.New(ctx, func(transport *awsmail.Builder) {
	transport.Region("eu-west-1")
})
```

Check configuration-loading errors before creating the facade. Environment,
shared profiles, and instance/workload roles follow AWS resolution. `Region` is
the only driver setting; omitting it leaves region resolution to the chain.
Construction does not call SES or confirm permission to send.

### Native Client

For custom credentials, endpoint, HTTP policy, or retries, configure an SDK client
first. With `config` of type `aws.Config`, imports `net/http`, `time`, and
`github.com/aws/aws-sdk-go-v2/service/sesv2`:

```go
client := sesv2.NewFromConfig(config, func(options *sesv2.Options) {
	options.HTTPClient = &http.Client{Timeout: 10 * time.Second}
})
driver := awsmail.NewFrom(client)
```

`NewFrom` has no settings callbacks and neither copies nor closes the client.
`driver.Client()` returns the live SES client, or nil when wrapped around a test
double. Configure native policies before sharing; AWS SDK retry policies remain
in effect. Use the [welcome send](https://arkoren.dev/docs/guides/mail/overview.md#sending-mail) with a verified
sender. The returned ID is SES acceptance, not delivery evidence. MIME checks
happen before the SES sending API is called.

### Local Emulator

With Docker installed, run an isolated SES emulator on loopback using fake
local-only credentials:

```sh
docker run --detach --rm --name acme-ses --publish 127.0.0.1:14569:4566 --env AWS_ACCESS_KEY_ID=000000000000 --env AWS_SECRET_ACCESS_KEY=test --env AWS_DEFAULT_REGION=us-east-1 ministackorg/ministack:1.5.8
```

Wait for `http://127.0.0.1:14569/_ministack/health` to respond. Import
`github.com/aws/aws-sdk-go-v2/aws`, `github.com/aws/aws-sdk-go-v2/credentials`,
and `github.com/aws/aws-sdk-go-v2/service/sesv2`:

```go
client := sesv2.NewFromConfig(aws.Config{
	Region: "us-east-1",
	Credentials: credentials.NewStaticCredentialsProvider("000000000000", "test", ""),
}, func(options *sesv2.Options) {
	options.BaseEndpoint = aws.String("http://127.0.0.1:14569")
})
driver := awsmail.NewFrom(client)
```

Wrap it and send the welcome message with synthetic addresses. A nonempty
receipt ID confirms emulator acceptance only. `docker stop acme-ses` removes
the disposable container and its temporary state.

See the [`awsmail` API](https://pkg.go.dev/arkoren.dev/driver/aws/awsmail).

---

Source: https://arkoren.dev/docs/guides/mail/composing-messages.md
[HTML documentation](https://arkoren.dev/docs/guides/mail/composing-messages)

# Composing Messages

A report-ready email can start as plain text and later gain HTML or an attached
file without changing how you submit it. Build those choices into a
`mail.Message` with `mail.NewMessage`. Recipients and body are required inputs;
the construction callback sets optional sender, subject, reply-to, and attachments.

Examples import `arkoren.dev/core/service/mail` and assume a
[configured mailer](https://arkoren.dev/docs/guides/mail/drivers.md#connecting-mailers). Send each completed value
with `mailer.Send(ctx, message)` as shown in [Mail Overview](https://arkoren.dev/docs/guides/mail/overview.md#sending-mail).
These are focused composition variants, not consecutive declarations to paste
into one scope.

## Message Inputs

Use application data that already identifies an authorized recipient and the
content they may receive. Mailbox syntax validation does not authorize sending
a private report:

```go
type ReportEmail struct {
	Recipient mail.Address
	Subject   string
	Text      string
	HTML      string
}

notice := ReportEmail{
	Recipient: mail.Address{Name: "Ada", Email: "ada@example.com"},
	Subject:   "Your weekly report",
	Text:      "Your weekly report is ready in Acme.",
	HTML:      "<p>Your weekly report is ready in Acme.</p>",
}
```

Start with the text representation and optional subject:

```go
message := mail.NewMessage(mail.Recipients{
	To: []mail.Address{notice.Recipient},
}, mail.Body{Text: notice.Text}, func(letter *mail.MessageBuilder) {
	letter.Subject(notice.Subject)
})
err := message.Err()
```

`Err()` reports the first construction error, wrapping `mail.ErrInvalidMessage`,
without requiring a sender that the mailer can supply. At least one recipient
across all groups and one nonempty body representation are required. Neither
can be repaired by the optional builder. The zero message is invalid.

`Subject` may be empty, but CR or LF is rejected. Each `mail.Address` requires
a bare mailbox in `Email`; `Name` is an optional display name. Address and display
name line breaks are rejected to prevent header injection. No DNS lookup or
recipient-ownership check is performed.

## Recipient Groups

When support should see a copy while an audit recipient stays out of visible
headers, select the groups explicitly:

```go
recipients := mail.Recipients{
	To:  []mail.Address{notice.Recipient},
	CC:  []mail.Address{{Email: "support@acme.test"}},
	BCC: []mail.Address{{Email: "audit@acme.test"}},
}
message := mail.NewMessage(recipients, mail.Body{Text: notice.Text})
```

`CC` is visible to recipients; `BCC` is not placed in visible delivery headers.
BCC-only mail is valid. This boundary does not conceal addresses from the
provider, your application, or the [Log driver](https://arkoren.dev/docs/guides/mail/drivers.md#log), whose summary
includes BCC. Treat recipient lists as sensitive data.

## Body Alternatives

If your template produces HTML, provide plain text too for clients that cannot
render it. Render and escape untrusted values before construction; Arkoren does
not render templates or sanitize HTML:

```go
message := mail.NewMessage(mail.Recipients{
	To: []mail.Address{notice.Recipient},
}, mail.Body{Text: notice.Text, HTML: notice.HTML})
```

These are two representations of one email, not two submissions. Text-only
and HTML-only messages are also valid. Arkoren does not derive one representation
from the other; getters return exactly the supplied strings.

## File Attachments

If the report is small enough for your provider's encoded-message limit, attach
its already-loaded bytes. Here `reportPDF` is an application-owned `[]byte`
containing the authorized report:

```go
message := mail.NewMessage(mail.Recipients{
	To: []mail.Address{notice.Recipient},
}, mail.Body{Text: "Your weekly report is attached."}, func(letter *mail.MessageBuilder) {
	letter.Attachments(mail.Attachment{
		Filename:    "weekly-report.pdf",
		ContentType: "application/pdf",
		Data:        reportPDF,
	})
})
```

Filename, content type, and nonempty unencoded data are all required. Empty
`ContentType` does not select `application/octet-stream`. Omitted disposition
becomes `mail.AttachmentDispositionAttachment`, a downloadable file.
`Attachments` replaces the whole list; pass all files in one call rather than
calling it repeatedly to append.

Construction copies attachment records and bytes, so later changes to
`reportPDF` cannot change the message. That copying and encoding consume memory;
for large reports, use [file storage](https://arkoren.dev/docs/guides/file-storage/overview.md) and send an
[authorized signed URL](https://arkoren.dev/docs/guides/file-storage/signed-urls.md) instead. Keep confidential
files out of diagnostic dumps even after construction.

Media types can include parameters such as `text/plain; charset=UTF-8`.
SMTP and SES preserve parameters through MIME encoding, except that `Filename`
supplies the `name` parameter as well as the disposition filename. Resend and
Postmark forward media types to their APIs; Mailgun's SDK selects them instead.
Check the [compatibility matrix](https://arkoren.dev/docs/guides/mail/drivers.md#compatibility-matrix) before relying
on identical wire representations.

## Inline Images

To embed a logo rather than offer it only as a download, give the inline
attachment a bare content ID and reference it through a matching HTML `cid:` URL.
Here `logoPNG` contains an already-loaded image:

```go
message := mail.NewMessage(mail.Recipients{
	To: []mail.Address{notice.Recipient},
}, mail.Body{HTML: `<p>Your report is ready.</p><img src="cid:logo.png">`},
	func(letter *mail.MessageBuilder) {
		letter.Attachments(mail.Attachment{
			Filename:    "logo.png",
			ContentType: "image/png",
			Data:        logoPNG,
			Disposition: mail.AttachmentDispositionInline,
			ContentID:   "logo.png",
		})
	})
```

An inline part requires `ContentID`; a regular attachment must omit it. Content
IDs cannot contain CR, LF, spaces, or angle brackets. Supply the bare ID, not
the `cid:` URL or a bracketed header value.

Using the filename as the ID also meets [Mailgun's restriction](https://arkoren.dev/docs/guides/mail/drivers.md#attachment-mapping).
[Postmark](https://arkoren.dev/docs/guides/mail/drivers.md#postmark) adds `cid:` only in its native payload; your
message keeps the bare value. Driver acceptance does not guarantee that a
particular mail client displays the image.

## Immutable Values

A published message has getters, not mutable fields or setters. Read
`From()`, `To()`, `CC()`, `BCC()`, `ReplyTo()`, `Subject()`, `TextBody()`,
`HTMLBody()`, or `Attachments()`; construct another message to change content.
Recipient lists are captured before callbacks. Optional lists and attachment
bytes are copied when set and again at publication. Slice getters return
independent copies, including each attachment's bytes.

With a message that has at least one attachment, preparing an export copy does
not rewrite the original message:

```go
files := message.Attachments()
files[0].Filename = "download.pdf"
```

Only `files` changes; sending `message` still uses its original filename and bytes.
Retaining the construction draft cannot change the published value either.

Callbacks run once, synchronously in order on a shared draft, even when required
inputs are invalid. Optional setters replace previous values, but the first
construction error remains sticky after replacement. Fix invalid input by
constructing a new message, not by overwriting a bad setting later. Supplied nil
callbacks and callback panics propagate; expanding a nil or empty callback slice
supplies no edits.

`Validate()` checks a complete message including the sender, without applying
defaults. An omitted sender can pass `Err()` but fail `Validate()` until the
mailer completes it. Direct driver sends and MIME encoding require the complete
form. An explicitly invalid `MessageBuilder.From` is a construction error, not
a request to fall back to a [sender default](https://arkoren.dev/docs/guides/mail/configuration.md).

## MIME Encoding

SMTP and SES encode MIME automatically. If an integration needs raw bytes,
construct a message with an explicit valid sender and pass a modern RFC 5322
Message-ID; this function does not apply mailer defaults:

```go
contents, err := mail.EncodeMIME(message, "<weekly-report-42@acme.test>")
```

On success, `contents` is newly owned sensitive message data. BCC is excluded
from visible headers. Text and HTML bodies are quoted-printable encoded, and
related multipart containers declare their actual root media type: text/plain,
text/html, or multipart/alternative. This also holds when regular attachments
place that container inside multipart/mixed.

Validation alone does not guarantee encoding. Unformattable metadata or any
physical line longer than **998 bytes excluding CRLF** returns an error wrapping
`mail.ErrInvalidMessage` and no output. Existing address and encoded-word folding
is retained; oversized plain headers and attachment metadata are rejected, not
truncated or rewritten to fit. SMTP and SES stop before submission on these errors.

The supplied Message-ID must be unfolded and uncommented, with angle brackets
around a dot-atom left side, `@`, and a dot-atom or bracketed no-fold-literal
right side. Dot-atoms contain nonempty ASCII components separated by single
dots. Empty IDs, whitespace, comments, controls, obsolete quoted forms, and
malformed syntax fail with `mail.ErrInvalidMessage` and no bytes. Accepted IDs
are preserved verbatim, including case and punctuation.

To generate an ID, call `mail.NewMessageID("mail.acme.test")`. It validates the
domain with the same right-side syntax and returns an empty ID with
`mail.ErrInvalidMessage` for malformed input. Valid domain bytes are preserved,
except raw IPv6 is wrapped as `[IPv6:address]`. Neither function checks DNS or
domain ownership; random-ID generation can also return an entropy-source error.
`EncodeMIME` checks the physical-line limit separately.

For every function and option, see the
[`mail` API reference](https://pkg.go.dev/arkoren.dev/core/service/mail).

---

Source: https://arkoren.dev/docs/guides/mail/configuration.md
[HTML documentation](https://arkoren.dev/docs/guides/mail/configuration)

# Sender Defaults

Most application mail shares a sender and a place for replies. Configure those
addresses once on the mailer, then set explicit message values only when a
particular email needs a different policy. Provider credentials and transport
settings belong to the [driver](https://arkoren.dev/docs/guides/mail/drivers.md), not these defaults.

Examples import `arkoren.dev/core/service/mail` and assume an existing
`mail.Driver` named `driver` whose construction succeeded. `mail.New` returns
a mailer without validating its address defaults; sending applies defaults
and validates the completed message before driver submission.

## Shared Addresses

To send notifications from one address while directing replies to support,
construct the mailer with both defaults:

```go
mailer := mail.New(driver, func(sender *mail.Builder) {
	sender.From(mail.Address{Name: "Acme", Email: "notifications@acme.test"})
	sender.ReplyTo(mail.Address{Name: "Acme Support", Email: "support@acme.test"})
})
```

Every message that omits its sender inherits `From`. Every empty message
reply-to list inherits the configured `ReplyTo` list. The two decisions are
independent. Without either a default sender or an explicit valid message
sender, `Send` fails with an error wrapping `mail.ErrInvalidMessage`.

Choose an address your provider permits: API providers require a verified
domain or sender, and SMTP relays apply their own rules. Syntactic validity
does not prove sending permission. A provider can reject the completed sender
at submission even when local validation passes.

`Builder.ReplyTo` accepts multiple addresses and replaces the whole list.
Calling it without arguments clears the mailer's default reply-to policy.
An empty `Builder.From` similarly restores the empty default, leaving normal
send-time validation in place.

## Message Overrides

A support reply can use a distinct approved sender and route replies to its
assigned team. Treat this as application policy, not permission for an arbitrary
request to choose an identity. Here `reply` is already validated, its recipient
authorized, and its sender selected from addresses your provider allows:

```go
type SupportReply struct {
	Recipient mail.Address
	Sender    mail.Address
	ReplyTo   mail.Address
	Text      string
}

reply := SupportReply{
	Recipient: mail.Address{Email: "ada@example.com"},
	Sender:    mail.Address{Name: "Acme Support", Email: "help@acme.test"},
	ReplyTo:   mail.Address{Email: "team@acme.test"},
	Text:      "Your report access has been restored.",
}
```

Set these optional addresses while constructing the immutable message, then
submit it with your nonnil request or job context `ctx`:

```go
message := mail.NewMessage(mail.Recipients{
	To: []mail.Address{reply.Recipient},
}, mail.Body{Text: reply.Text}, func(letter *mail.MessageBuilder) {
	letter.From(reply.Sender)
	letter.ReplyTo(reply.ReplyTo)
	letter.Subject("Your support request")
})
receipt, err := mailer.Send(ctx, message)
```

On success the driver accepts the message with those addresses; the receipt
does not establish inbox delivery.
An explicit sender wins over the default, and a nonempty reply-to list replaces
the default rather than adding to it. A message that overrides only `From`
still inherits the mailer's reply-to list.

An empty message reply-to list cannot suppress a configured default.
`MessageBuilder.ReplyTo()` clears the message setting, so inheritance resumes.
If an email must have no reply-to header, use a mailer with no default reply-to.
An explicitly empty or malformed `MessageBuilder.From` records a sticky
construction error; it does not request fallback to the default.

Defaults are applied to a local copy. Sending does not change your original
message, replay its callbacks, or replace its getters with completed values.
Use `message.Err()` to check construction before defaults; `message.Validate()`
requires a complete sender without applying them. See
[immutable composition](https://arkoren.dev/docs/guides/mail/composing-messages.md#immutable-values) for ownership
and error behavior.

## Reusable Policy

If multiple mailers share these addresses, reuse a construction policy and
apply local overrides afterward. With existing `mail.Address` values `from`
and `support`:

```go
policy := func(sender *mail.Builder) {
	sender.From(from)
	sender.ReplyTo(support)
}
mailer := mail.New(driver, policy, func(sender *mail.Builder) {
	sender.From(mail.Address{Name: "Acme Reports", Email: "reports@acme.test"})
})
```

Callbacks run once, synchronously in order on one fresh draft; the later sender
wins while the reply-to policy remains. Supplied nil callbacks and callback
panics propagate; expanding a nil or empty callback slice supplies no edits.

Reply-to input slices are copied when set and again when publishing the mailer.
Retained drafts and later input edits cannot reconfigure it. `mailer.Config()`
is a reporting snapshot with its own reply-to slice, not a constructor input or
runtime setter. Its event bus, telemetry service, and the driver remain shared
dependencies rather than cloned resources. Configure those reporting services
on [Events and Telemetry](https://arkoren.dev/docs/guides/mail/events-and-telemetry.md).

For every function and option, see the
[`mail` API reference](https://pkg.go.dev/arkoren.dev/core/service/mail).

---

Source: https://arkoren.dev/docs/guides/mail/events-and-telemetry.md
[HTML documentation](https://arkoren.dev/docs/guides/mail/events-and-telemetry)

# Events and Telemetry

When an expected email is missing, first distinguish a rejected submission from
an accepted one. Mail outcome events report what `Send` returned; telemetry
measures the attempt. Neither tells you whether an inbox received the email.
For later delivery or bounce notifications, configure the optional
[provider receiver](https://arkoren.dev/docs/guides/mail/drivers.md#provider-notifications) separately.

## Observation Setup

Events require an explicit bus. Telemetry uses an explicit service or the
[shared process handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle), resolved at
each send; it records nothing until its instrumentation is active. With import
`arkoren.dev/core/service/mail`, an existing driver named `driver`, and your
`*application.Application` named `app`, attach the application's handles:

```go
mailer := mail.New(driver, func(sender *mail.Builder) {
	sender.From(mail.Address{Email: "notifications@acme.test"})
	sender.Events(app.Events())
	sender.Telemetry(app.Telemetry())
})
```

The bus and telemetry service remain shared, not cloned or managed by the
mailer. Omitted or nil `Events` publishes nothing; omitted or nil `Telemetry`
follows the process default, including later changes. Direct driver or native
client calls bypass these facade signals.

## Events

With a configured bus and nonnil context, a normally returning `Send` publishes
one outcome synchronously before returning:

| Topic | Payload | Meaning |
| --- | --- | --- |
| `mail.Sent` (`mail.sent`) | `mail.SentEvent` | Driver acceptance, not recipient delivery. |
| `mail.Failed` (`mail.failed`) | `mail.FailedEvent` | A returned validation or driver error. |

Both payloads count recipient entries across To, CC, and BCC, including repeats,
and count regular and inline attachments together. `SentEvent` adds the
`Receipt`, whose opaque ID can be empty. `FailedEvent` instead adds a bounded
`ErrorClass`; it carries no error object, diagnostic text, or native type name.
Log-driver success still publishes `Sent` without sending mail externally.

Classification follows wrapped and joined causes in this priority order:

| `ErrorClass` | Matching Cause |
| --- | --- |
| `context.Canceled` | Cancellation, even if other known causes also match. |
| `context.DeadlineExceeded` | Deadline expiry without cancellation. |
| `mail.ErrInvalidMessage` | Invalid input without either context class. |
| `error` | Any other returned failure. |

To alert on submission failures without exporting provider diagnostics, listen
using the standard-library `context` import. Here `alerts.Notify` is your
application-owned method with signature
`Notify(context.Context, string, string) error`:

```go
unsubscribe, err := app.Events().Listen(mail.Failed,
	func(ctx context.Context, observed mail.FailedEvent) error {
		if observed.ErrorClass == "mail.ErrInvalidMessage" {
			return nil
		}

		return alerts.Notify(ctx, "Mail submission failed", observed.ErrorClass)
	})
```

Handle the registration error and retain `unsubscribe` for the listener's
lifetime. This example omits invalid-input alerts so application validation can
report them separately; it does not classify every other failure as retryable.
See [listener ownership](https://arkoren.dev/docs/guides/events/listeners.md) for registration and removal.

If you want an acceptance feed for support, a separate listener can call your
`receipts.Record(context.Context, string, int) error` method:

```go
unsubscribe, err := app.Events().Listen(mail.Sent,
	func(ctx context.Context, observed mail.SentEvent) error {
		return receipts.Record(ctx, observed.Receipt.ID, observed.Recipients)
	})
```

This is an alternative registration fragment, not a redeclaration in the same
scope. Receipts identify submissions, not your application record. For durable
correlation with an order or account, record the result around the original
`Send` call where that identity is known; best-effort events are not an outbox.

Listeners run on the sending goroutine with its context. Slow listeners delay
the caller; hand slow work to a [queue](https://arkoren.dev/docs/guides/queues/overview.md). A listener error
or panic is logged by the bus and stops later listeners, but cannot change the
submission result. A retry produces a separate outcome for each normally
returning attempt. A nil context fails before any event or telemetry.

## Send Telemetry

When enabled, each nonnil-context call creates one client span named `mail send`,
starting before validation and ending after submission but before outcome
listeners. It is a child of the calling request/job span when one exists.
SMTP connection, TLS, authentication, and message exchange are inside that
duration; listener latency is not.

| Span Attribute | Value |
| --- | --- |
| `arkoren.mail.operation` | `send` |
| `arkoren.mail.recipients` | Total To, CC, and BCC entries |
| `arkoren.mail.attachments` | Regular and inline attachment count |
| `error.type` | Class of a returned error; absent for success or a driver panic |

The counter `arkoren.mail.operations` counts attempts, not accepted messages.
The histogram `arkoren.mail.operation.duration` records seconds. Both use
`arkoren.mail.operation` and `error.type`; recipient and attachment counts are
span-only. Duration bucket boundaries are 0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5,
10, 30, and 60 seconds.

No provider or message-purpose label is added automatically. Separate mailer
instances do not split metrics into welcome/report or SMTP/API groups. If you
need that comparison, add your own bounded application labels without addresses
or individual message IDs. See [instrumented components](https://arkoren.dev/docs/guides/observability/instrumented-components.md#mail)
for the shared inventory and [Telemetry](https://arkoren.dev/docs/guides/observability/telemetry.md) for export
setup. Disabled instrumentation records nothing and adds no telemetry allocations;
normal driver submission costs remain.

## Failure Signals

Returned validation failures are observed even though no driver submission
occurs. A returned error marks the span as error and sets `error.type` to
`context.Canceled`, `context.DeadlineExceeded`, or `mail.ErrInvalidMessage`, in
that precedence order. Otherwise telemetry uses the innermost wrapped Go error
type. Unlike event `ErrorClass`, its fallback is not restricted to the string
`error`. Error text is not recorded.

A rise in invalid-message failures points to composition or sender setup.
Context interruption and other provider/network classes need separate
investigation; a failure signal alone is not evidence that remote acceptance
did not happen. `Send` still returns the original error with causes available
through `errors.Is` and `errors.As`.

A driver panic propagates unchanged and publishes neither outcome event.
The mailer still ends its own span and records count and duration without
ending the parent's span. A panic alone leaves span status unset and adds no
`error.type`, panic value, exception event, or panic log. These signals measure
an interrupted attempt, not successful submission or confirmed rejection. Add
panic reporting at your existing application recovery boundary rather than
inferring it from missing mail events or retrying blindly.

## HTTP Detail

If a slow attempt needs transport-level investigation, supply an HTTP client
whose transport comes from your telemetry service. With standard-library
imports `net/http` and `time`:

```go
client := &http.Client{
	Timeout:   10 * time.Second,
	Transport: app.Telemetry().Transport(nil),
}
```

Configure it before concurrent use. Postmark accepts it through
`postmarkmail.Builder.HTTPClient` during construction. For Resend, Mailgun,
or SES, configure the native SDK with this client before `NewFrom`; the
[driver sections](https://arkoren.dev/docs/guides/mail/drivers.md) explain the respective constructor and borrowed
ownership. HTTP spans nest under `mail send`. SMTP does not use HTTP and has
only the facade span.

## Privacy Boundaries

Mail's facade telemetry omits addresses, subject, text/HTML, attachment names and
bytes, headers, error text, and receipt ID. Events omit message content and
addresses; success events deliberately retain the opaque receipt, while failure
events retain only a fixed class and counts.

Those omissions do not redact the separate error returned by `Send`, data in
your context, native-client diagnostics, or the Log driver's records. A returned
error can quote a rejected recipient or up to part of a provider response.
Select and redact diagnostics before sending them to logs or alerts, and do not
attach the raw error chain to a bounded failure event.

Verify application assertions with the [mail fake](https://arkoren.dev/docs/guides/testing/fakes.md#assert-mail),
[injected failures](https://arkoren.dev/docs/guides/testing/fakes.md#inject-failures), and
[captured telemetry](https://arkoren.dev/docs/guides/testing/fakes.md#capture-telemetry). These tests can prove
requested submission and recorded signals, not real provider delivery.

For every function and option, see the
[`mail` API](https://pkg.go.dev/arkoren.dev/core/service/mail),
[`event` API](https://pkg.go.dev/arkoren.dev/core/event), and
[`telemetry` API](https://pkg.go.dev/arkoren.dev/core/service/telemetry).

---

Source: https://arkoren.dev/docs/guides/push/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/push/overview)

# Overview

Push notifications tell a customer about an update even when your site is not
open. A `push.Sender` submits one message to its configured driver; the browser's
service worker displays it. Provider acceptance does not establish delivery or
display, and push is not a durable notification inbox.

A **subscription** contains a browser-issued endpoint and encryption keys. Your
application stores it for the signed-in user, authorizes recipient selection,
and handles replacement or removal. Arkoren does not register devices, store
subscriptions, route between providers, or retry sends. Configure a sender in
[Drivers](https://arkoren.dev/docs/guides/push/drivers.md) and obtain subscriptions through
[Browser Integration](https://arkoren.dev/docs/guides/push/browser-integration.md).

## Preparing Messages

These focused examples assume a configured `sender`, a `context.Context` named
`ctx`, and a `push.Subscription` named `subscription` loaded from an authorized
registration record. Import `arkoren.dev/core/service/push`. Snippets retain
error assignments but omit routine propagation; complete functions and security
decisions handle errors before proceeding.

For an order update, keep application input typed before translating it into
the message's string metadata:

```go
type OrderNotice struct {
	OrderID string
	Path    string
}

notice := OrderNotice{OrderID: "123", Path: "/orders/123"}
```

Choose the subscription on the server, not from an arbitrary recipient in the
request. Required arguments identify that recipient and a nonempty UTF-8 title;
the construction callback adds optional body text and application data:

```go
message := push.NewMessage(subscription, "Order shipped", func(notification *push.MessageBuilder) {
	notification.Body("Your order is on its way.")
	notification.Data(map[string]string{
		"order_id": notice.OrderID,
		"url":      notice.Path,
	})
})
```

`Data` accepts string values, not arbitrary JSON or a provider notification
object. Body text and metadata keys and values must also be valid UTF-8.
`Message` is an immutable snapshot: construction copies the map, and
`Data()` returns a detached copy. `Data` replaces earlier metadata rather than
merging it. `Err()` and `Validate()` expose the first declaration error, even if
a later callback supplies a valid replacement.

Pass a `push.Subscription` or `push.Token` value, not a pointer. Tokens are
interpreted only by a driver that supports them; the Web Push driver accepts
subscriptions only. Subscription validation checks HTTPS structure and browser
encryption keys without network I/O. It does not approve a destination for
outgoing requests; [driver endpoint policy](https://arkoren.dev/docs/guides/push/drivers.md#endpoint-policy) does that.

## Submitting Notifications

Submit the published message once through the sender:

```go
receipt, err := sender.Send(ctx, message)
```

`Send` validates before calling the driver. Invalid portable input never reaches
it. A nil error means acceptance; `Receipt.ID` is optional opaque acceptance
metadata and can be empty. Web Push deliberately returns an empty ID rather than
exposing a sensitive provider URL.

Do not put sensitive information in display text: notifications can appear on a
locked device. Keep subscription endpoints, keys, native tokens, payloads, and
receipt IDs out of general logs and telemetry. Top-level diagnostics redact
recipient details, but explicitly unwrapped native errors can still contain URLs.

## Choosing Retention

The time to live (**TTL**) requests how long the provider can retain an update
while the browser is offline. It is neither a delivery deadline nor a promise
that the provider stores or displays the notification.

The sender requests 24 hours by default. For a status update that becomes stale
after five minutes, import `time` and override retention for this send:

```go
receipt, err := sender.Send(ctx, message, push.TTL(5*time.Minute))
```

An omitted option or `push.TTL(0)` inherits the sender default; a negative option
requests no offline storage. No-store does not prevent immediate delivery.
[Drivers](https://arkoren.dev/docs/guides/push/drivers.md#sender-settings) shows how to change the sender default.
Direct driver calls differ: raw `push.SendConfig{TTL: 0}` means no-store, and a
negative raw TTL is invalid. Web Push rounds positive durations down to whole
seconds, so a duration below one second also becomes no-store.

## Handling Outcomes

Classify failures with standard-library `errors.Is`. Check
`push.ErrOutcomeUnknown` before cancellation or provider failures: a wrapped or
joined error can match several causes without resolving whether acceptance
occurred. Every failed send returns a zero receipt.

| Error | Application Response |
| --- | --- |
| `push.ErrOutcomeUnknown` | Acceptance is uncertain; decide whether a duplicate is acceptable before replaying. |
| `push.ErrInvalidMessage` | Correct input, endpoint policy, payload size, or retention before resubmission. |
| `push.ErrUnsupported` | Choose a recipient or operation supported by the configured driver. |
| `push.ErrNotFound` | Invalidate only the stored registration version used for this send. |
| `push.ErrRateLimited` | Apply an explicit bounded scheduling policy. |
| `push.ErrUnavailable` | Report availability failure and apply application policy; do not delete the registration. |

Web Push maps authoritative HTTP 404/410 responses to `ErrNotFound`.
Authentication failures do not prove expiry: repair credentials rather than
deleting live subscriptions. Conditional invalidation prevents an older send
from deleting a registration refreshed while the request was in flight.

> [!WARNING]
> A timeout after dispatch can leave acceptance unknown. Automatic replay can
> produce duplicate notifications. Neither the sender nor the Web Push driver
> retries automatically or provides exactly-once delivery.

Permission revocation, offline expiry, operating-system settings, and worker
failures can prevent display after acceptance. Use [Testing](https://arkoren.dev/docs/guides/push/testing.md) to check
application submission and browser display as separate outcomes.

## Observing Submissions

For optional in-process observations, configure `push.Builder.Events` with a
shared event bus as shown in [sender settings](https://arkoren.dev/docs/guides/push/drivers.md#sender-settings).
`push.Accepted` and `push.Failed` report returned driver outcomes, not delivery.
Their `SubmissionEvent` contains only duration and a bounded error class.

Portable preflight failures emit no event. A failed event with
`ErrorClass == "outcome_unknown"` means acceptance could have occurred.
Acceptance has an empty class; other classes identify cancellation, deadline,
push sentinels, or an unclassified error. Listener failures and panics do not
change `Send`'s result; driver panics propagate without an outcome event. These
best-effort observations are not a durable audit log or retry mechanism.

Telemetry follows the process default on each call unless explicitly selected.
It records bounded operation/error classes, never message, recipient, receipt,
or native error text. Unknown acceptance takes classification precedence over
joined cancellation or deadline errors without replacing their returned causes.
See the [push instruments](https://arkoren.dev/docs/guides/observability/instrumented-components.md#push-notifications)
for span and metric names.

For message, recipient, and sender operations, see the
[push API reference](https://pkg.go.dev/arkoren.dev/core/service/push).

---

Source: https://arkoren.dev/docs/guides/push/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/push/drivers)

# Drivers

Use the core test fake to check what your application submits without making
network requests. Choose Web Push to send encrypted notifications to browser
subscriptions, including supported Safari web apps. Neither backend tracks
device delivery or stores your application's registrations.

## Compatibility Matrix

These capabilities describe Arkoren's exposed behavior. The fake records local
acceptance only; its browser-shaped input does not emulate a browser or provider.

| Driver | Browser Subscriptions | Native Tokens | External Submission | No-Store TTL | Delivery Tracking | Sender Events |
| --- | --- | --- | --- | --- | --- | --- |
| [Test Fake](#test-fake) | Yes — structural validation only | Yes — synthetic input | No | Yes — records resolved zero | No | Yes — opt-in |
| [Web Push](#web-push) | Yes | No | Yes | Yes — zero TTL header | No | Yes — opt-in |

Events belong to the sender, not the driver. Direct driver calls emit none.
`push.Token` does not select a provider automatically, and Web Push is not a
native mobile push integration.

## Test Fake

`pushtest.Fake` is bundled with `arkoren.dev/core`. It needs no credentials,
construction options, external process, or lifecycle hooks. In a test, import
`arkoren.dev/core/service/push` and its `pushtest` subpackage:

```go
fake := pushtest.New()
sender := push.New(fake)
```

Pass `sender` to the application code under test. The fake records immutable
messages and resolved TTL values; [submission tests](https://arkoren.dev/docs/guides/push/testing.md#asserting-submissions)
show precise assertions and failure injection. `Sent()` returns a detached slice
and message metadata accessors return copies.

Portable validation happens before recording or consuming an injected failure.
The fake does not enforce Web Push origin or payload limits, encrypt messages,
persist data, or simulate delivery. No cleanup is required; `Reset` clears
recordings and failure settings when a test deliberately reuses the instance.

See the [pushtest API reference](https://pkg.go.dev/arkoren.dev/core/service/push/pushtest).

## Web Push

`webpushpush` submits standards-based encrypted Web Push. You need a browser-issued
subscription, a [persistent VAPID identity](https://arkoren.dev/docs/guides/push/vapid-keys.md), and an operator contact
URI. Browser Web Push does not require a native APNs certificate or Apple
Developer Program membership; [browser integration](https://arkoren.dev/docs/guides/push/browser-integration.md)
still supplies registration and display code.

### Installing Web Push

From your application's module directory, install the driver:

```sh
go get arkoren.dev/driver/webpush
```

With `identity` as a loaded `key.Signing` value, construct the driver using
`arkoren.dev/driver/webpush/webpushpush` and wrap it with
`arkoren.dev/core/service/push`:

```go
driver, err := webpushpush.New(identity, "mailto:push@example.com")
```

Handle a construction error before wiring the driver. `New` validates the key,
contact, and settings without contacting a provider or registering a browser.
The contact is an operator's `mailto:` address or HTTPS URI, not the customer's
notification address.

```go
sender := push.New(driver)
```

Reuse the constructor's pointer for concurrent sends; do not copy its
synchronized authorization cache. The driver owns its signing-scalar copy and
hardened HTTP client, not your original identity. It has no `Start`, `Stop`,
`Ping`, native-client accessor, or credential-zeroization operation.

### Bounding Requests

A complete send has a ten-second timeout by default; a shorter caller context
deadline wins. If your request budget is three seconds, import `time` and use
this alternative construction:

```go
driver, err := webpushpush.New(identity, "mailto:push@example.com", func(connection *webpushpush.Builder) {
	connection.Timeout(3 * time.Second)
})
```

Zero restores ten seconds; a negative timeout fails construction. This bound
controls request work, not the notification's offline retention.

### Endpoint Policy

Subscription endpoints are untrusted capability URLs. By default the driver
permits `https://fcm.googleapis.com`,
`https://updates.push.services.mozilla.com`, and DNS hosts below
`.push.apple.com` at a label boundary. The bare `push.apple.com` host and
lookalike suffixes do not match. It requires canonical lowercase DNS names and
HTTPS on port 443; explicit `:443` is accepted. IP literals, URL credentials,
fragments, and noncanonical authorities are rejected.

Keep this curated policy unless you need a reviewed alternative. To restrict
submission to Google's origin, replace the defaults at construction:

```go
driver, err := webpushpush.New(identity, "mailto:push@example.com", func(connection *webpushpush.Builder) {
	connection.AllowedOrigins("https://fcm.googleapis.com")
})
```

`AllowedOrigins` replaces rather than extends the defaults. Entries are exact
HTTPS origins without paths, queries, trailing slashes, or wildcards; an empty
list denies all origins. This is trusted deployment configuration: never build
the allowlist from a browser's registration payload.

> [!WARNING]
> Approving an arbitrary subscription origin can expose your server to SSRF.
> Keep a narrow reviewed allowlist. Even allowed origins must resolve entirely
> to public addresses; the driver validates DNS at dialing and connects to a
> validated address while TLS verifies the original hostname.

The owned transport does not follow redirects or use environment proxies, and
there is no custom HTTP-client bypass. Request duration and response reads are
bounded. Private-network emulators and localhost providers are not supported
targets; use the fake for offline application tests.

### Sending Notifications

Send the [typed order notice](https://arkoren.dev/docs/guides/push/overview.md#preparing-messages) through the configured
sender using a stored `push.Subscription` value. The browser receives flat JSON
with `title`, optional `body`, and optional string-valued `data`, not a native
provider envelope. Recipient endpoints and encryption keys are absent from that
payload; the [service worker](https://arkoren.dev/docs/guides/push/browser-integration.md#displaying-notifications)
must read this shape.

The encoded JSON must fit within 3,993 plaintext bytes, including field names,
punctuation, and escaping. Encryption overhead and the final-record delimiter
bring the supported envelope to 4,096 bytes. Oversized input fails with
`push.ErrInvalidMessage` before transport; measuring body characters alone is
not sufficient.

TTL is rounded down to whole seconds. The representable limit is 2,147,483,647
whole seconds; a value whose whole-second count exceeds it fails rather than
being clamped. Providers can impose smaller bounds. Raw `SendConfig.TTL == 0`
requests no-store, unlike zero in the sender's per-call option.

### Interpreting Responses

HTTP 201 establishes acceptance and returns an empty `Receipt.ID`, even if
optional headers are missing or response cleanup fails. It does not establish
browser display. Other response classes are:

| Response | Classification |
| --- | --- |
| 404 or 410 | `push.ErrNotFound` |
| 429 | `push.ErrRateLimited` |
| 400 or 413 | `push.ErrInvalidMessage` |
| 500–599 | `push.ErrUnavailable` |
| Other 2xx | `push.ErrOutcomeUnknown` |

Authentication failures, redirects, and other unclassified responses remain
typed provider errors, not expired-registration or unsupported-operation signals.
Use `errors.As` with `*webpushpush.ProviderError` to inspect numeric `StatusCode`
and nonnegative `RetryAfter`; these fields do not schedule retries. Provider
bodies and capability URLs are not retained in that error.

Transport failures after dispatch conservatively preserve unknown acceptance
and native/context causes. Do not replay them automatically or delete a
registration after an authentication failure. On confirmed not-found responses,
invalidate only the submitted registration version; see
[outcome handling](https://arkoren.dev/docs/guides/push/overview.md#handling-outcomes).

### Credential Ownership

Concurrent sends share an origin-scoped VAPID authorization until five minutes
before its twelve-hour expiry. This cache belongs to one driver instance, not
other replicas. It holds at most 256 unexpired origins; adding another beyond
that limit fails closed. Keep the identity stable and the origin policy narrow.

Formatting the constructor pointer with `%v`, `%+v`, or `%#v` writes
`[REDACTED Web Push driver]`. This does not make arbitrary serialization,
reflection, driver copies, or native objects safe to log. Explicitly unwrapped
transport causes can contain sensitive URLs even when top-level error text is
bounded.

See the [Web Push API reference](https://pkg.go.dev/arkoren.dev/driver/webpush/webpushpush).

## Sender Settings

Sender settings apply to either backend. With an existing `driver`, import
`time` and `arkoren.dev/core/service/push` to request a one-hour default:

```go
sender := push.New(driver, func(delivery *push.Builder) {
	delivery.TTL(time.Hour)
})
```

A zero builder TTL restores 24 hours; a negative value disables offline storage.
The construction draft cannot reconfigure a published sender. The sender shares
the driver and does not take over its lifecycle.

If you need submission observations, supply a shared `*event.Bus` named `bus`
and a shared `*telemetry.Telemetry` named `instrumentation`:

```go
sender := push.New(driver, func(delivery *push.Builder) {
	delivery.Events(bus)
	delivery.Telemetry(instrumentation)
})
```

Nil events disable notifications; nil telemetry follows the process default on
each call. The [submission observation contract](https://arkoren.dev/docs/guides/push/overview.md#observing-submissions)
explains privacy and outcome limits. These settings add neither delivery
tracking nor background work.

See the [sender API reference](https://pkg.go.dev/arkoren.dev/core/service/push).

---

Source: https://arkoren.dev/docs/guides/push/vapid-keys.md
[HTML documentation](https://arkoren.dev/docs/guides/push/vapid-keys)

# VAPID Keys

A browser subscription created with your application server's public key requires
sends authenticated by the matching private key. **VAPID** is that server
authentication scheme; it is separate from the browser-generated encryption
keys in `push.Subscription.Keys`.

Provision a dedicated P-256 signing identity once, persist it in protected secret
storage, and load the same identity on every application replica and restart.
The [Web Push driver](https://arkoren.dev/docs/guides/push/drivers.md#web-push) consumes your identity; it does not
generate or persist one.

> [!WARNING]
> Generating a new identity at startup breaks sends to existing subscriptions.
> Protect and back up the private key. Never put it in source control, browser
> responses, container images, or logs; publish only the public component.

## Provisioning Identity

Use the [installed CLI](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren) in a protected
provisioning directory, outside server startup:

```sh
arkoren key generate signing --out vapid-private.pem
```

This creates an unencrypted PKCS #8 P-256 private-key PEM file and refuses to
replace an existing file or symlink. Move the identity into your deployment's
secret storage and mount it read-only for the application. Follow the generic
[key commands](https://arkoren.dev/docs/guides/cli/keys.md) for file permissions, backups, and partial-write
recovery rather than regenerating an established identity.

Before deployment, check the stored representation and export its public point:

```sh
arkoren key check signing vapid-private.pem
arkoren key public signing vapid-private.pem --format vapid
```

The check establishes local format validity, not provider acceptance. Explicit
`--format vapid` prints the public point as unpadded base64url, suitable for
`applicationServerKey`; the default public output is SPKI PEM and is not the
browser registration value. These commands do not upload keys or edit `.env`.

## Loading Identity

Install `arkoren.dev/foundation/key` in your application module and import
`arkoren.dev/foundation/key`. Load the protected file once during startup:

```go
identity, err := key.LoadSigning("vapid-private.pem")
```

Handle failure before publishing configuration or constructing a sender.
Loading rejects malformed key pairs, other curves or algorithms, leading junk,
and extra PEM blocks. If your secret store already supplies PEM bytes, use
`key.ParseSigning(contents)` instead. See [Application Keys](https://arkoren.dev/docs/guides/keys/overview.md)
for typed key selection and loading, including importing native signing keys.

Pass this `key.Signing` value to the
[driver constructor](https://arkoren.dev/docs/guides/push/drivers.md#installing-web-push). The driver snapshots its
own key material; your application retains ownership of the loaded identity.

## Publishing Public Keys

Export only the browser-ready public component from the successfully loaded
identity:

```go
publicKey := identity.VAPIDPublicKey()
```

Serve `publicKey` as plain text from your application's `GET /push/public-key`,
or place it in public page configuration. That route is application-owned, not
automatically installed by Arkoren. Do not serialize the identity or return its
private PEM. Public-key exposure is intentional and does not disclose the
private scalar.

Keep the served public key aligned with the signer used for new subscriptions.
Store a server-selected key identifier with each registration so later sends
can select the matching identity without trusting a client-supplied key choice.

## Rotating Identity

Changing the public key requires new subscriptions. Retain old identities for
registrations still bound to them during a migration; do not send every existing
subscription with the new private key.

Use an explicit browser flow to unsubscribe and resubscribe with the new public
key, then replace the stored registration and its key identifier atomically.
Keep a registration version as well, so cleanup after an older failed send
cannot remove its replacement. Losing the old identity prevents continued sends
to subscriptions bound to it; generating a replacement does not restore that
relationship.

Continue with [Browser Integration](https://arkoren.dev/docs/guides/push/browser-integration.md) to use the public key
in subscription registration. For typed exports and parsing, see the
[key API reference](https://pkg.go.dev/arkoren.dev/foundation/key).

---

Source: https://arkoren.dev/docs/guides/push/browser-integration.md
[HTML documentation](https://arkoren.dev/docs/guides/push/browser-integration)

# Browser Integration

To notify a customer outside an open page, connect three application-owned pieces:
a browser subscription, a stored registration for the signed-in customer, and a
service worker that displays incoming messages. The Web Push driver encrypts and
submits the message; it installs none of these pieces.

This walkthrough assumes an HTTPS application, an existing
[session login flow](https://arkoren.dev/docs/guides/authentication/session-login.md), and a
[persistent VAPID identity](https://arkoren.dev/docs/guides/push/vapid-keys.md). Serve its public key as plain text at
`GET /push/public-key`. You will serve `/service-worker.js` and `/push.js` on
the same origin and implement `POST /push/subscriptions` in your application.

## Browser Requirements

Feature-detect service workers, `PushManager`, and `Notification`; do not choose
code by browser name. Request permission only in response to an explicit user
action, such as clicking “Enable order notifications.” The
[subscription API](https://developer.mozilla.org/en-US/docs/Web/API/PushManager/subscribe)
requires a secure context and documents the user-gesture requirement.

Chrome and Firefox support standards-based Web Push. Safari supports it starting
with Safari 16.1 on macOS Ventura. On iOS and iPadOS 16.4 or later, use an installed
Home Screen web app, not an ordinary browser tab. Supply a web app manifest with
`display: "standalone"` or `"fullscreen"` for that flow.
[WebKit's guidance](https://webkit.org/blog/13878/web-push-for-web-apps-on-ios-and-ipados/)
also confirms that browser Web Push needs neither Apple Developer Program
membership nor native APNs credentials. Feature support is not proof that a
particular device will display your notification.

## Displaying Notifications

Serve the following JavaScript at `/service-worker.js`; its root location permits
root scope. The driver's decrypted application payload is flat JSON, with no
`notification` wrapper:

```json
{
  "title": "Order shipped",
  "body": "Your order is on its way.",
  "data": {
    "order_id": "123",
    "url": "/orders/123"
  }
}
```

`title` is always present; empty body and data fields are omitted. Display that
shape with `showNotification`. Use `waitUntil` to extend the worker event while
the display promise completes:

```js
self.addEventListener("push", (event) => {
  event.waitUntil((async () => {
    let payload = { title: "Order update" };
    try {
      const decoded = event.data?.json();
      if (decoded && typeof decoded.title === "string") payload = decoded;
    } catch {
      // Malformed data still gets a visible fallback notification.
    }
    await self.registration.showNotification(payload.title || "Order update", {
      body: payload.body,
      data: payload.data,
    });
  })());
});
```

Notifications can appear on a locked device, so avoid sensitive display text.
Application data is navigation input, not permission to read an order. Add a
click handler to the same worker file that permits only your own origin:

```js
self.addEventListener("notificationclick", (event) => {
  event.notification.close();
  let target = new URL("/", self.location.origin);
  try {
    const requested = new URL(event.notification.data?.url ?? "/", self.location.origin);
    if (requested.origin === self.location.origin) target = requested;
  } catch {
    // Invalid URLs fall back to the application's home page.
  }
  event.waitUntil(self.clients.openWindow(target.href));
});
```

The destination page must still authenticate the viewer and authorize access to
the order. Cross-origin or malformed URLs open the home page instead.

## Preparing Registration

On the signed-in page, add a disabled button with `id="enable-push"` and a status
element with `id="push-status"`. Load `/push.js` with
`<script type="module" src="/push.js"></script>`. The following blocks form that
module in order.

First decode the public base64url point—not the private key—and prepare the worker
and configuration before the button can request permission:

```js
function publicKeyBytes(encoded) {
  const text = encoded.replace(/-/g, "+").replace(/_/g, "/");
  const binary = atob(text + "=".repeat((4 - text.length % 4) % 4));
  return Uint8Array.from(binary, (character) => character.charCodeAt(0));
}

async function preparePush() {
  if (!("serviceWorker" in navigator) || !("PushManager" in window) || !("Notification" in window)) {
    throw new Error("Push is unavailable in this browser context.");
  }
  await navigator.serviceWorker.register("/service-worker.js");
  const registration = await navigator.serviceWorker.ready;
  const response = await fetch("/push/public-key", { credentials: "same-origin" });
  if (!response.ok) throw new Error("Push configuration is unavailable.");
  return { registration, applicationServerKey: publicKeyBytes((await response.text()).trim()) };
}
```

Preparation creates no subscription and does not request notification permission.
Leave the button disabled if it fails, and show a fixed status instead of logging
native errors or subscription material:

```js
const button = document.querySelector("#enable-push");
const status = document.querySelector("#push-status");
let configuration;
button.disabled = true;
try {
  configuration = await preparePush();
  button.disabled = false;
} catch {
  status.textContent = "Notifications are unavailable. Check browser support and configuration.";
}
```

## Saving Subscriptions

Before enabling uploads, protect the registration route as described in
[Registration Ownership](#registration-ownership). Reusing a subscription here
assumes the same persisted VAPID key; after a key change, use the explicit
[rotation flow](https://arkoren.dev/docs/guides/push/vapid-keys.md#rotating-identity) instead.

Add this click handler to `/push.js`. Only a successful server response produces
the “enabled” status:

```js
button.addEventListener("click", async () => {
  if (!configuration) return;
  button.disabled = true;
  try {
    const permission = await Notification.requestPermission();
    if (permission !== "granted") {
      status.textContent = "Notifications were not enabled. Check browser permissions.";
      return;
    }
    const { registration, applicationServerKey } = configuration;
    const existing = await registration.pushManager.getSubscription();
    const subscription = existing ?? await registration.pushManager.subscribe({
      userVisibleOnly: true,
      applicationServerKey,
    });
    const saved = await fetch("/push/subscriptions", {
      method: "POST",
      credentials: "same-origin",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(subscription.toJSON()),
    });
    if (!saved.ok) throw new Error("Registration was not saved.");
    status.textContent = "Order notifications are enabled for this browser.";
  } catch {
    status.textContent = "Could not enable notifications. Try again from this page.";
  } finally {
    button.disabled = false;
  }
});
```

`userVisibleOnly: true` commits this subscription to visible notifications, not
silent background jobs. `toJSON()` supplies `endpoint` and `keys`, with `p256dh`
and `auth`; browsers can also include `expirationTime`. A failed upload can leave
a browser subscription without a stored application registration. A later
explicit attempt reuses it rather than claiming that preferences were saved.

## Registration Ownership

The server must derive the owner from authentication, never from JSON. Configure
[session loading](https://arkoren.dev/docs/guides/http-capabilities/sessions.md),
[CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md), authentication, a
[body limit](https://arkoren.dev/docs/guides/http-fundamentals/body-limit.md), JSON
[content-type policy](https://arkoren.dev/docs/guides/http-fundamentals/content-type.md), and
[rate limiting](https://arkoren.dev/docs/guides/abuse-prevention/rate-limiting.md) before mounting the route.
CORS alone supplies neither authentication nor CSRF protection.

Your storage callback owns authorization, per-user quotas, persistence, and atomic
versioned replacement. For this example, define a server-side draft using
`arkoren.dev/core/service/push`:

```go
type RegistrationDraft struct {
	OwnerID      string
	VAPIDKeyID   string
	Subscription push.Subscription
}
```

`OwnerID` comes from your authenticated `User` value; `VAPIDKeyID` comes from server
configuration matching the served public key. Storage assigns a registration ID
and a new version. Enforce endpoint uniqueness and reject an endpoint already
owned by another account; do not silently transfer ownership when someone signs
in on a shared browser. A subscription is a sensitive capability, not proof of
physical-device ownership.

The following complete handler factory uses your existing `User` type with a
string `ID`. Import `context`, `net/http`, `arkoren.dev/core/http/request`,
`arkoren.dev/core/http/router`, and `arkoren.dev/core/service/push`. The `save`
callback implements the storage contract above:

```go
func registrationHandler(keyID string, save func(context.Context, RegistrationDraft) error) router.Handler {
	return func(writer http.ResponseWriter, incoming *http.Request) error {
		user, err := request.User[User](incoming)
		if err != nil {
			return err
		}
		subscription, err := request.JSON[push.Subscription](incoming, request.MaxSize(8<<10))
		if err != nil {
			return err
		}
		if err := push.NewMessage(subscription, "Registration check").Validate(); err != nil {
			http.Error(writer, "Invalid push subscription", http.StatusBadRequest)
			return nil
		}
		draft := RegistrationDraft{OwnerID: user.ID, VAPIDKeyID: keyID, Subscription: subscription}
		if err := save(incoming.Context(), draft); err != nil {
			return err
		}
		writer.WriteHeader(http.StatusCreated)
		return nil
	}
}
```

This handler applies an 8 KiB body limit. Default JSON decoding ignores the browser's
extra `expirationTime`; it is not part of `push.Subscription` and does not promise
an unlimited lifetime. Structural validation checks the HTTPS endpoint and
canonical encryption keys without sending a notification. `p256dh` is an
unpadded base64url encoding of a valid 65-byte uncompressed P-256 public point;
`auth` encodes exactly 16 bytes. These are browser subscription keys, not the
server's VAPID signing key. Endpoints cannot contain URL credentials or a literal
`#`, even an empty trailing fragment; encoded `%23` remains opaque path/query data.

With your already-protected router named `protected`, your server-selected
`keyID`, and application callback `saveRegistration`, mount the handler:

```go
protected.Post("/push/subscriptions", registrationHandler(keyID, saveRegistration))
```

The route's authentication must load the exact `User` type read by the handler.
Rate limiting bounds requests, not stored device counts; storage quotas remain
necessary. Keep registration endpoints and auth secrets out of logs. Never use
client input to configure allowed origins: a valid stored subscription remains
subject to the driver's [SSRF and public-network policy](https://arkoren.dev/docs/guides/push/drivers.md#endpoint-policy)
at send time.

## Sending Updates

Load only registrations authorized for the order's customer, select their matching
VAPID signer, and submit the [order notice](https://arkoren.dev/docs/guides/push/overview.md#preparing-messages).
The UI's “enabled” status means storage succeeded. A nil send error means provider
acceptance. A visible notification is a third checkpoint, verified separately
with the [browser checklist](https://arkoren.dev/docs/guides/push/testing.md#verifying-browsers).

Browser permission, subscriptions, and server storage have separate lifetimes.
On preference removal, stop targeting the server registration and call
`subscription.unsubscribe()` in the browser, handling its result; protect the
application deletion route with authentication, CSRF, and ownership checks.
Neither action recalls an already submitted notification.

Decide whether registrations survive logout on a shared device. Invalidate stale
page UI on account changes and do not reassign the old account's endpoint to the
new account implicitly. Reconcile subscription state on later signed-in visits:
permission can be revoked and subscriptions can expire or change. A confirmed
not-found send invalidates only the submitted stored version; authentication
failures and uncertain outcomes do not justify registration deletion.

For the message and subscription types, see the
[push API reference](https://pkg.go.dev/arkoren.dev/core/service/push).

---

Source: https://arkoren.dev/docs/guides/push/testing.md
[HTML documentation](https://arkoren.dev/docs/guides/push/testing)

# Testing

Test notification submission without asking for browser permission or contacting
a provider. `pushtest.Fake` records accepted messages and resolved retention so
you can inspect the application's intent. It does not prove encryption,
provider acceptance, delivery, or service-worker display; those need a separate
browser check.

## Asserting Submissions

The fake is included in `arkoren.dev/core` and requires no credentials or network
access. In your application's `push_test.go`, import these packages and define
the small input used by the test:

```go
package notification_test

import (
	"context"
	"errors"
	"testing"
	"time"

	"arkoren.dev/core/service/push"
	"arkoren.dev/core/service/push/pushtest"
)

type OrderNotice struct {
	OrderID string
	Path    string
}
```

Construct a fresh fake per test and inject its sender into your application.
This complete test demonstrates the submission boundary directly; replace its
`Send` call and message construction with your order service invocation when
testing that service's behavior:

```go
func TestOrderNotice(t *testing.T) {
	t.Parallel()
	fake := pushtest.New()
	sender := push.New(fake)
	recipient := push.Token("test-device")
	notice := OrderNotice{OrderID: "123", Path: "/orders/123"}
	message := push.NewMessage(recipient, "Order shipped", func(notification *push.MessageBuilder) {
		notification.Body("Your order is on its way.")
		notification.Data(map[string]string{"order_id": notice.OrderID, "url": notice.Path})
	})
	if _, err := sender.Send(t.Context(), message, push.TTL(5*time.Minute)); err != nil {
		t.Fatal(err)
	}
	fake.AssertSentCount(t, 1)
	fake.AssertSent(t, func(submission pushtest.Submission) bool {
		return submission.Message.To() == recipient &&
			submission.Message.Title() == "Order shipped" &&
			submission.Message.Body() == "Your order is on its way." &&
			submission.Message.Data()["order_id"] == notice.OrderID &&
			submission.Message.Data()["url"] == notice.Path &&
			submission.Config.TTL == 5*time.Minute
	})
}
```

The count excludes unexpected additional submissions; the predicate checks the
recipient, content, and raw retention received by the driver. No start, stop, or
cleanup hook is needed. Synthetic `push.Token` input is convenient here but does
not imply that Web Push supports native tokens.

`AssertSent` requires one recorded submission to match every supplied predicate.
`AssertNotSent` fails if any submission matches them all; with no predicates it
requires no recordings. `AssertNothingSent` checks the same empty state.
`Sent()` returns an independent slice in acceptance order, and message metadata
accessors return copies. Assertion failures do not print sensitive content.

## Testing Uncertainty

Your application must not mistake cancellation for proof of non-acceptance. Add
this second complete test to the same file to preserve both causes and verify
that failure returns no receipt or accepted recording:

```go
func TestUnknownAcceptance(t *testing.T) {
	t.Parallel()
	fake := pushtest.New()
	fake.FailNext(errors.Join(context.Canceled, push.ErrOutcomeUnknown))
	sender := push.New(fake)
	message := push.NewMessage(push.Token("test-device"), "Order shipped")
	receipt, err := sender.Send(t.Context(), message)
	if !errors.Is(err, push.ErrOutcomeUnknown) || !errors.Is(err, context.Canceled) {
		t.Fatal("expected uncertain acceptance with preserved cancellation")
	}
	if receipt != (push.Receipt{}) {
		t.Fatal("failed submission returned a receipt")
	}
	fake.AssertNothingSent(t)
}
```

The fake does not claim that an unknown submission failed to reach a real
provider; it records only accepted test submissions. Adapt this injection to
your application's policy test and assert that it does not blindly replay an
uncertain notification or remove a refreshed registration.

`FailNext` replaces the next valid send's failure and takes precedence over
`FailWith`, which persists until cleared. Invalid input does not consume the
one-shot failure. To exercise persistent throttling with an existing `fake`:

```go
fake.FailWith(push.ErrRateLimited)
```

Subsequent valid sends return that failure until `fake.FailWith(nil)` clears it.
`FailNext(nil)` clears only the one-shot failure. `Reset()` clears both failure
settings and recordings, and acceptance IDs restart at `fake-1`; it does not
simulate device state or delivery.

Run `go test ./...` from your application module. These tests make no external
requests. If you implement a custom driver, the
[conformance suite](https://arkoren.dev/docs/guides/testing/conformance-suites.md) and `pushtest.Run` check
portable validation, ownership, raw TTL, context errors, and concurrency—not
provider wire compatibility or actual device delivery.

## Verifying Browsers

Use the [browser walkthrough](https://arkoren.dev/docs/guides/push/browser-integration.md) on an HTTPS deployment with
a protected test account. Do not paste real endpoints or subscription keys into
logs or issue reports. Test supported browsers independently; on iOS and iPadOS
use the installed Home Screen app in the supported version context.

1. Click the enable button. Confirm permission and one stored registration owned
   by the signed-in account, with the correct VAPID identity and a version. A
   failed upload must not show the enabled status.
2. Submit a small notification using that registration and its matching signer.
   Check nil error separately from a visible notification; operating-system
   settings can suppress display after acceptance.
3. Click the notification. Confirm same-origin navigation and authorization on
   the order page. Send a cross-origin `data.url` and confirm fallback to the
   home page, not navigation to the other origin.
4. Restart the server with the same persisted signing key. Check that the existing
   registration still accepts sends. Exercise offline expiry and no-store without
   assuming that accepted notifications must eventually appear.
5. Revoke permission or unsubscribe and exercise application cleanup. Confirm
   that a definitive not-found response invalidates only the submitted version,
   and authentication errors do not delete live registrations.
6. Sign out and switch accounts on the same browser. Confirm the chosen ownership
   policy prevents implicit transfer or disclosure of the first account's updates.

This is a manual acceptance checklist, not a report of executed browser tests.
Neither passing fake assertions nor seeing one notification establishes
exactly-once delivery, an inbox, or guaranteed future display.

For recording, injection, and assertion operations, see the
[pushtest API reference](https://pkg.go.dev/arkoren.dev/core/service/push/pushtest).

---

Source: https://arkoren.dev/docs/guides/payments/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/payments/overview)

# Payments Overview

A buyer's approval and a provider's collection are different events. Use
`payment.Client` to request payment operations and read their outcomes, while
your application decides whether those outcomes satisfy an authorized order.
Start with a [configured client](https://arkoren.dev/docs/guides/payments/drivers.md) named `payments` and a caller context
named `ctx`; the facade needs no application container, worker, or webhook.

## Payment Resources

A **checkout** is an approval flow. Its completion does not necessarily mean
money moved. A **payment** describes an individual money movement; a **refund**
has its own progress toward returning funds. A **subscription** is a recurring
agreement, not evidence that its latest invoice was paid.

The provider owns these resources and subscription renewals. Your application
owns authorization, server-side pricing, merchant and test/live associations,
durable operation records, and fulfillment. Arkoren stores no local billing
projection and supplies no tax, catalog, invoice, dispute, or entitlement engine.

Resource IDs are opaque provider-scoped handles, not authorization. Save them
against the initiating account and order, and preserve their complete values.
Never convert a checkout or session ID into a financial payment ID. Missing
relationships stay absent; successful mutations are not followed by hidden reads
to fill them in.

## Reading Payments

To check an order, first authorize access to its saved payment ID and select the
client for its persisted merchant/environment. With `paymentID` and `amount`
loaded from that order, retrieve the financial resource:

```go
import "arkoren.dev/core/service/payment"

observed, err := payments.GetPayment(ctx, paymentID)
```

`GetPayment` only reads state. Before fulfilling, require a successful read,
the expected identity and amount (including currency), and successful collection.
With `errors` imported, the consequential checks are:

```go
if err != nil {
    return err
}
if observed.ID != paymentID || observed.Amount != amount {
    return errors.New("payment does not match order")
}
if observed.Status != payment.StatusSucceeded {
    return errors.New("payment is not collected")
}
```

Commit fulfillment once against the saved order. Pending, requires-action,
unknown, failed, and canceled payments cannot satisfy that success condition.
A browser return, SDK callback, active subscription, or successful API call
alone cannot replace this evidence.

Amounts use exact integer currency minor units. `NewMoney("EUR", 2500)` represents
EUR 25.00; `NewMoney("JPY", 2500)` represents JPY 2500. `ParseMoney` accepts exact
decimal text without rounding. `CurrencyExponent` lists supported currencies;
unknown currencies return `ErrUnsupported`, and providers can accept fewer.

## Request Values

Construct operation inputs with their typed `New*Request` functions rather than
assigning request fields. Required values are positional; optional settings use
construction callbacks. Published requests expose getters and are immutable.
Callbacks run once synchronously in order, even if required input is invalid;
a supplied nil callback panics, while an expanded empty/nil callback slice does
not supply an edit.

`Err()` preserves the first invalid construction input, even after a later
setting replaces it. Build a new request to correct it. `Validate(ctx)` checks
the context before construction validity, and facade calls validate before
dispatch. Valid request JSON preserves its field names; encoding invalid
requests fails, and decoding validates temporary state before replacing a value.
`Money` remains a plain value, not a builder.

## Choosing Flows

Use [Hosted Checkout](https://arkoren.dev/docs/guides/payments/hosted-checkout.md) for a provider-hosted approval page,
or [Mobile Payments](https://arkoren.dev/docs/guides/payments/mobile-payments.md) to prepare a typed client SDK handoff.
Both require server-owned pricing, durable operation keys, and provider evidence
before fulfillment. Use [Payment Refunds](https://arkoren.dev/docs/guides/payments/refunds.md) to return an explicit amount,
and [Recurring Subscriptions](https://arkoren.dev/docs/guides/payments/subscriptions.md) to enroll in an existing provider
plan and stop future renewals.

> [!WARNING]
> An interrupted mutation may already have taken effect. Preserve its original
> request, operation key, merchant/environment, and any returned references.
> A later error does not erase earlier uncertainty. Follow
> [Payment Recovery](https://arkoren.dev/docs/guides/payments/recovery.md) rather than submitting a new-key charge.

Focused fragments keep error results visible and omit routine propagation;
production code handles every error. Checks that determine ownership, recovery,
or fulfillment remain explicit. [Testing](https://arkoren.dev/docs/guides/payments/testing.md) demonstrates local outcomes
without moving money.

Provider notifications are optional, separate integrations described under
[Drivers](https://arkoren.dev/docs/guides/payments/drivers.md). They install no automatic capture or fulfillment path.
Mollie's [unsigned callbacks](https://arkoren.dev/docs/guides/payments/mollie-callbacks.md) require an authenticated lookup,
not trust in the incoming request.

For types, constructors, and validation rules, see the
[payment API](https://pkg.go.dev/arkoren.dev/core/service/payment).

---

Source: https://arkoren.dev/docs/guides/payments/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/payments/drivers)

# Payment Drivers

Choose Memory for local simulation, Stripe for hosted or native card payments,
PayPal for wallet Orders, or Mollie for hosted one-off card payments. Stripe and
PayPal also enroll buyers in existing recurring plans. Changing drivers does not
migrate resources, credentials, or customer agreements.

Install core from your application's module directory:

```sh
go get arkoren.dev/core
```

The fragments use `arkoren.dev/core/service/payment`. Load credentials from
server configuration, check construction errors, and begin with test/sandbox
credentials. None of these drivers needs lifecycle hooks, a worker, or a webhook
to make outbound calls. Construction performs no network I/O.

## Compatibility Matrix

This compares Arkoren's shipped operations and companions, not each vendor's
full product. Memory supplies local state, not a real buyer page or collection.

| Driver | Hosted buyer approval | Merchant completion | Payment lookup | Refund creation and lookup | Subscription enrollment | Immediate cancellation | Client preparation | Authenticated notifications | Facade call observations |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [Memory](#memory) | No — Non-routable handles. | Yes — Simulated. | Yes — Simulated. | Yes — Simulated. | Partial — No renewals. | Yes — Simulated. | Partial — Pending payment and inert token. | No | Yes — Opt-in, in-process only. |
| [Stripe](#stripe) | Yes | No — Observes hosted completion. | Yes | Yes | Partial — Existing recurring Price, quantity one. | Yes | Partial — Guest card intent; no return URL or device reference. | Yes — Separate receiver. | Yes — Opt-in, in-process only. |
| [PayPal](#paypal) | Yes | Yes — Order capture. | Partial — Captures only. | Partial — Captures only. | Partial — Existing Plan. | Partial — Active/suspended; canceled succeeds. | Partial — Native Order; creation-only risk data; no web/backend return URL. | Yes — Separate receiver. | Yes — Opt-in, in-process only. |
| [Mollie](#mollie) | Partial — One-off creditcard, automatic capture only. | No — ErrUnsupported. | Yes | Yes — Compound refund handle required. | No | No | No | No — Unsigned ID hints only. | Yes — Opt-in, in-process only. |

Approval is not collection, refund acceptance is not returned funds, and an
active subscription is not paid access. Cancellation is immediate, without
proration or refunds. Retain durable operation keys and respect
[provider-specific recovery limits](https://arkoren.dev/docs/guides/payments/recovery.md) before replaying any mutation.

## Memory

Use the bundled simulator when no external financial effect should occur. Each
instance has independent state that disappears on exit; it needs no credentials
or separate module.

```go
import (
    "arkoren.dev/core/service/payment"
    "arkoren.dev/core/service/payment/memorypayment"
)

simulator := memorypayment.New()
payments := payment.New(simulator)
```

There are no optional settings or native client. Use the
[local payment journey](https://arkoren.dev/docs/guides/payments/testing.md#simulating-payments) to supply approval and
settlement explicitly. Completion creates a pending payment; advancing time
does not settle it or collect a renewal.

Preparation creates a pending payment and inert `memory.authorization` token,
not a checkout or SDK credential. `ResolvePayment` supplies its result.
`ResolveSubscription` separately supplies subscription lifecycle, not invoice
payment. For ambiguous replies rather than simulated state, use the
[scripted fake](https://arkoren.dev/docs/guides/payments/testing.md#scripting-outcomes).

See the [memorypayment API](https://pkg.go.dev/arkoren.dev/core/service/payment/memorypayment).

## Stripe

Choose Stripe for hosted Checkout or guest native card intents. Obtain a Stripe
test-mode API key; recurring enrollment additionally needs an existing recurring
Price. One-time checkout uses an inline amount rather than a precreated Price.

```sh
go get arkoren.dev/driver/stripe
```

### Connecting Stripe

With `stripeKey` loaded from server configuration, create the backend and facade:

```go
import "arkoren.dev/driver/stripe/stripepayment"

backend, err := stripepayment.New(stripeKey)
payments := payment.New(backend)
```

The key must be nonempty without whitespace or controls. Invalid keys return
`payment.ErrInvalidRequest` before callbacks run. Valid-key callbacks run once
in order; nil callbacks and callback panics propagate. HTTP settings are checked
afterwards.

To use a shorter timeout, construct the backend with `net/http` and `time`:

```go
backend, err := stripepayment.New(stripeKey, func(connection *stripepayment.Builder) {
    connection.HTTPClient(&http.Client{Timeout: 8 * time.Second})
})
```

The default/zero timeout is 30 seconds; a negative timeout fails construction.
`New` copies HTTP settings, refuses redirects, and disables SDK retries and
standard HTTP mutation replays, logging, and telemetry. Borrowed transports stay
caller-owned and must not independently replay mutations.

`NewFrom` instead preserves the supplied SDK client's retry, redirect, logging,
and timeout policies. `Client()` exposes the retained native client; direct
native operations and their safety remain yours. The driver closes neither
client nor transport. Do not mutate shared clients during requests.

### Payment Flows

Follow [Hosted Checkout](https://arkoren.dev/docs/guides/payments/hosted-checkout.md) for a redirect or
[Recurring Subscriptions](https://arkoren.dev/docs/guides/payments/subscriptions.md) with a recurring Price at quantity
one. Stripe's hosted UI collects payment; `CompleteCheckout` only observes
completion and never charges again. Checkout Session, PaymentIntent, Refund,
and Subscription IDs retain their distinct native families.

For [Mobile Payments](https://arkoren.dev/docs/guides/payments/mobile-payments.md), `PreparePayment` creates an unconfirmed,
card-only PaymentIntent with automatic confirmation/capture settings. The
sensitive `stripe.payment_intent` authorization goes only to the authenticated
owner. Empty, web, iOS, and Android channels use the same guest profile. ReturnURL
and DeviceReference must be empty; configure client return routing in the SDK.
No Customer, CustomerSession, ephemeral key, or saved-card setup is supplied.

Refunds perform a currency-checking PaymentIntent preflight, then submit the
explicit refund without post-success hydration. Immediate subscription
cancellation requests neither proration nor a new final invoice. `NextBillingAt`
stays absent: item period boundaries do not establish another collection.
Dispatched ambiguous mutations can wrap `ErrOutcomeUnknown`; preserve the
original request/key even if a later call fails differently.

### Stripe Notifications

Notifications are optional and use `stripewebhook`, installed by the same vendor
module, independently of the payment backend. Configure a public HTTPS endpoint
in Stripe with snapshot event version `stripe.APIVersion` from
`github.com/stripe/stripe-go/v87`. Its endpoint signing secret is not the API key.

The fragments assume `r *router.Router` and an application-owned
`accept webhook.AcceptFunc[payment.Observation]` that durably accepts observations
and returns its error. Import `arkoren.dev/core/http/webhook` and construct the
source first:

```go
import "arkoren.dev/driver/stripe/stripewebhook"

source, err := stripewebhook.New(endpointSecret)
if err != nil {
    return err
}
```

Bind only needed events and mount the checked handler:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[stripewebhook.Envelope]) {
    receiver.Handle(stripewebhook.PaymentIntentSucceeded, accept)
})
if err != nil {
    return err
}
r.Post("/payments/stripe/events", handler)
```

Verification uses original bytes and timestamped signatures without network I/O.
The secret needs `whsec_`, a nonempty suffix, and no whitespace/controls before
callbacks run. Signature tolerance defaults to five minutes; change it with
`stripewebhook.Builder.Tolerance`. Zero restores the default; negative is invalid.
Freshness is not durable replay prevention.

Bound snapshot version/schema mismatches fail decoding retryably; they are not
authentication failures. Thin events and fetching missing relationships are not
supported. `Envelope.CreatedAt` is native Unix seconds (zero means absent), not
`time.Time`; JSON still uses `created`. Positive values populate portable
`Observation.OccurredAt`.

Definitions also cover Checkout completion, intent failure, refund updates,
subscription creation/update/deletion, and invoice paid/payment-failed/
invoice-payment-paid. Checkout/subscription lifecycle is not collection evidence;
`invoice.paid` can describe zero or externally satisfied invoices. Correlate
invoice references without guessing missing relationships. Events can repeat or
arrive out of order, and refunds can remain pending.

Apply the [receiver acceptance rules](#notification-acceptance) below. See the
[Stripe payment API](https://pkg.go.dev/arkoren.dev/driver/stripe/stripepayment)
and [Stripe webhook API](https://pkg.go.dev/arkoren.dev/driver/stripe/stripewebhook).

## PayPal

Choose PayPal for hosted or native wallet Orders. Create a developer application
and use sandbox merchant credentials. Subscription enrollment additionally needs
an existing account Plan; a Stripe Price is not interchangeable.

```sh
go get arkoren.dev/driver/paypal
```

### Connecting PayPal

Load the sandbox application's credentials from server configuration:

```go
import "arkoren.dev/driver/paypal/paypalpayment"

backend, err := paypalpayment.New(merchantID, merchantSecret)
payments := payment.New(backend)
```

Callbacks run once in order before nonblank credential validation; nil callbacks
panic. Sandbox is the default. To bound requests, use `net/http` and `time`:

```go
backend, err := paypalpayment.New(merchantID, merchantSecret, func(merchant *paypalpayment.Builder) {
    merchant.HTTPClient(&http.Client{Timeout: 8 * time.Second})
})
```

HTTP settings are copied, redirects refused, and a nonpositive timeout becomes
30 seconds. Transports/jars remain borrowed and are never closed. Fixed trusted
API origins and a private token cache require no refresh worker. Requests,
including failed mutations, are not automatically replayed; custom transports
must preserve that boundary. `HTTPClient()` exposes the actual client, not an
SDK; treat it as read-only during use.

Only after sandbox verification, use production credentials and
`merchant.Live(true)` in construction. That setting selects a live origin and
can move real money; merchant credentials, resources, and environment must match.

### Payment Flows

Use [Hosted Checkout](https://arkoren.dev/docs/guides/payments/hosted-checkout.md) and explicitly complete an approved
Order with a separately persisted completion key. An approved Order can expire
without capture; neither a notification decoder nor a return link captures it.
Coordinate the same completion operation across authorized POSTs and workers.

For [Mobile Payments](https://arkoren.dev/docs/guides/payments/mobile-payments.md), preparation returns a native
`paypal.order` action and a `CheckoutID`, not a financial PaymentID. Empty, iOS,
and Android channels work; web and nonempty ReturnURL are unsupported. No backend
approval link is required. Optional DeviceReference carries at most 255 visible
ASCII bytes as `PayPal-Client-Metadata-Id` on creation only. It is risk correlation,
never authorization. Client SDK approval must still precede server completion.

Preserve returned resource-family prefixes. Payment reads/refunds support
one-time Payments v2 captures, not subscription sales. Sale observations do not
enable deprecated sale lookup/refund; those operations return `ErrUnsupported`.
PayPal supports a subset of core currencies. HUF amounts must represent whole
forints using canonical two-decimal minor units; unsupported currencies and
fractional forints fail before authentication/payment I/O.

[Subscriptions](https://arkoren.dev/docs/guides/payments/subscriptions.md) activate through buyer approval. Cancellation
supports active/suspended subscriptions; already-canceled succeeds without a
mutation. There is no period-end scheduling, plan change, or proration.
Provider-supplied `NextBillingAt` and active status do not establish payment.

Malformed successful GET JSON wraps `ErrUnavailable` without outcome uncertainty;
semantic snapshot failures retain their specific validation/capability errors.
Unusable OAuth responses also wrap `ErrUnavailable` before business dispatch.
Neither resolves an earlier uncertain mutation. Successful mutation decoding
failures remain outcome-unknown. `paypalpayment.Error` exposes safe status and
machine-token fields, but unwrapped causes still require redaction.

### PayPal Notifications

Use the separate `paypalwebhook` package from the same module. Configure a public
HTTPS webhook endpoint and selected event types in the matching sandbox account.
The webhook registration ID and merchant credentials must share its environment.
Use `r *router.Router`, import `arkoren.dev/core/http/webhook`, and supply an
application-owned `accept webhook.AcceptFunc[payment.Observation]` that durably
accepts observations and returns its error:

```go
import "arkoren.dev/driver/paypal/paypalwebhook"

source, err := paypalwebhook.New(merchantID, merchantSecret, registrationID)
if err != nil {
    return err
}
```

Bind capture completion when your application needs collection observations:

```go
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[paypalwebhook.Envelope]) {
    receiver.Handle(paypalwebhook.PaymentCaptureCompleted, accept)
})
if err != nil {
    return err
}
r.Post("/payments/paypal/events", handler)
```

Production is selected separately with `paypalwebhook.Builder.Live(true)` and
matching production registration/credentials. Its HTTPClient setting copies
client settings, refuses redirects, and defaults nonpositive timeouts to 30
seconds; its transport stays caller-owned.

The source verifies original bytes remotely at a fixed PayPal origin using the
request context. It never fetches arbitrary certificate URLs or enriches events.
Verifier outages are retryable, not invalid signatures. Authentication precedes
duplicate suppression, so repeated events can cause verification requests.
Delivery identity is the event ID, not a transmission attempt ID.

Other definitions cover Order approval, capture outcomes, sale collection/refund,
refund progress, and subscription lifecycle. Binding `CheckoutOrderApproved`
does not capture: your authorized worker calls `CompleteCheckout` with the saved
completion key. Missing relationships stay absent; events can repeat or arrive
out of order. Mock events do not prove signature verification; validate actual
sandbox deliveries separately.

Apply the [receiver acceptance rules](#notification-acceptance). See the
[PayPal payment API](https://pkg.go.dev/arkoren.dev/driver/paypal/paypalpayment)
and [PayPal webhook API](https://pkg.go.dev/arkoren.dev/driver/paypal/paypalwebhook).

## Mollie

Choose Mollie for hosted, one-off `creditcard` payments with automatic capture
and explicit refunds. Create a website profile, enable that method, and obtain
a test profile API key. Other methods, manual capture, customer/mandate setup,
recurring APIs, and native SDK preparation are not exposed.

```sh
go get arkoren.dev/driver/mollie
```

### Connecting Mollie

With the profile key loaded from server configuration:

```go
import "arkoren.dev/driver/mollie/molliepayment"

backend, err := molliepayment.New(profileKey)
payments := payment.New(backend)
```

Keys need a `test_` or `live_` prefix and alphanumeric suffix. Their prefix selects
mode at the fixed API origin; OAuth/organization credentials are unsupported.
A live key can move money. Callbacks run once in order, even for invalid keys;
nil callbacks panic.

Callbacks are disabled by default. To receive status hints, configure a reachable
absolute HTTPS URL without credentials/fragments, using `net/http` and `time`:

```go
backend, err := molliepayment.New(profileKey, func(merchant *molliepayment.Builder) {
    merchant.WebhookURL("https://shop.example/payments/mollie")
    merchant.HTTPClient(&http.Client{Timeout: 8 * time.Second})
})
```

This installs no receiver or authentication. Nonpositive/omitted timeouts become
30 seconds. HTTP settings are copied, redirects refused, and borrowed
transport/jar objects never closed. `HTTPClient()` exposes the actual client;
do not mutate it during use or weaken redirect policy. Custom transports must
neither replay mutations nor log credentials.

### Payment Flows

Use [Hosted Checkout](https://arkoren.dev/docs/guides/payments/hosted-checkout.md). The driver fixes `method: creditcard`,
`sequenceType: oneoff`, and `captureMode: automatic`. An empty description becomes
`Payment`; Reference becomes correlation metadata, not idempotency.
Checkout/payment IDs refer to the same `tr_` resource using different Go ID types.
`NeedsCompletion` is false and explicit completion returns `ErrUnsupported`.
Only native `paid` maps to succeeded; `authorized` and `pending` are not collection.

Use [Payment Refunds](https://arkoren.dev/docs/guides/payments/refunds.md) with the original currency. Mollie enforces
currency and remaining balance without hidden preflight or post-mutation reads.
Save the entire compound refund handle: it includes the parent payment needed
for lookup. Queued/pending/processing refunds remain pending; only `refunded`
succeeds, and unfamiliar native states stay unknown.

Subscriptions and preparation validate portable input, then return
`ErrUnsupported` without HTTP. This does not disable hosted one-off checkout.
Mollie Components, native wallets, and Mollie's recurring product are not shipped
capabilities of this driver.

> [!WARNING]
> Mollie caches POST responses for one hour per exact API credential. Save the
> request, key, and credential identity before dispatch. Credential rotation or
> replay after expiry can create another effect, even within the same account.
> See [Payment Recovery](https://arkoren.dev/docs/guides/payments/recovery.md#replaying-requests) before replaying.

There is one mutation attempt without automatic retries. Transport loss,
conflicts, server failures, redirects, and unusable mutation successes can wrap
`ErrOutcomeUnknown`. Malformed/unusable successful GET snapshots instead wrap
`ErrUnavailable`, including missing required amounts or mismatched identities;
transport/non-2xx failures retain their categories. None clears earlier uncertainty.

`molliepayment.Error.Status` retains a non-2xx status or zero if none was retained.
Zero can follow a successful HTTP reply that could not be read/decoded; it does
not prove no reply arrived. Errors omit provider bodies. Driver `fmt`/`slog`
formatting yields `Driver{REDACTED}`, hiding owned credentials and callback URL,
but borrowed HTTP objects and explicitly unwrapped causes remain your redaction
responsibility.

### Mollie Callbacks

Classic callbacks are unsigned form POSTs with an `id` hint. Anyone can forge
them; there is no shipped `molliewebhook` or authenticated Source. Rate-limit
and bound the route, accept exactly one ID, check its persisted merchant/test-live
association, and read authoritative provider state with that merchant's client.

With `mux *http.ServeMux` and your application handler, register a plain route:

```go
mux.HandleFunc("POST /payments/mollie", handleMollie)
```

[Mollie Callbacks](https://arkoren.dev/docs/guides/payments/mollie-callbacks.md) shows parsing, ownership, lookup, and durable
acceptance. Unknown IDs receive an acknowledgement without order disclosure.
Require succeeded state and matching order amount/currency before fulfillment;
duplicate hints must not duplicate effects. Refund callbacks carry the payment
ID, so reconcile the compound refund handles you already saved. Next-gen
webhooks are outside scope.

See the [Mollie payment API](https://pkg.go.dev/arkoren.dev/driver/mollie/molliepayment).

## Notification Acceptance

Stripe/PayPal receivers use the [HTTP webhook receiver](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md).
By default, it answers 200 after acceptance, including unbound authenticated
types; 503 for decode/accept errors; 400 for rejected authentication; and 413
for an oversized body. The receiver commits its failure response before returning
an error. Receipt 4xx errors match `webhook.ErrRejected`; 503 errors match
`webhook.ErrUnavailable`, retaining causes. An outer error mapper cannot rewrite
that response, and unwrapped causes require redaction.

Accept observations durably before acknowledging. A 200 does not establish
asynchronous completion or successful collection. `webhook.Delivery` carries
native event identity; `payment.Observation` carries facts/time. Use `Event.Map`
for application queue DTOs. Scope deduplication by provider, merchant,
environment, and consumer; cache deduplication replaces neither a durable
inbox/outbox nor operation keys.

## Local Monitoring

To observe facade calls locally, supply an in-process event bus at construction.
With `driver payment.Driver` and `bus *event.Bus` already configured:

```go
payments := payment.New(driver, func(client *payment.Builder) {
    client.Events(bus)
})
```

Register listeners on `payment.OperationCompleted` and `OperationFailed` using
the [event bus](https://arkoren.dev/docs/guides/events/overview.md). Their `payment.OperationEvent` contains
only a fixed operation name and bounded `ErrorType`, not IDs, amounts, URLs,
keys, secrets, or arbitrary error text. Omitted/nil Events disables observations;
direct driver calls bypass them.

Each driver-returned call emits once: nil error completes, non-nil error fails.
Validation failures, missing drivers, and pre-canceled calls emit nothing.
`payment.ErrOutcomeUnknown` takes precedence over joined error classes, while
the returned error retains its causes. Listeners run synchronously after
telemetry with the operation context; their errors/panics cannot alter the API
result, but slow listeners delay return.

These best-effort observations include reads and replays. They prove neither
money movement nor durable reconciliation and must not trigger fulfillment.
Provider notifications and application-owned operation records remain separate.
For facade configuration, see the
[payment API](https://pkg.go.dev/arkoren.dev/core/service/payment).

---

Source: https://arkoren.dev/docs/guides/payments/hosted-checkout.md
[HTML documentation](https://arkoren.dev/docs/guides/payments/hosted-checkout)

# Hosted Checkout

Hosted checkout sends the buyer to a provider's approval page. Your backend
creates the flow, saves its identity, and later checks whether the provider
collected the expected payment. Use a [configured client](https://arkoren.dev/docs/guides/payments/drivers.md) named
`payments`, a caller context `ctx`, and `arkoren.dev/core/service/payment`.

## Starting Checkout

Authenticate the buyer and authorize their saved order before creating checkout.
Derive the amount, currency, reference, and redirect destinations from server
data, not request-supplied prices or arbitrary URLs. Save the immutable request,
merchant/environment association, and a unique operation key before dispatch.
Use one key for this logical creation, not a new key for each HTTP retry.

For a server-priced book order, construct an explicit EUR 25.00 request:

```go
amount, err := payment.NewMoney("EUR", 2500)
request := payment.NewCheckoutRequest(
    amount, "https://shop.example/payments/return", "book-184-checkout")
```

`NewMoney` uses exact minor units; provider currency limits still apply.
Request constructors perform no I/O. Requests are immutable; `Err()` exposes
construction errors, and facade calls validate before dispatch. To add
correlation or a cancellation destination, construct this alternative before
saving the request:

```go
request := payment.NewCheckoutRequest(amount,
    "https://shop.example/payments/return", "book-184-checkout",
    func(details *payment.CheckoutRequestBuilder) {
        details.Description("Book order")
        details.Reference("book-184")
        details.CancelURL("https://shop.example/cart")
    })
```

Reference is optional correlation, not authorization, idempotency, or a searchable
provider identity. Hosted redirect validation permits HTTPS and loopback HTTP;
it does not decide whether your application trusts a destination.

> [!WARNING]
> Creation can take effect before a response is lost. Retain the original
> request/key and any references returned with an error. A recording failure or
> later rejection does not prove no effect occurred. Reconcile through
> [Payment Recovery](https://arkoren.dev/docs/guides/payments/recovery.md), not a fresh-key checkout.

Submit the saved request:

```go
checkout, err := payments.CreateCheckout(ctx, request)
```

After checking the result, durably associate `checkout.ID` with the order,
initiating account, and merchant/environment before redirecting to
`checkout.ApprovalURL`. Treat the approval URL as sensitive and exclude it from
logs. Preserve any returned payment references even when the response is partial;
creation does not perform a follow-up read to fill missing fields.

## Returning Buyers

A browser return is navigation, not payment evidence. Authorize the returning
caller against the saved order and use its persisted `checkoutID`, never a
replacement provider ID supplied by the browser:

```go
checkout, err := payments.GetCheckout(ctx, checkoutID)
```

This read neither captures funds nor activates an agreement. A read-only return
or status endpoint can report the observation without triggering a mutation.
Keep earlier references if a later snapshot omits them; missing relationships
do not authorize casting another resource's ID.

## Explicit Completion

PayPal Orders need merchant capture after buyer approval. Stripe hosted Checkout
collects through its UI; its completion operation only observes. Mollie's
automatic-capture flow reports `NeedsCompletion == false` and explicit completion
returns `ErrUnsupported`.

When completion is required, use an authorized POST or coordinated worker, not
a status GET or browser-return GET. Save a separate `completionKey` and coordinate
it across all completion paths. With the saved `checkoutID`, expected `amount`,
and `errors` imported, first check the retrieved checkout:

```go
if err != nil {
    return err
}
if checkout.ID != checkoutID || checkout.Amount == nil || *checkout.Amount != amount {
    return errors.New("checkout does not match order")
}
if checkout.NeedsCompletion && checkout.Status != payment.CheckoutStatusApproved {
    return errors.New("checkout is not approved")
}
```

Only then submit required completion with the persisted key:

```go
if checkout.NeedsCompletion {
    checkout, err = payments.CompleteCheckout(ctx,
        payment.NewCompleteCheckoutRequest(checkoutID, completionKey))
}
```

Record the result and any returned references before continuing, including those
returned with errors. Completion can have an uncertain outcome too: preserve
the original key and reconcile instead of issuing another capture. An approved
PayPal Order can expire without capture if no application path completes it.

## Verifying Collection

Completion and `NeedsCompletion == false` do not prove success. For this paid
order, use the financial `PaymentID` observed and saved from checkout, under its
same merchant/environment. If none has been observed, keep the attempt unresolved;
there is no universal search by operation key or reference.

```go
observed, err := payments.GetPayment(ctx, paymentID)
if err != nil {
    return err
}
if observed.ID != paymentID || observed.Amount != amount {
    return errors.New("payment does not match order")
}
if observed.Status != payment.StatusSucceeded {
    return errors.New("payment is not collected")
}
```

Only a matching succeeded payment satisfies collection. Commit fulfillment once
against the authorized order; pending, unknown, or requires-action states do not
qualify. Checkout `StatusNotRequired` can describe a no-payment-required flow,
not newly collected funds. Your application owns when to retrieve state and
when to fulfill; optional [provider notifications](https://arkoren.dev/docs/guides/payments/drivers.md) install no polling
or automatic fulfillment.

To return funds later, use [Payment Refunds](https://arkoren.dev/docs/guides/payments/refunds.md). For request types and
status definitions, see the
[payment API](https://pkg.go.dev/arkoren.dev/core/service/payment).

---

Source: https://arkoren.dev/docs/guides/payments/mobile-payments.md
[HTML documentation](https://arkoren.dev/docs/guides/payments/mobile-payments)

# Mobile Payments

Use `PreparePayment` when a native client presents the payment UI instead of a
hosted redirect. Your Go backend prepares a typed action for the client, then
checks provider state before fulfilling the order. The client SDK and its UI
remain your application's responsibility. These provider handoffs do not
implement App Store or Google Play billing.

Start with a [configured client](https://arkoren.dev/docs/guides/payments/drivers.md) named `payments`, caller context
`ctx`, and `arkoren.dev/core/service/payment`. Stripe supplies a guest card intent;
PayPal supplies a native Order needing later server capture. Mollie does not
support preparation. A channel names a client platform, not wallet acceptance.

## Preparing Handoffs

Authenticate the caller, authorize the order, and calculate its price on the
server. Persist one immutable request, operation key, and merchant/environment
binding before preparation. A client-supplied price or provider ID must not choose
what gets charged. For an authorized EUR 25.00 order on iOS:

```go
amount, err := payment.NewMoney("EUR", 2500)
request := payment.NewPreparePaymentRequest(amount, "book-184-native",
    func(details *payment.PreparePaymentRequestBuilder) {
        details.Channel(payment.ClientChannelIOS)
        details.Reference("book-184")
    })
```

Use `ClientChannelAndroid` for Android. Reference is optional correlation, not
authorization or idempotency. Leave ReturnURL and DeviceReference unset unless
the [selected profile](https://arkoren.dev/docs/guides/payments/drivers.md) supports them. Core callback-URI syntax checks
do not establish application ownership; any configured destination still needs
your server-side scheme/host allowlist.

> [!WARNING]
> Preparation can create a provider resource before the response or local save
> fails. Preserve the original request/key and any returned references, even
> with an error. Do not replace an unresolved attempt with a fresh key; follow
> [Payment Recovery](https://arkoren.dev/docs/guides/payments/recovery.md).

Submit the saved request:

```go
prepared, err := payments.PreparePayment(ctx, request)
```

`prepared.Action` is the typed client handoff. `PaymentID`, `CheckoutID`, and
`SessionID` are optional observed references, not interchangeable identities.
Save those actually returned and the protected action before delivering it.
Preparation does not confirm, capture, or prove payment.

## Protecting Actions

Deliver the saved preparation only to the authenticated order owner over HTTPS.
With `encoding/json/v2` imported, encode the typed result directly:

```go
body, err := json.Marshal(prepared)
```

JSON intentionally contains handoff credentials. Send it with
`Cache-Control: no-store`, exclude response bodies from logs/analytics, and apply
access control on every resume request. Formatting/`slog` redaction does not make
JSON safe to log. No backend driver downcast is needed to serialize the action.

For reconnects, return the saved action only while its protocol and lifetime
permit reuse. There is no universal preparation lookup or expiry API. Stripe
returns a sensitive `stripe.payment_intent` authorization; PayPal returns a
`paypal.order` action. Neither an SDK success callback nor a return link proves
provider approval or successful collection.

## Completing Orders

If preparation returned only a financial PaymentID, skip checkout completion.
For a saved CheckoutID, first read provider state under the same persisted
merchant/environment. The following uses saved `checkoutID`, expected `amount`,
and `errors`:

```go
checkout, err := payments.GetCheckout(ctx, checkoutID)
if err != nil {
    return err
}
if checkout.ID != checkoutID || checkout.Amount == nil || *checkout.Amount != amount {
    return errors.New("checkout does not match order")
}
if checkout.NeedsCompletion && checkout.Status != payment.CheckoutStatusApproved {
    return errors.New("checkout is not approved")
}
```

An SDK callback cannot bypass these checks. Run required completion from an
authorized POST or coordinated worker, never a read-only status/return GET.
Use the separate completion key already saved for this attempt:

```go
if checkout.NeedsCompletion {
    checkout, err = payments.CompleteCheckout(ctx,
        payment.NewCompleteCheckoutRequest(checkoutID, completionKey))
}
```

Persist results and any returned payment reference, including with errors.
Preserve previously observed references when a later reply omits them.
Completion errors can be outcome-unknown; keep the same operation/key pending
reconciliation. `NeedsCompletion == false` means no explicit completion is
required, not that money was collected.

## Verifying Payments

Use the saved financial `paymentID` rather than a checkout/session ID. With
the same server-derived `amount`, require an authoritative matching observation:

```go
observed, err := payments.GetPayment(ctx, paymentID)
if err != nil {
    return err
}
if observed.ID != paymentID || observed.Amount != amount {
    return errors.New("payment does not match order")
}
if observed.Status != payment.StatusSucceeded {
    return errors.New("payment is not collected")
}
```

Commit fulfillment once against the saved order. Pending, unknown, failed,
canceled, and requires-action states do not qualify. If no financial ID has
been observed, retain the unresolved attempt; never manufacture one from another
resource ID. A read-only status endpoint reports state without fulfilling.

## Recovering Attempts

Arkoren neither persists attempts nor automatically retries these mutations.
Coordinate concurrent preparation and completion calls in your application.
A timeout, expired claim, or failed local save can follow a provider effect;
preserve context errors and inspect `errors.Is(err, payment.ErrOutcomeUnknown)`.
Keep earlier uncertainty even when a later same-key call is rejected or fails
before dispatch. The latest error does not authorize another charge.

[Payment Recovery](https://arkoren.dev/docs/guides/payments/recovery.md) explains bounded reads and provider-scoped replay.
[Payment Refunds](https://arkoren.dev/docs/guides/payments/refunds.md) uses the same facade with the saved financial ID;
a pending refund is not returned money. Use [Testing](https://arkoren.dev/docs/guides/payments/testing.md) for scripted
preparation, approval, completion, and uncertain replies, and test ownership,
durable recording, protected resume, and one-time fulfillment separately.
Sandbox/device checks are needed for the actual SDK approval boundary.

For typed actions and request options, see the
[payment API](https://pkg.go.dev/arkoren.dev/core/service/payment).

---

Source: https://arkoren.dev/docs/guides/payments/refunds.md
[HTML documentation](https://arkoren.dev/docs/guides/payments/refunds)

# Payment Refunds

A refund is a separate resource whose progress can outlast the request that
created it. Use a [configured client](https://arkoren.dev/docs/guides/payments/drivers.md) named `payments`, caller context
`ctx`, and the original financial payment ID saved against your order. Import
`arkoren.dev/core/service/payment` for the request and status types.

## Choosing Amounts

Authorize the refund against the saved order/account and select its persisted
merchant and test/live environment. Do not accept an arbitrary provider ID from
the caller. Calculate the refund under your server-side policy and keep its
currency equal to the original payment's currency.

For EUR 5.00 of an existing EUR payment, construct an exact positive amount:

```go
refundAmount, err := payment.NewMoney("EUR", 500)
```

A full refund also requires an explicit positive amount. Zero does not mean
“the remainder.” The provider enforces the remaining refundable balance,
including concurrent requests; a locally calculated balance is not a reservation.

## Submitting Refunds

Give this logical refund its own durable operation key, distinct from checkout
and completion keys. Save the immutable request, original payment ID, and
merchant/environment before dispatch:

```go
request := payment.NewRefundRequest(paymentID, refundAmount, "book-184-refund-1")
```

Reuse that key only for the identical request within the provider's documented
scope and retention. A second intended partial refund is a different operation;
it must not become a way to repeat an unresolved first refund.

> [!WARNING]
> A failed refund reply can follow an accepted refund. Keep the original
> request/key and any returned refund references even with an error. A later
> rejection or read failure does not settle that earlier attempt. Follow
> [Payment Recovery](https://arkoren.dev/docs/guides/payments/recovery.md) rather than issuing a fresh-key refund.

Submit the saved request:

```go
refund, err := payments.RefundPayment(ctx, request)
```

Record the result before telling the buyer it completed. An accepted
`RefundStatusPending` is normal provider progress, not returned money and not
an outcome-unknown error. The facade does not perform a hidden balance lookup
or post-mutation read; a driver can document a required currency preflight.

## Reading Progress

Save the entire `refund.ID`, not a token extracted from its value. To obtain a
later observation, read the saved `refundID` without resubmitting the refund:

```go
observed, err := payments.GetRefund(ctx, refundID)
```

After a successful read, reconcile the returned identity, expected amount, and
any observed payment relationship against your saved operation. Missing
relationships remain absent rather than being guessed.

| Status | Interpretation |
| --- | --- |
| `RefundStatusSucceeded` | The provider confirms refund success. |
| `RefundStatusPending` | The refund is not finalized. |
| `RefundStatusRequiresAction` | Further action is required; funds are not confirmed returned. |
| `RefundStatusFailed` | A known unsuccessful refund outcome. |
| `RefundStatusCanceled` | Canceled before success. |
| `RefundStatusUnknown` | No mapped state; do not treat it as success. |

Your application chooses when to read and how to notify the buyer. Provider
notifications can repeat or arrive out of order, so persist reconciliation
idempotently. Refunding a payment does not cancel a recurring agreement;
[subscription cancellation](https://arkoren.dev/docs/guides/payments/subscriptions.md#stopping-renewals) is separate.

## Provider Limits

Stripe checks the PaymentIntent's currency before submitting an explicit refund.
PayPal refunds Payments v2 captures only; subscription sale lookup/refund returns
`ErrUnsupported`. A sale ID cannot be substituted for a capture ID.

Mollie's opaque compound refund handle carries the parent payment needed for
lookup. Preserve it verbatim. Its queued/pending/processing states remain pending;
only `refunded` succeeds. Mollie caches POST replies for one hour per exact
credential, so credential rotation or replay after expiry can duplicate an
unresolved refund. Its classic callback carries a payment ID, not this compound
handle: read the refund IDs already saved by your application.

Use [Testing](https://arkoren.dev/docs/guides/payments/testing.md) to script pending, succeeded, and uncertain replies
without financial effects. For request and result types, see the
[payment API](https://pkg.go.dev/arkoren.dev/core/service/payment).

---

Source: https://arkoren.dev/docs/guides/payments/subscriptions.md
[HTML documentation](https://arkoren.dev/docs/guides/payments/subscriptions)

# Recurring Subscriptions

Provider-managed subscriptions let the provider collect renewals instead of
your application scheduling repeated checkouts. Arkoren starts enrollment,
reads agreement state, and requests immediate cancellation. Your application
still owns billing-account authorization and paid-access policy.

Use a [configured client](https://arkoren.dev/docs/guides/payments/drivers.md) named `payments`, caller context `ctx`,
and `arkoren.dev/core/service/payment`. Stripe and PayPal support this flow;
Mollie's driver does not. Memory simulates lifecycle without collecting renewals.

## Selecting Plans

Create the recurring Price or Plan outside Arkoren, then map your application's
offering to that provider reference on the server. Stripe accepts an existing
recurring Price at quantity one; PayPal accepts its account's Plan ID. These are
not interchangeable catalog records.

Authenticate the buyer and authorize enrollment for the saved billing account.
Choose the plan, correlation reference, and return/cancel destinations from
trusted server configuration. A request-supplied plan or arbitrary redirect must
not select the agreement your backend creates.

## Starting Enrollment

Save the immutable enrollment request, a unique operation key, and the selected
merchant/test-live environment before calling the provider. With `planReference`
loaded from your server-owned mapping, build the approval request:

```go
request := payment.NewSubscriptionCheckoutRequest(planReference,
    "https://shop.example/account/billing", "membership-184-enroll")
```

If you need cancellation navigation and correlation, use this alternative
construction before saving the request:

```go
request := payment.NewSubscriptionCheckoutRequest(planReference,
    "https://shop.example/account/billing", "membership-184-enroll",
    func(details *payment.SubscriptionCheckoutRequestBuilder) {
        details.CancelURL("https://shop.example/membership")
        details.Reference("membership-184")
    })
```

Reference is correlation only, not authorization or idempotency. The constructor
does not check plan existence, create a catalog entry, or grant access.

> [!WARNING]
> Enrollment can take effect before its response or local recording fails.
> Preserve the original request/key and any references returned with an error.
> A later same-key rejection does not resolve earlier uncertainty. Use
> [Payment Recovery](https://arkoren.dev/docs/guides/payments/recovery.md) instead of creating another agreement.

Submit the saved request:

```go
enrollment, err := payments.CreateSubscriptionCheckout(ctx, request)
```

After checking the result, save `enrollment.ID` against the initiating account
before sending the buyer to its sensitive `ApprovalURL`. Persist any observed
`SubscriptionID` too. Stripe can allocate that ID only after checkout completes;
PayPal can allocate it while buyer approval is pending. Missing IDs stay absent.

## Reading Agreements

On return, authorize the caller and read the saved checkout with `GetCheckout`.
The return URL is not approval evidence. PayPal subscription approval activates
the agreement without a separate activate call; its completion operation observes
approval rather than capturing a one-time Order. No driver polls for approval.

Once a subscription ID has actually been observed, read it with the same
merchant/environment client:

```go
subscription, err := payments.GetSubscription(ctx, subscriptionID)
```

`SubscriptionStatusActive` describes lifecycle, not successful payment or a
paid-through entitlement. The provider's `NextBillingAt`, when present, is a
schedule observation, not proof of the last collection. Stripe leaves that
field absent; PayPal can supply it. Your application decides access and grace
periods using evidence appropriate to its policy.

Optional [provider notifications](https://arkoren.dev/docs/guides/payments/drivers.md) can supply lifecycle and collection
observations, but they neither grant access nor fill missing relationships.
Stripe `invoice.paid` can include zero or externally satisfied invoices; correlate
actual invoice/payment references rather than assuming every active agreement
or completed checkout represents newly collected money. PayPal sale observations
do not enable portable sale lookup/refund.

## Stopping Renewals

Authorize cancellation against the saved billing account and its observed
subscription ID. Record the intended cancellation durably before dispatch;
the cancellation request has no caller-supplied idempotency-key field.

```go
request := payment.NewCancelSubscriptionRequest(subscriptionID,
    func(details *payment.CancelSubscriptionRequestBuilder) {
        details.Reason("Membership canceled by owner")
    })
```

This requests immediate cancellation, not an end-of-period change. It neither
refunds prior payments nor requests proration, and it does not decide application
access. Existing liabilities and in-flight payments can remain. Plan changes and
scheduled cancellation are outside the contract.

```go
err := payments.CancelSubscription(ctx, request)
```

Stripe requests no new final invoice; PayPal accepts active/suspended states
and treats already-canceled as success without another mutation. Unsupported
states fail explicitly. A successful cancellation has no mandatory follow-up
read that could obscure that success.

If cancellation returns an uncertain outcome, preserve the intended operation
and read the known subscription to reconcile. Do not assume that a failed reply
means renewals continue or blindly replay an effectful request. See
[Payment Recovery](https://arkoren.dev/docs/guides/payments/recovery.md) for the broader error policy.

For subscription types and validation, see the
[payment API](https://pkg.go.dev/arkoren.dev/core/service/payment).

---

Source: https://arkoren.dev/docs/guides/payments/recovery.md
[HTML documentation](https://arkoren.dev/docs/guides/payments/recovery)

# Payment Recovery

A provider can accept a payment operation even when your application receives
no usable reply. Recovery means finding what happened to that saved operation,
not assuming the error permits a new attempt. This applies to checkout creation,
preparation, enrollment, completion, refunds, and cancellation.

## Recording Attempts

Before dispatch, save the authorized application account/order, immutable request,
logical operation identity, provider, merchant, and test/live environment. Save
the operation key where the request supports one. Cancellation has no portable
caller-key field, but still needs an application-owned operation record.

Coordinate concurrent callers so browser retries, workers, return paths, and
notifications do not start independent mutations for one intended effect.
Persist every observed resource ID and result, including references returned
with an error. A timeout, expired application claim, or failed local save is not
evidence that the provider did nothing.

Keep the exact credential identity when it affects replay scope, without placing
the credential itself in ordinary logs. Protect saved approval URLs and client
handoff credentials as secrets. Arkoren supplies no durable ledger, transaction
coordinator, local billing projection, or background reconciliation worker.

## Preserving Uncertainty

`errors.Is(err, payment.ErrOutcomeUnknown)` means a dispatched mutation may have
taken effect. Cancellation after dispatch can retain both outcome uncertainty
and the caller's context error. Pre-dispatch cancellation alone has no effect
from that call; it says nothing about an earlier call.

> [!WARNING]
> A later same-key rejection or pre-dispatch failure does not resolve an earlier
> uncertain attempt. Preserve the original request/key and unresolved state.
> Neither the latest error nor an unavailable read authorizes a fresh-key charge,
> refund, or fulfillment.

Model that uncertainty separately from the latest error. For example, an
application-owned saved checkout record can include these fields:

```go
import "arkoren.dev/core/service/payment"

type CheckoutAttempt struct {
    Request   payment.CheckoutRequest
    Checkout  payment.CheckoutID
    Uncertain bool
}
```

This is a storage-model fragment, not an Arkoren persistence API. Store its
account/merchant/environment association alongside it. With `attempt` loaded
under that association and `errors` imported, preserve uncertainty after a call:

```go
if errors.Is(err, payment.ErrOutcomeUnknown) {
    attempt.Uncertain = true
}
```

Persist that update with the observed references and outcome. Do not assign
`attempt.Uncertain = errors.Is(err, payment.ErrOutcomeUnknown)`: a later known
failure would erase an earlier unknown effect. Only reconciliation evidence
can resolve the saved uncertainty. Preserve context error identity with
`errors.Is` as well; do not infer dispatch state from error text.

## Reading Evidence

If a resource ID is known, read that resource using its persisted provider,
merchant, and environment. `GetCheckout`, `GetPayment`, `GetRefund`, and
`GetSubscription` are reads, not hidden capture or settlement operations.
Match observations against the saved target, amount/currency where applicable,
and account association before changing application state.

A succeeded payment can satisfy a matching order's collection condition;
pending/unknown cannot. Refund progress is independent: pending means accepted
work is not finalized, not that the mutation outcome is unknown. Subscription
lifecycle is not evidence of paid access. Preserve previously observed
relationships if a later snapshot omits them.

Choose bounded current-state reads through your application's recovery policy,
not an unbounded polling loop. A failed or not-found read does not by itself
settle an earlier uncertain mutation. A successful HTTP status with an unusable
body is not a valid financial observation either.

If creation returned no ID, the contract has no list/search operation, universal
preparation/session lookup, or lookup by operation key or Reference. Correlation
text does not make a missing resource discoverable. Use supported provider
replay or provider-specific investigation rather than casting another ID.

## Replaying Requests

Replay only the identical saved request/key within that operation's documented
provider scope and retention. Keep merchant/environment fixed. Do not treat a
provider idempotency mechanism as permanent exactly-once processing or assume
all operations share its guarantees.

Stripe and PayPal replay behavior is provider/operation-specific; verify the
relevant [Stripe idempotency policy](https://docs.stripe.com/api/idempotent_requests)
or [PayPal idempotency policy](https://developer.paypal.com/api/rest/reference/idempotency/)
before choosing a recovery window. The portable contract provides no common
retention duration or guarantee that every endpoint supports a replay key.

Mollie's [POST response cache](https://docs.mollie.com/reference/api-idempotency)
lasts one hour per exact API credential. Rotating credentials, even within one
account, loses that replay scope. Reusing a key after expiry can create a new
effect, particularly another partial refund. Retain the credential association
and original dispatch timing before considering replay.

Outside a supported replay window, investigate through the provider's dashboard
or a provider-specific process before authorizing a new attempt. Do not turn
missing evidence into permission to retry with a new key.

[Driver-owned clients](https://arkoren.dev/docs/guides/payments/drivers.md) do not automatically replay mutations.
Stripe `NewFrom` preserves the supplied SDK/HTTP policies instead; configure
those deliberately. Custom transports must not add mutation replay independently.
No driver performs post-success hydration to fill missing fields, although a
documented preflight read can guard currency or lifecycle.

## Classifying Errors

Use `errors.Is`/`errors.As`, not message matching. Classify the current call
without erasing the operation's saved history:

| Error | Response |
| --- | --- |
| `ErrInvalidRequest`, `ErrUnsupported` | Correct input or choose a supported operation; waiting does not fix it. |
| `ErrUnauthorized`, `ErrForbidden` | Fix provider credentials/permissions; buyer approval does not fix provider authorization. |
| `ErrDeclined` | Use an application-owned buyer recovery path for this known rejection. |
| `ErrInvalidState` | Reconcile lifecycle before considering another mutation. |
| `ErrRateLimited`, `ErrUnavailable` | Apply a bounded provider-aware policy; do not automatically repeat a charge. |
| `ErrNotFound` | Recheck the persisted identity/scope; it does not resolve an earlier ambiguous effect. |
| `ErrOutcomeUnknown` | Preserve the original operation and reconcile its possible effect. |

For PayPal, malformed successful GET JSON and unusable OAuth replies wrap
`ErrUnavailable` without adding `ErrOutcomeUnknown`. Semantic GET snapshot
failures keep their specific validation/capability errors. For Mollie, malformed
or unusable successful GET snapshots wrap `ErrUnavailable`. Successful mutation
decoding failures retain outcome uncertainty in both drivers. A pre-dispatch
OAuth failure can leave an earlier same-key attempt unresolved.

Mollie's retained native error status can be zero after a successful response
that could not be read or decoded. Zero does not prove that no response arrived.
Safe driver error messages do not sanitize explicitly unwrapped causes; redact
those and borrowed HTTP diagnostics before logging.

Use [Testing](https://arkoren.dev/docs/guides/payments/testing.md#testing-uncertainty) to exercise a lost reply followed by
a later rejection without moving money. For error identities and operation
contracts, see the [payment API](https://pkg.go.dev/arkoren.dev/core/service/payment).

---

Source: https://arkoren.dev/docs/guides/payments/mollie-callbacks.md
[HTML documentation](https://arkoren.dev/docs/guides/payments/mollie-callbacks)

# Mollie Callbacks

Mollie's classic callback is an unsigned form POST containing a payment ID.
Anyone can send one. Treat it as a hint to read provider state, never as evidence
that an order was paid. Use a plain application handler, not `webhook.Handler`:
there is no shipped `molliewebhook` or authenticated Mollie Source.

Configure the reachable HTTPS callback URL on the
[Mollie driver](https://arkoren.dev/docs/guides/payments/drivers.md#connecting-mollie) before creating checkout. The
following fragments assume `payments *payment.Client` for one fixed merchant
and environment, `mux *http.ServeMux`, and application-owned durable storage.
Apply route-level rate limits before mounting the handler; a body limit does
not limit request frequency.

## Binding Ownership

Save each returned payment ID with its order, application account, merchant,
and test/live environment. For multiple merchants, select credentials from this
persisted association, never from callback input. A valid-looking ID alone must
not cause an authenticated lookup under arbitrary credentials.

The handler uses two application functions, not Arkoren APIs:

- `ownsPayment(context.Context, payment.ID) (bool, error)` checks that the ID is
  saved under the fixed client's merchant/environment.
- `reconcile(context.Context, payment.Payment) error` checks the returned identity,
  order association, expected amount/currency, and permitted state transition;
  it commits processing idempotently and durably. Only a matching succeeded
  payment can satisfy its collection condition.

An unknown ID is acknowledged without revealing order data. A known ID still
requires an authoritative provider read before reconciliation.

## Parsing Hints

Use a small parser to require exactly one body form field with one valid ID.
Import `context`, `errors`, `net/http`, and `arkoren.dev/core/service/payment`. The caller
applies a 1024-byte limit before invoking this helper:

```go
func readMollieHint(r *http.Request) (payment.ID, error) {
    if err := r.ParseForm(); err != nil {
        return "", err
    }
    ids := r.PostForm["id"]
    if len(r.PostForm) != 1 || len(ids) != 1 {
        return "", errors.New("expected one payment id")
    }
    id := payment.ID(ids[0])
    if err := id.Validate(r.Context()); err != nil {
        return "", err
    }
    return id, nil
}
```

This validates the hint's shape, not its authenticity. Query parameters do not
choose the payment: only the parsed POST body is used.

## Reconciling State

With the application functions above supplied, mount the handler and reject
unusable input before any provider request:

```go
mux.HandleFunc("POST /payments/mollie", func(w http.ResponseWriter, r *http.Request) {
    r.Body = http.MaxBytesReader(w, r.Body, 1024)
    id, err := readMollieHint(r)
    if err != nil {
        w.WriteHeader(http.StatusBadRequest)
        return
    }
    known, err := ownsPayment(r.Context(), id)
    if err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    if !known {
        w.WriteHeader(http.StatusOK)
        return
    }
    reconcileMollieHint(w, r, id, payments, reconcile)
})
```

Only a saved association reaches the authenticated lookup. The next helper
uses the same fixed merchant's `payments` client and application `reconcile`:

```go
func reconcileMollieHint(
    w http.ResponseWriter,
    r *http.Request,
    id payment.ID,
    payments *payment.Client,
    reconcile func(context.Context, payment.Payment) error,
) {
    observed, err := payments.GetPayment(r.Context(), id)
    if err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    if err := reconcile(r.Context(), observed); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}
```

These helpers show application collaborators; bind the client and reconciliation
function in your handler's own dependency wiring. A 200 acknowledges durable
handling or an intentionally ignored unknown ID, not successful collection.
Provider read/reconciliation failures return 503 to permit retries. A malformed
or unusable successful GET response wraps `payment.ErrUnavailable`; it cannot
supply payment evidence or resolve an earlier uncertain mutation.

Duplicate and delayed hints must not repeat fulfillment. Commit the order
transition and fulfillment handoff atomically, or use a durable outbox. A
callback can fail after a partial application effect; acknowledgement alone is
not a transaction boundary.

## Following Refunds

Refund callbacks contain the payment ID, not a compound refund handle. Read
the refund IDs you already saved with `GetRefund` to reconcile returned funds;
do not reconstruct them from the hint. [Payment Refunds](https://arkoren.dev/docs/guides/payments/refunds.md) distinguishes
pending progress from confirmed refund success. Next-gen Mollie webhooks are
not implemented.

Test malformed/repeated IDs, unknown IDs, wrong merchant/environment, outages,
malformed successful GET bodies, unusable snapshots, duplicates, and recording
failures. None may grant fulfillment without a matching succeeded payment.
Use [Testing](https://arkoren.dev/docs/guides/payments/testing.md) for provider outcomes and `net/http/httptest` for parsing
and HTTP responses; no callback signature should be expected.

For callback configuration and snapshot behavior, see the
[Mollie payment API](https://pkg.go.dev/arkoren.dev/driver/mollie/molliepayment).

---

Source: https://arkoren.dev/docs/guides/payments/testing.md
[HTML documentation](https://arkoren.dev/docs/guides/payments/testing)

# Testing Payments

Use `memorypayment` when a test needs coherent local payment state. Use
`paymenttest.Fake` when it needs an exact provider reply, including a lost
response or missing relationship. Neither moves money, runs a buyer UI, or
schedules renewals.

The fragments belong inside your application tests with `t *testing.T` and
`ctx := t.Context()`. Import `arkoren.dev/core/service/payment` plus the simulator
or fake subpackage used below. They demonstrate the provider boundary; inject
the resulting `payments` client into your real application helpers to test
authorization, durable recording, and fulfillment policy.

## Simulating Payments

Start a fresh simulator per test. It is bundled with core and requires no account:

```go
import "arkoren.dev/core/service/payment/memorypayment"

simulator := memorypayment.New()
payments := payment.New(simulator)
```

Create an ordinary checkout with an explicit amount and key, as in
[Hosted Checkout](https://arkoren.dev/docs/guides/payments/hosted-checkout.md#starting-checkout). With that `checkout`
already created, supply buyer approval and merchant completion separately:

```go
if err := simulator.ApproveCheckout(ctx, checkout.ID); err != nil {
    t.Fatal(err)
}
checkout, err = payments.CompleteCheckout(ctx,
    payment.NewCompleteCheckoutRequest(checkout.ID, "book-184-complete"))
if err != nil {
    t.Fatal(err)
}
if checkout.PaymentStatus != payment.StatusPending {
    t.Fatal("completion must leave collection pending")
}
```

Completion creates a pending payment, not financial success. Resolve it explicitly
before reading through the facade:

```go
if err := simulator.ResolvePayment(ctx, checkout.PaymentID, payment.StatusSucceeded); err != nil {
    t.Fatal(err)
}
observed, err := payments.GetPayment(ctx, checkout.PaymentID)
if err != nil {
    t.Fatal(err)
}
if observed.Status != payment.StatusSucceeded {
    t.Fatalf("payment status = %q", observed.Status)
}
```

The read reports the state you supplied; it proves no external settlement.
Repeating completion cannot create another payment. Advancing time changes
neither pending settlement nor recurring collection. `ResolveSubscription`
supplies agreement lifecycle separately and creates no invoice/payment.

## Scripting Outcomes

For a policy test that must see a pending payment, script that exact reply.
With a checked `amount` from `payment.NewMoney("EUR", 2500)`:

```go
import "arkoren.dev/core/service/payment/paymenttest"

fake := paymenttest.New()
fake.ReplyGetPayment(payment.Payment{
    ID:     "payment-test",
    Amount: amount,
    Status: payment.StatusPending,
}, nil)
payments := payment.New(fake)
```

Call your application's payment-checking operation with that client and assert
that it does not fulfill. Scripts are independent FIFO queues per operation:
a completion reply never creates a payment-read reply or a ledger entry.
A valid unexpected call returns `paymenttest.ErrUnexpectedCall`, not invented
success.

`Attempted()` and `Accepted()` return copied typed histories. Inspect
`Attempted().ReadPayments` for reads or use mutation assertion helpers with
`testing.TB` and request predicates. Acceptance models what the test declares
accepted, not successful collection.

`FailNext(err)` overrides the next valid operation without consuming its script;
`FailWith(err)` persists until cleared. One-shot failure takes precedence.
Invalid input consumes neither failure. `Reset()` clears scripts and history.
These controls are not a financial ledger or provider replay simulator.

## Testing Uncertainty

To model accepted creation with a lost reply, queue an observed checkout ID with
`ErrOutcomeUnknown`, then separately queue a later retrieval:

```go
fake.ReplyCreateCheckout(payment.Checkout{
    ID: "checkout-test",
}, payment.ErrOutcomeUnknown)
fake.ReplyGetCheckout(payment.Checkout{
    ID:     "checkout-test",
    Status: payment.CheckoutStatusOpen,
}, nil)
```

Invoke your real creation/reconciliation helpers and assert they save the
original request/key, merchant/environment, and returned ID. `Accepted()` records
the modeled effect even with the uncertain error. For accepted cancellation
whose reply was lost, use `ReplyCanceledSubscription(payment.ErrOutcomeUnknown)`;
ordinary `ReplyCancelSubscription(err)` does not declare acceptance with an error.

Also test creation with no returned ID, followed by a known rejection on a later
same-key call:

```go
fake.ReplyCreateCheckout(payment.Checkout{}, payment.ErrOutcomeUnknown)
fake.ReplyCreateCheckout(payment.Checkout{}, payment.ErrUnauthorized)
```

After your application consumes these replies, assert that durable uncertainty
remains, the original request/key stays unchanged, and neither fresh-key dispatch
nor fulfillment is authorized. The later error does not establish the earlier
provider effect. A missing fake `Accepted()` entry reflects the script, not
proof that a real provider did nothing.

With no resource ID, portable lookup by key/Reference is unavailable. Test that
your [recovery policy](https://arkoren.dev/docs/guides/payments/recovery.md) respects provider replay scope/retention or
requires investigation instead of manufacturing an ID or another charge.

## Testing Handoffs

For [Mobile Payments](https://arkoren.dev/docs/guides/payments/mobile-payments.md), use a fictitious protocol to isolate
backend handling from a native SDK:

```go
fake.ReplyPreparePayment(payment.Preparation{
    CheckoutID: "checkout-test",
    Action:    payment.NewClientOrder("example.native", "client-order-test"),
}, nil)
```

Script `ReplyGetCheckout` with an approved checkout, expected amount, and
`NeedsCompletion: true`; then script `ReplyCompleteCheckout` with a financial
PaymentID. Queue pending and succeeded `ReplyGetPayment` replies independently.
Call the application's actual helpers and assert that approval is checked,
completion uses its saved key, and only the matching succeeded payment permits
one fulfillment commit.

For refund progress, submit a pending reply independently of the later read.
With saved `paymentID` and explicit `refundAmount`:

```go
fake.ReplyRefundPayment(payment.Refund{
    ID:        "refund-test",
    PaymentID: paymentID,
    Amount:    refundAmount,
    Status:    payment.RefundStatusPending,
}, nil)
fake.ReplyGetRefund(payment.Refund{
    ID:        "refund-test",
    PaymentID: paymentID,
    Amount:    refundAmount,
    Status:    payment.RefundStatusSucceeded,
}, nil)
```

Assert that your application reports the first outcome as pending and only the
second as confirmed refund success. Also test mismatched/missing references,
unapproved checkout, protected/no-store JSON, owner-only resume, secret-log
exclusions, recording crashes, and concurrent requests. Fake success does not
prove native approval, persistence, or HTTP security.

## Checking Conformance

If you implement a driver for your application, use `paymenttest.Run` with a
fresh `Harness` per scenario. Its complete Driver and bounded fixture hooks
supply approval, settlement, and subscription activation without waiting for
a real buyer. The factory owns cleanup through `t`.

Full journeys need `PlanReference` and `ActivateSubscription`. `Harness.Amount`
selects a positive supported one-off amount; zero selects 1200 USD, and refunds
use the observed payment currency. If completion exposes no PaymentID, provide
`ResolvePaymentID` for one bounded fixture observation. The suite neither polls
nor invents the relationship.

| Option | Declared limitation |
| --- | --- |
| `WithoutApproval()` | Omits lifecycle journeys, retaining validation and standalone unsupported subscription/refund-lookup checks. Completion/refund-creation checks are omitted with their journey; it does not declare approval unsupported. |
| `WithoutCompletion()` | Requires `ErrUnsupported` and an unchanged approved checkout that does not need completion. |
| `WithoutRefunds()` | Requires unsupported refund creation without changing the original payment. |
| `WithoutRefundLookup()` | Requires unsupported refund reads; creation still runs unless separately disabled. |
| `WithoutSubscriptions()` | Requires unsupported enrollment, lookup, and cancellation; no plan/activation hook is needed. |
| `WithoutPreparation()` | Requires unsupported valid preparation with no handoff; preparation checks remain independent of hosted approval. |

With `factory func(*testing.T) paymenttest.Harness` supplying fresh controlled
Mollie fixtures, supported EUR amounts, and approval/settlement hooks, declare
only the operations actually unsupported:

```go
paymenttest.Run(t, factory,
    paymenttest.WithoutPreparation(),
    paymenttest.WithoutCompletion(),
    paymenttest.WithoutSubscriptions(),
)
```

This still exercises one-off payments and refunds. Do not add WithoutApproval
to hide absent hooks: doing so omits the journey rather than proving it.

## Sandbox Boundaries

Wire fixtures prove serialization, mapping, and attempt counts—not live settlement.
The existing opt-in provider checks also have narrower scopes than a buyer journey:

| Provider | Opt-in coverage | Not established |
| --- | --- | --- |
| Stripe | Intent preparation/read and synthetic test-card confirmation with refund/read. | Native devices, PaymentSheet, wallets, 3DS, or signed delivery. |
| PayPal | Unapproved hosted sandbox Order creation/read. | Native preparation, SDK approval, capture, subscription collections, or signed delivery. |
| Mollie | Read-only profile authentication and optional existing test payment read. | Hosted creation/approval, refunds, or callbacks. |

These describe coverage, not tests run for this guide. Never substitute production
credentials or infer provider retention from a local fixture. With a dedicated
test merchant and your own opt-in configuration, separately exercise hosted
buyer approval, explicit reads/completion, partial refund progress, and supported
subscription approval/cancellation. Stripe/PayPal notification verification needs
real signed deliveries, not just synthetic JSON; Mollie needs the unsigned-hint
route plus authoritative API read.

Test native UI separately with the provider's test-mode SDK/device setup,
including cancellation and additional authentication. PayPal SDK approval still
needs explicit server capture. Client UI success must never bypass backend
ownership and collection checks.

For controls, histories, and harness options, see the
[paymenttest API](https://pkg.go.dev/arkoren.dev/core/service/payment/paymenttest)
and [memorypayment API](https://pkg.go.dev/arkoren.dev/core/service/payment/memorypayment).

---

Source: https://arkoren.dev/docs/guides/queues/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/queues/overview)

# Overview

A report request can finish before the report is generated. A queue holds the
work so a worker can pick it up later, independently of the requesting process.
Use it when work needs a handler and an explicit completion or failure outcome;
use [events](https://arkoren.dev/docs/guides/events/overview.md) for in-process notifications or
[broadcasting](https://arkoren.dev/docs/guides/broadcasting/overview.md) for notifications between processes.

The queue facade separates three responsibilities. A **job definition** names a
kind of work and its payload type. A **dispatcher** submits instances of that
work. A **consumer** receives messages, calls registered handlers, and asks the
driver to acknowledge, retry, reject, or fail each delivery.

Choose and configure a [driver](https://arkoren.dev/docs/guides/queues/drivers.md) before using the examples below.
Memory keeps work in one process; broker-backed drivers let producers and workers
run separately. Focused fragments leave error results visible without routine
propagation; your application must handle them.

## Job Definitions

Keep the payload and definition in a package that both producers and workers can
import. For a report, carry a stable report ID rather than all its source data:

```go
import "arkoren.dev/core/service/queue"

type BuildReport struct {
	ReportID string `json:"report_id"`
}
```

Give that work a stable name and a finite attempt limit:

```go
var BuildReportJob = queue.NewJob[BuildReport]("build-report", func(job *queue.JobBuilder) {
	job.OnQueue("reports")
	job.Retry(3)
})
```

This declares work on the logical `reports` queue; it sends nothing. `Retry(3)`
means three total attempts, including the first. The worker uses its own copy of
this policy, not policy supplied in the message. [Producing Jobs](https://arkoren.dev/docs/guides/queues/producing-jobs.md)
covers payload evolution and routing.

## Sending Work

With an existing configured `driver`, build the producer-facing facade:

```go
dispatcher := queue.NewDispatcher(driver)
```

After the report request is durably stored, submit its ID using the request's
operation context `ctx`:

```go
err := dispatcher.Dispatch(ctx, BuildReportJob, BuildReport{ReportID: "rpt-204"})
```

The compiler checks the payload type. The dispatcher JSON-encodes it and creates
a message with a fresh random ID, job name, and queue name. Success means driver
acceptance, **not report completion**. A transport error can leave acceptance
uncertain, so repeating the call can create another message.

A database commit and dispatch are not atomic. If a lost handoff is unacceptable,
record dispatch intent in the same transaction as the report request and use a
[durable outbox handoff](https://arkoren.dev/docs/guides/queues/producing-jobs.md#durable-handoff).

## Handling Work

In the worker, wrap its configured driver in a consumer. With `context` imported
and your application's `reports.Build(ctx, reportID) error` operation available,
register the handler before running the consumer:

```go
consumer := queue.NewConsumer(driver)
err := consumer.Handle(BuildReportJob, func(ctx context.Context, payload BuildReport) error {
	return reports.Build(ctx, payload.ReportID)
})
```

Each delivery attempt decodes a fresh payload. A `nil` result requests
acknowledgement; an ordinary error requests retry while attempts remain, then
terminal failure handling. Pass `ctx` to blocking work: timeouts and shutdown
cancel the context but cannot kill your handler.

[Consuming Jobs](https://arkoren.dev/docs/guides/queues/consuming-jobs.md) shows how to run and stop the worker. A web
process needs only the dispatcher; a worker needs the consumer and its handlers.
Both must agree on job names, payloads, logical queues, and driver routing.

## Delivery Boundaries

An attempt is one delivery to a worker, not a promise of exclusive or exactly-once
execution. If report generation succeeds but acknowledgement fails, the message
can return. Lost leases can also let two handlers overlap. Make repeats harmless
using the report ID and enforce uniqueness where the effect is stored.

No portable unconditional at-least-once guarantee covers every driver and retry
path. Memory loses pending work on process exit or driver stop; broker retention,
delivery caps, and retry forwarding can also remove work. Competing workers and
retries can reorder jobs. Enforce required ordering in application state rather
than relying on dispatch order.

[Delivery and Retries](https://arkoren.dev/docs/guides/queues/delivery-and-retries.md) explains acknowledgement gaps,
retry policy, poison messages, and remote-effect recovery. The
[compatibility matrix](https://arkoren.dev/docs/guides/queues/drivers.md#compatibility-matrix) compares backend limits.

## Testing Jobs

To test a producer without executing its handler, use the bundled
`arkoren.dev/core/service/queue/queuetest` driver. In a test with `t *testing.T`,
give your application's `RequestReport(ctx, dispatcher, reportID) error`
operation a fake-backed dispatcher:

```go
fake := queuetest.New()
dispatcher := queue.NewDispatcher(fake)
if err := RequestReport(t.Context(), dispatcher, "rpt-204"); err != nil {
	t.Fatal(err)
}
```

Assert the requested work, not the transport envelope's generated ID:

```go
fake.AssertDispatched(t, BuildReportJob, func(payload BuildReport) bool {
	return payload.ReportID == "rpt-204"
})
```

This proves that the application submitted the expected payload, not that a
broker retained it or a handler completed it. To exercise consumer policy, use
the fake's [Drain support](https://arkoren.dev/docs/guides/testing/fakes.md#separate-execution).
It runs due jobs synchronously; delayed retries stay pending unless you pass
`queuetest.WithScheduled()`, and a delivery limit detects endless retry loops.
The fake records failed and rejected messages too. Use `memoryqueue` when a test
needs a live consumer with delayed retries; it works with `testing/synctest`.

See the [`queue` API](https://pkg.go.dev/arkoren.dev/core/service/queue) and
[`queuetest` API](https://pkg.go.dev/arkoren.dev/core/service/queue/queuetest).

---

Source: https://arkoren.dev/docs/guides/queues/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/queues/drivers)

# Drivers

Choose memory for single-process development and tests. For work shared across
processes, use a broker you operate—RabbitMQ or NATS JetStream—or a managed queue,
Amazon SQS or Cloudflare Queues. Decide what must survive worker restarts, how
long handlers run, and whether terminal failures need an archive before choosing.

The examples map the logical `reports` queue from the [overview](https://arkoren.dev/docs/guides/queues/overview.md) to
provider resources. After construction succeeds, use the driver with
`queue.NewDispatcher(driver)` and `queue.NewConsumer(driver)`. Neither facade
starts or stops the driver. [Consuming Jobs](https://arkoren.dev/docs/guides/queues/consuming-jobs.md#worker-lifetime)
owns runner wiring; the provider sections below own connection setup.

Run installation commands from your application's module directory. Driver
construction callbacks run once in order; nil callbacks panic. Required routing
maps are copied, and retained drafts cannot reconfigure published drivers.
Native clients, loggers, option functions, and their captured state remain shared;
configure them before concurrent use. Native calls bypass facade instrumentation.

Local worker commands below assume your existing application runs its registered
consumer from `./cmd/worker`; substitute your actual worker entry point. They do
not refer to an Arkoren CLI subcommand or create a worker for you.

## Compatibility Matrix

This compares Arkoren's exposed behavior, not every vendor feature. Restart
survival means pending work can outlive the application process; it is conditional
on broker storage, retention, delivery limits, and available infrastructure.
A failure archive is an application-level transfer configured on the driver,
not a native dead-letter policy.

| Driver | Worker Restart | Delayed Retry | Lease Renewal | Failure Archive |
| --- | --- | --- | --- | --- |
| [Memory](#memory) | No | Yes — Process-local pending work. | No | No |
| [RabbitMQ](#rabbitmq) | Yes — Durable queues and persistent publications. | Partial — TTL forwarding and queue expiry can lose work. | No — Uses unacknowledged deliveries. | Partial — Requires a distinct FailureQueue and a confirmed routed transfer. |
| [NATS](#nats) | Partial — Stream storage and retention are operational policy. | Yes | Yes — Renewal stops after the first unexpected heartbeat failure. | Partial — Requires a distinct FailureSubject with separately provisioned storage. |
| [Amazon SQS](#amazon-sqs) | Yes — Subject to queue retention. | Partial — At most 12 hours and limited by remaining visibility allowance. | Yes — Limited to 12 hours from the receive request. | Partial — Requires FailureQueueURL. |
| [Cloudflare](#cloudflare) | Yes — Subject to provider retention and delivery limits. | Yes — At most 24 hours. | No | No |

All shipped drivers report attempts, allowing the facade to enforce `Retry(n)`
as **n total attempts**. Failed settlement and lost leases can duplicate work.
There is no portable exactly-once execution or unconditional at-least-once
guarantee; see [Delivery and Retries](https://arkoren.dev/docs/guides/queues/delivery-and-retries.md).

## Memory

`memoryqueue` is bundled in `arkoren.dev/core`. Share one driver instance between
the dispatcher and consumer; separate instances do not share work.

### Connecting

Construction starts no goroutines and needs no server:

```go
import "arkoren.dev/core/service/queue/memoryqueue"

driver := memoryqueue.New()
```

`Start` does nothing on an open driver. There is no `Ping`. Register `Stop` for
cleanup, but cancel and join consumers first: `Stop` discards ready jobs and
delayed retries rather than draining them, even with an expired stop context.
Process exit also loses pending work.

### Retry Timing

Without a job backoff, retries are immediate. If a process-local retry needs a
default pause, use an alternative construction with `time` imported:

```go
driver := memoryqueue.New(func(broker *memoryqueue.Builder) {
	broker.RetryDelay(time.Second)
})
```

Consumers promote delayed work while polling; no background timer worker is
required. Set a finite job attempt limit to avoid an endlessly failing job.
`Logger` selects a shared logger, with `slog.Default()` as the default.

### Processing Work

Use the shared [dispatch example](https://arkoren.dev/docs/guides/queues/producing-jobs.md#dispatching-work) and
[handler example](https://arkoren.dev/docs/guides/queues/consuming-jobs.md#registering-handlers). Consumers on the same
queue compete. Canceling a consumer preserves work not yet admitted and lets an
active handler finish settlement; stopping the driver instead closes admissions
and discards pending work. `Stop` waits for active consumers only until its
context ends and cannot kill a handler. An admitted delivery can still be
acknowledged after stop, but retrying into the closed driver fails with
`memoryqueue.ErrDriverClosed`. New valid dispatch and consume calls also fail
with that error; an already-canceled context takes precedence.

Terminal failures are logged and dropped; rejection drops the job. For dispatch
assertions without a live consumer, use the [queue fake](https://arkoren.dev/docs/guides/queues/overview.md#testing-jobs).

See the [`memoryqueue` API](https://pkg.go.dev/arkoren.dev/core/service/queue/memoryqueue).

## RabbitMQ

Choose RabbitMQ for durable AMQP work queues and confirmed persistent publishes.
You need an AMQP endpoint, credentials, and permission to declare the configured
exchange, queues, bindings, and retry topology. Install from your application:

```sh
go get arkoren.dev/driver/rabbitmq
```

### Connecting

Load `amqpURL` from trusted configuration such as `AMQP_URL`; it contains
credentials, so use `amqps://` outside local development and keep it out of logs. Map logical names
to distinct physical queues and routing keys:

```go
import "arkoren.dev/driver/rabbitmq/rabbitmqqueue"

routes := map[string]rabbitmqqueue.Queue{
	"reports": {Name: "app.reports", RoutingKey: "reports"},
}
driver, err := rabbitmqqueue.New(amqpURL, routes)
```

`New` validates configuration without connecting. Handle `err` before using the
driver. `New` uses a connection without SDK automatic recovery. `Start` connects
and declares a durable direct exchange (by default, `arkoren:jobs`) and work
queues. In your existing composition callback, `composer` is a
`*application.Composer` from `arkoren.dev/core/application`. Register cleanup
before startup:

```go
composer.Cleanup(driver.Stop)
composer.Startup(driver.Start)
composer.Check("queue", driver.Ping)
```

Serialize `Start` and `Stop`; cancel and join
consumers before stopping the driver. An owned `Stop` releases the connection,
bounded by five seconds or its context ending sooner, but does not join handlers.
An expired context still releases the socket. A second stop returns
`ErrDriverClosed`.

### Failure Archives

If terminal failures need inspection, configure a separate archive queue in an
alternative construction. Restrict its access and retention first: failure
envelopes contain job data, error text, and the original raw body even for valid
jobs.

```go
driver, err := rabbitmqqueue.New(amqpURL, routes, func(broker *rabbitmqqueue.Builder) {
	broker.FailureQueue("app.report-failures")
})
```

`Start` declares that queue. A failure transfer must receive a positive confirm
without an unroutable return before the source is acknowledged. Without an
archive, terminal failure rejects without requeue; rejection always does so.
Work queues have no driver-declared dead-letter exchange, so rejected work is
dropped unless you add a broker policy.

### Delivery Policy

Use [Dispatching Work](https://arkoren.dev/docs/guides/queues/producing-jobs.md#dispatching-work) after startup. Dispatch
waits for a positive publisher confirm without an unroutable `basic.return`.
That establishes acceptance by at least one queue, not execution or correctness
of the intended binding. Keep exchange administration trusted. An unroutable or
negatively confirmed publish returns `ErrPublishNotConfirmed`. Lost confirmations
can still mean accepted work, and transfer plus source acknowledgement is not
atomic.

`Prefetch` defaults to one and accepts 1–65535; zero restores one. Increasing it
reserves more work but does not run handlers concurrently within one consume
call. `Exchange` selects a different direct exchange. `RetryDelay` supplies a
one-second default when a retry requests zero. `MaxRetryDelay` bounds requested
delays, with zero permitting 4294967295 milliseconds (about 49 days); the default
retry delay must fit that bound. Invalid settings wrap `ErrInvalidConfig`.

If facade settlement fails, the driver logs and returns the error and closes the
consumer channel, releasing unsettled deliveries for possible redelivery.
`Consumer.Run` then cancels its sibling queue consumers and returns after they
stop. This differs from an ordinary handler error whose retry settled successfully.

### Retry Forwarding

Retries use durable per-delay queues with per-message TTL. Positive delays round
up into buckets: seconds through a minute, 15 seconds through an hour, five
minutes through a day, then hours, capped at the TTL maximum. A delay above
`MaxRetryDelay` returns `ErrUnsupportedRetryDelay` without settling the source.
TTL expiry is not exact scheduling; older queued messages can delay forwarding.

> [!WARNING]
> A retry confirmation covers the retry queue, not its later dead-letter
> forwarding to the work exchange. Default forwarding can lose messages if the
> destination is unavailable or unroutable. Provision quorum retry queues and
> protected at-least-once dead-lettering with `reject-publish` overflow, the
> required `stream_queue` feature flag, permissions, bindings, and storage before
> relying on this path. The driver neither selects nor verifies these policies.

Set queue-type defaults before retry queues are created; redeclaration cannot
change an existing type. Even protected forwarding can duplicate work. Retry
queues expire after an unused interval of twice their bucket delay plus one
hour, including while forwarding is blocked. Expiry, deletion, and retention
can therefore still discard work during prolonged outages. See
[RabbitMQ's protected dead-lettering requirements](https://www.rabbitmq.com/docs/4.3/quorum-queues#dead-lettering).

### Borrowed Connections

If your application owns a `*amqp091.Connection` from
`github.com/rabbitmq/amqp091-go`, it must have automatic recovery disabled:

```go
driver, err := rabbitmqqueue.NewFrom(connection, routes)
```

`NewFrom` rejects recovery-enabled connections with `ErrInvalidConfig` without
closing or reconfiguring them. Never enable recovery or reconnect that connection
in place: reused delivery tags could let an old handler settle a different job.
`Start` still declares topology. `Stop` closes the publisher channel, not your
connection; close it yourself after consumers stop. `Connection()` exposes the
native pointer without transferring ownership.

For owned connections, cancellation during startup, readiness, consume setup, or
a blocked SDK write can close the socket, interrupting every channel. Normal
consume cancellation attempts channel-only cleanup after the handler returns,
then aborts an unresponsive owned connection after five seconds. Reconstruct the
driver after transport failure. Borrowed connections are never closed to enforce
cancellation: SDK channel/topology/close RPCs need a responsive peer or
caller-managed socket deadlines/close and can outlast a driver context.

### Running Locally

With Docker installed, start a disposable server:

```sh
docker run --detach --name report-rabbit --publish 127.0.0.1:5672:5672 --env RABBITMQ_DEFAULT_USER=local --env RABBITMQ_DEFAULT_PASS=local-only rabbitmq:4-management
docker exec report-rabbit rabbitmq-diagnostics -q ping
```

Wait for the diagnostic command to succeed. With your worker loading `AMQP_URL`
into `amqpURL` and registering the driver hooks and consumer runner, start it:

```sh
AMQP_URL=amqp://local:local-only@localhost:5672/ go run ./cmd/worker
```

Dispatch a report job from your application and verify its stored result; acceptance
alone does not prove handler execution. Stop and join the worker before removing
only this disposable server and its anonymous volumes:

```sh
docker rm --force --volumes report-rabbit
```

See the [`rabbitmqqueue` API](https://pkg.go.dev/arkoren.dev/driver/rabbitmq/rabbitmqqueue).

## NATS

Choose NATS if you operate JetStream streams and durable pull consumers. Core
NATS alone is insufficient. Install the driver in your application:

```sh
go get arkoren.dev/driver/nats
```

### Connecting

Provision a stream and durable consumer with your operational tooling. Map each
logical queue to a distinct literal subject and durable name:

```go
import "arkoren.dev/driver/nats/natsqueue"

routes := map[string]natsqueue.Queue{
	"reports": {Subject: "jobs.reports", Durable: "reports-worker"},
}
driver, err := natsqueue.New("REPORTS", routes)
```

Construction validates settings without connecting. By default `Start` connects
to `nats://127.0.0.1:4222` and checks resources without creating them. Register
`Stop`, `Start`, and `Ping` on the composer as in RabbitMQ. An owned connection
is drained by `Stop`; context cancellation forces close, including with an
already-expired context. Drain also observes native `DrainTimeout`, preserving
`nats.ErrDrainTimeout`. Repeated stop is harmless; a stopped driver cannot restart.
Cancel and join handlers first.

For local development where resource creation is permitted, use this alternative:

```go
driver, err := natsqueue.New("REPORTS", routes, func(broker *natsqueue.Builder) {
	broker.Provision(true)
})
```

It creates missing work resources only, never repairs incompatible resources and
never creates failure-subject storage. Missing or incompatible resources match
`ErrResourcesNotProvisioned`; authorization, connection, and cancellation errors
retain their own identities. Choose stream storage, replication, and retention
operationally; successful startup does not establish all durability policies.
Failed startup closes a newly owned connection, not a borrowed one.

### Resource Policy

The stream must list each work subject explicitly; a covering wildcard alone is
not sufficient. Consumers must be durable pull consumers with bodies, explicit
acknowledgements, and the configured literal filter. Their `AckWait` and
`MaxAckPending` must match the driver. Positive native batch limits must accommodate
the configured batch, and positive pull-expiry limits must allow ten seconds.
An explicit driver `MaxDeliver` must match too. Each consume binding rechecks the
current durable without creating or overwriting it.

`Batch` defaults to one; `MaxAckPending` defaults to the completed batch size and
cannot be smaller. Handlers run sequentially within a batch. `AckWait` defaults
to 30 seconds; zero restores that default and values below two nanoseconds are
invalid. This floor is not a useful production deadline. Pull expiry is separately
ten seconds or the caller's earlier deadline. `MaxDeliver(0)` leaves the native
cap alone; exhausting a native cap does not transfer the message to a failure
subject.

### Authentication Policy

For owned connections, configure server URLs and native TLS/authentication options
before startup. With `github.com/nats-io/nats.go` imported, an authorized user
NKey seed file named by `seedPath` can supply authentication:

```go
authentication, err := nats.NkeyOptionFromSeed(seedPath)
if err != nil {
	return err
}
```

Pass that option to a new driver, using your TLS-protected `serverURL`:

```go
driver, err := natsqueue.New("REPORTS", routes, func(broker *natsqueue.Builder) {
	broker.URLs(serverURL)
	broker.Options(authentication)
})
```

The helper takes a file path, not seed text; passing seed text as a path can expose
it in filesystem errors. For in-memory keys, use native `nats.Nkey` with a public
key and signing callback. Keep credential-bearing URLs and native options out of
logs. Options and their captured state remain application-owned.

### Processing Work

Use [Dispatching Work](https://arkoren.dev/docs/guides/queues/producing-jobs.md#dispatching-work); dispatch waits for
JetStream acceptance. Workers sharing the durable compete. Raw dispatch uses a
nonempty `Message.ID` as `Nats-Msg-Id`, enabling deduplication within the stream's
duplicate window. Facade dispatch generates a fresh ID every time, so resubmitting
the same payload is not automatically deduplicated or exactly-once execution.

All fetched messages receive progress heartbeats every half `AckWait`, including
those waiting behind another handler. Renewal continues through active-handler
drain, but the first unexpected renewal error is logged and stops that delivery's
heartbeat worker. Redelivery can then overlap the original handler. Renewal and
settlement are serialized so successful delayed retry is not overwritten by a
later heartbeat.

Retry uses delayed negative acknowledgement; zero is immediate. Rejection uses
JetStream termination, not a dead-letter transfer. To archive terminal failures,
set `FailureSubject("jobs.report-failures")` in the construction callback and
separately provision JetStream storage for that distinct literal subject. Failure
publication must succeed before source acknowledgement; the two are not atomic.
Without a failure subject, terminal failure logs and acknowledges the message.
Protect archive data and error text with access and retention controls.

Settlement errors are logged without automatically acknowledging or ending
consumption; an unsettled job can return after `AckWait`. Empty pulls and pulls
interrupted by reconnect resume polling; other fetch errors end consumption.
Cancellation stops fetching and leaves unhandled batch messages unacknowledged.

### Borrowed Connections

To reuse an existing `*nats.Conn`, supply it before the stream and routing map:

```go
driver, err := natsqueue.NewFrom(connection, "REPORTS", routes)
```

`Start` still checks or provisions resources. `URLs` and `Options` are ignored;
configure the native connection first. Neither failed startup nor `Stop` closes
or drains a borrowed connection. The caller drains/closes it after consumers
stop. `Connection()` exposes it without transferring ownership; native calls
bypass facade telemetry and routing. Delivered message payload and metadata
snapshots are copied, so edits cannot alter subsequent settlement.

### Running Locally

With Docker installed, start JetStream and enable its monitoring endpoint:

```sh
docker run --detach --name report-nats --publish 127.0.0.1:4222:4222 --publish 127.0.0.1:8222:8222 nats:2 -js -m 8222
curl --fail http://localhost:8222/healthz
```

Wait for health to succeed. Configure your worker with the local construction and
`Provision(true)`, then run its consumer:

```sh
go run ./cmd/worker
```

Successful `Start` confirms compatible work resources. Dispatch a report job from
your application and verify its stored result. This setup is disposable,
not a persistent production deployment. Stop and join the worker before removing
the server and its state:

```sh
docker rm --force --volumes report-nats
```

See the [`natsqueue` API](https://pkg.go.dev/arkoren.dev/driver/nats/natsqueue).

## Amazon SQS

Choose SQS for managed AWS queues and long-poll consumers. Provision work queues
before construction. Producers need `sqs:SendMessage`; consumers need
`sqs:ReceiveMessage`, `sqs:DeleteMessage`, and `sqs:ChangeMessageVisibility`.
A failure archive also needs `sqs:SendMessage` on its destination.

```sh
go get arkoren.dev/driver/aws
```

### Connecting

With your provisioned queue URL `reportURL` and configuration-loading context
`ctx`, construct the driver:

```go
import "arkoren.dev/driver/aws/awsqueue"

driver, err := awsqueue.New(ctx, map[string]string{"reports": reportURL})
```

`New` validates routes and optional settings before loading region and credentials
from the AWS default chain. There are no `Start`, `Stop`, or `Ping` hooks; register
only the consumer runner. Use [Dispatching Work](https://arkoren.dev/docs/guides/queues/producing-jobs.md#dispatching-work)
and [Registering Handlers](https://arkoren.dev/docs/guides/queues/consuming-jobs.md#registering-handlers) unchanged.

### Polling Policy

Long polling defaults to 20 seconds, with `WaitTimeSeconds` accepting 1–20.
`MaxMessages` defaults to one and accepts 1–10. `VisibilityTimeout` defaults to
30 seconds and accepts one second through 12 hours. Zero restores each default;
invalid settings wrap `ErrInvalidConfig`.

Renewal runs every half configured visibility, including for messages waiting
in a batch and during active-handler shutdown. Wire visibility rounds fractional
seconds upward; heartbeat timing uses the unrounded setting. Each renewal SDK
operation gets the lesser of ten seconds and half the configured visibility,
including SDK retries and serialization wait. Renewal errors are logged and
later ticks can retry, but a lost lease can duplicate work. SQS cannot extend
visibility beyond 12 hours from the receive request; split longer work.

Settlement waits for earlier renewal within its own context budget; successful
settlement prevents further renewal. Failed settlement permits another attempt.
Handler completion cancels heartbeat I/O before joining its worker. Native HTTP,
credential, and middleware dependencies must honor contexts or cleanup can wait
past its intended bound. Settlement errors are logged, not automatically
acknowledged; polling continues until cancellation or a receive error.

### Failure Archives

Provision a separate failure queue if terminal jobs need retained reports. With
its URL `failureURL`, use an alternative construction:

```go
driver, err := awsqueue.New(ctx, map[string]string{"reports": reportURL}, func(broker *awsqueue.Builder) {
	broker.FailureQueueURL(failureURL)
})
```

The driver sends a failure envelope before deleting the source; failed publication
withholds deletion. The operations are not atomic and can duplicate work or reports.
Without this setting, terminal failure logs and deletes the source. Archive
contents include job data, error text, and undecodable raw bodies; restrict access,
set retention, and avoid secrets in errors.

Retry changes visibility, rounding positive fractional seconds up. Values above
12 hours return `ErrUnsupportedRetryDelay`; SQS can reject even an otherwise valid
delay if it exceeds the remaining visibility allowance. The driver does not
shorten it. Rejection leaves the message untouched for the queue's redrive policy.
Without redrive, rejection can return repeatedly; prefer `queue.Permanent` unless
you intentionally rely on configured broker redrive.

### FIFO Messages

With a configured native `client` and separately provisioned FIFO work and failure
URLs, wrap both destinations just as you would standard queues:

```go
driver, err := awsqueue.NewFrom(client, map[string]string{"reports": reportFIFOURL},
	func(broker *awsqueue.Builder) {
		broker.FailureQueueURL(failureFIFOURL)
	})
```

For destination URLs ending in `.fifo`, including failure queues, the driver adds
native group and deduplication fields automatically. The group is the logical
queue name (`failed` if missing); deduplication uses the message ID, or a fresh ID
if missing. Provision FIFO queues yourself and pass their URLs through the same
constructors; the JSON job and failure-envelope formats do not change. The
failure destination must be separate from work queues and have the same access
and retention protections described above.

Each facade dispatch still creates a fresh ID. FIFO deduplication therefore does
not make repeated payloads or handler execution exactly once. The consumer also
accepts SNS notification wrappers whose `Message` contains a queue envelope;
this does not turn arbitrary SNS payloads into typed jobs.

### Native Clients

For a custom region, endpoint, credentials, retries, or HTTP transport, configure
a `*sqs.Client` from `github.com/aws/aws-sdk-go-v2/service/sqs` first:

```go
driver, err := awsqueue.NewFrom(client, map[string]string{"reports": reportURL})
```

All queue settings still apply; nil clients are invalid. `NewFrom` makes no SQS
requests. `Client()` exposes the same borrowed client, which the driver never
closes. Native operations bypass facade encoding and telemetry.

### Running Locally

With Docker and the AWS CLI installed, start a disposable emulator:

```sh
docker run --detach --name report-sqs --publish 127.0.0.1:14568:4566 ministackorg/ministack:1.5.8
curl --fail http://localhost:14568/_ministack/health
```

Wait for health to succeed. In the same shell, use emulator-only credentials and
create a queue:

```sh
export AWS_ACCESS_KEY_ID=000000000000
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1
reportURL=$(aws --endpoint-url http://localhost:14568 sqs create-queue --queue-name reports --query QueueUrl --output text)
aws --endpoint-url http://localhost:14568 sqs get-queue-attributes --queue-url "$reportURL" --attribute-names QueueArn
```

The final command returns the queue ARN. Supply `reportURL` to your application.
With the AWS SDK packages below imported, configure a client for this emulator:

```go
import (
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/sqs"
)

cfg := aws.Config{
	Region:       "us-east-1",
	BaseEndpoint: aws.String("http://localhost:14568"),
	Credentials:  credentials.NewStaticCredentialsProvider("000000000000", "test", ""),
}
```

Wrap that client with the queue you created:

```go
driver, err := awsqueue.NewFrom(sqs.NewFromConfig(cfg), map[string]string{"reports": reportURL})
```

These credentials and plaintext endpoint are local-only. Make your existing worker
load `REPORT_QUEUE_URL` into `reportURL` and use the emulator client above:

```sh
REPORT_QUEUE_URL="$reportURL" go run ./cmd/worker
```

Dispatch a report job from your application and verify its stored result. Stop
and join the worker before removing the disposable emulator and its state:

```sh
docker rm --force --volumes report-sqs
```

See the [`awsqueue` API](https://pkg.go.dev/arkoren.dev/driver/aws/awsqueue).

## Cloudflare

Choose Cloudflare for Go services outside Workers that send and pull Cloudflare
Queues messages through REST. Create the queues and obtain an account ID, queue
IDs, and an API token with Queues read/write permission. Each consumed queue needs
an HTTP pull consumer, configured in the dashboard or with
`wrangler queues consumer http add <queue>`. Configure native retry limits and
a dead-letter queue there if you need broker retention after rejection.

```sh
go get arkoren.dev/driver/cloudflare
```

### Connecting

Load `token` and `accountID` from trusted configuration; `reportQueueID` is the
provider's queue ID, not its display name:

```go
import "arkoren.dev/driver/cloudflare/cloudflarequeue"

driver, err := cloudflarequeue.New(token, accountID, map[string]string{"reports": reportQueueID})
```

Construction validates the required inputs without a request and copies routing
before callbacks. There are no lifecycle hooks; run the shared consumer and use
[Dispatching Work](https://arkoren.dev/docs/guides/queues/producing-jobs.md#dispatching-work). A producer does not need
an HTTP pull consumer, but a worker does. A message is limited to 128 KB including
the envelope, which base64-encodes payload bytes.

### Lease Policy

Cloudflare cannot renew leases. A handler and the earlier handlers in its fetched
batch must finish before visibility expires to avoid concurrent redelivery.
Keep the default batch size of one for slow jobs. To allow more time, use an
alternative construction with `time` imported:

```go
driver, err := cloudflarequeue.New(token, accountID, map[string]string{"reports": reportQueueID},
	func(broker *cloudflarequeue.Builder) {
		broker.VisibilityTimeout(2 * time.Minute)
	})
```

Also give the job a shorter cooperative execution timeout, leaving time for
settlement. Visibility defaults to 30 seconds and accepts one millisecond through
12 hours; zero restores the default. `BatchSize` accepts 1–100, defaulting to one.
Empty-poll waits start at `PollInterval` (one second), double up to
`MaxPollInterval` (ten seconds), and reset after work arrives. Zero restores these
defaults; negative intervals fail construction, and a positive cap below the
initial interval is raised to it.

### Settlement Results

Cloudflare supplies delivery IDs and attempts. Acknowledgement requires exactly
one native acknowledgement count; retry requires exactly one retry count. HTTP
success, warnings alone, or a count for the other operation do not confirm
settlement. Matching-count warnings are logged without invalidating success.
Errors and unconfirmed counts leave settlement open, but do not prove the remote
operation had no effect.

Retry delays round positive fractions up to seconds and reject values above
24 hours rather than shortening them. Rejection settles locally without a remote
request: the lease expires normally, then native retry/dead-letter policy applies.
Without a dead-letter queue, native retry exhaustion discards the message.
Terminal failure logs and requests deletion; it creates no archive. Settlement
errors are logged and consumption continues; a pull failure after SDK retries
ends consumption. Cancellation checks every batch admission and leaves unused
leases to expire; an active handler must return cooperatively.

### Native Clients

For custom HTTP transport, retries, or middleware, configure a
`*cloudflare.Client` from `github.com/cloudflare/cloudflare-go/v7` first:

```go
driver, err := cloudflarequeue.NewFrom(client, accountID, map[string]string{"reports": reportQueueID})
```

The caller owns the client and transport cleanup. `Client()` exposes that same
pointer; configure shared clients and loggers before concurrent use. All builder
settings apply except `Endpoint`, which affects only the client created by `New`.
Native calls bypass facade telemetry and driver settlement checks. Clients,
native error causes, handler errors, failure causes, and logged warnings are not
recursively sanitized; keep credentials and sensitive payloads out of diagnostics
and apply application logger redaction where needed.

There is no local emulator for this driver's remote REST pull API. Use the
[queue fake](https://arkoren.dev/docs/guides/queues/overview.md#testing-jobs) locally and a dedicated Cloudflare
development queue for remote checks, never production messages.

See the [`cloudflarequeue` API](https://pkg.go.dev/arkoren.dev/driver/cloudflare/cloudflarequeue).

---

Source: https://arkoren.dev/docs/guides/queues/producing-jobs.md
[HTML documentation](https://arkoren.dev/docs/guides/queues/producing-jobs)

# Producing Jobs

When someone requests a report, store the request and let a worker generate the
file later. The producer defines the kind of work once, then dispatches a payload
for each report. The definition stays in application code; only the job name,
queue, JSON payload, and transport information travel in the message.

These examples assume a [configured dispatcher](https://arkoren.dev/docs/guides/queues/drivers.md) named `dispatcher`
and an operation context `ctx`. Keep the definition in a package that the worker
can also import. Retry, backoff, and timeout policy come from the worker's own
registered definition, not from the producer's message.

## Defining Jobs

Carry the report's stable identifier so the handler can load its current state.
Import `arkoren.dev/core/service/queue` as `queue` and `time` for the policy below:

```go
type BuildReport struct {
	ReportID string `json:"report_id"`
}
```

Declare an immutable job with a stable name and a finite execution policy:

```go
var BuildReportJob = queue.NewJob[BuildReport]("build-report", func(job *queue.JobBuilder) {
	job.OnQueue("reports")
	job.Retry(3)
	job.Backoff(time.Second, 2, time.Minute)
	job.Timeout(45 * time.Second)
})
```

This job allows three total attempts, not three additional retries. Its two
retries request one-second and two-second delays. Each attempt gets its own
45-second context deadline. A timeout is cooperative cancellation, not a hard
kill: the consumer still waits for the handler to return. A handler returning
`nil` is acknowledged even after its deadline.

Without explicit policy, `Retry(0)` leaves the limit to the driver, backoff uses
driver timing, and `Timeout(0)` imposes no execution deadline. Memory has no native
attempt cap and immediate retries by default, so an endlessly failing job can
spin. Prefer finite attempts and deliberate backoff. [Delivery and Retries](https://arkoren.dev/docs/guides/queues/delivery-and-retries.md#retry-policy)
owns policy variations, custom backoff, and backend delay limits.

### Stable Names

The name selects the worker's handler. Use 1–64 lowercase ASCII letters, digits,
or hyphens, with no hyphen at either end. Keep it stable while messages bearing
that name can still exist.

Always construct jobs with `NewJob`; a zero `queue.Job[T]` is invalid. Invalid
names or settings retain the first validation error in `job.Err()` rather than
panicking. `Dispatch` and `Handle` reject an invalid job with `queue.ErrInvalidName`
wrapping that cause, before encoding or registering it. Negative attempts,
timeouts, and backoff durations are invalid; an empty queue is invalid too.

Construction callbacks run once, synchronously in order, even for an invalid
name. Supplied nil callbacks panic. Published jobs have no setters or `Derive`;
retaining a draft cannot change them. To share policy, pass a
`queue.JobBuilderFunc` to fresh `NewJob` calls and use later callbacks for local
overrides. Custom backoff closures retain their application-owned captured state.
`Description` adds human-readable text without changing delivery.

## Queue Routing

Use separate queues when reports should not compete with urgent work for the
same consumer. `OnQueue("reports")` chooses a logical destination; without it,
a job uses the literal `default` queue.

The [driver's trusted routing](https://arkoren.dev/docs/guides/queues/drivers.md) maps that name to a RabbitMQ queue and
routing key, JetStream subject and durable consumer, SQS URL, or Cloudflare queue
ID. Broker drivers reject a logical name with no configured mapping. They never
take the physical destination from an untrusted message body.

The worker must register the job and consume its logical queue. A
[queue restriction](https://arkoren.dev/docs/guides/queues/consuming-jobs.md#selecting-queues) can dedicate a worker to
reports without changing producer routing or creating broker resources.

## Dispatching Work

Dispatch only after the data the handler needs is durable. Publishing before a
database transaction commits lets a fast worker read missing or rolled-back data.
For the already-stored report `rpt-204`, submit its ID:

```go
err := dispatcher.Dispatch(ctx, BuildReportJob, BuildReport{ReportID: "rpt-204"})
```

The compiler checks the payload type. The dispatcher JSON-encodes the payload,
adds a fresh random message ID, and calls the driver. Success means acceptance
by the driver, not execution or a completed report. There is no dispatch delay or
schedule option; delay policy applies to retries, not the initial submission.

An invalid job or JSON encoding failure occurs before the driver call and queues
nothing. Driver errors retain their causes. Once a send begins, a timeout or
connection error can mean that the broker accepted the message but its reply was
lost. Do not infer non-acceptance or blindly promise that retrying is harmless.

Every facade dispatch creates a new ID. Repeating a payload is not automatically
deduplicated, including on JetStream or SQS FIFO. Use the report ID as the business
identity and make the [handler's effects repeat-safe](https://arkoren.dev/docs/guides/queues/delivery-and-retries.md#duplicate-effects).

## Durable Handoff

Committing the report request and then dispatching avoids premature execution,
but a crash between those operations can still lose the work. If that gap is
unacceptable, use an application-owned **outbox**: a durable record of work still
to be dispatched.

Write the report request and its outbox intent in the same database transaction.
A separate relay reads committed intents, dispatches them, and marks them sent
only after driver acceptance. Keep unsuccessful or uncertain sends recoverable.
If multiple relays run, claim records safely and recover abandoned claims; do
not let two relays assume exclusive ownership from an unprotected read.

The relay can crash after acceptance but before recording success. It must then
be able to resend, so the outbox still requires idempotent handlers and a stable
business ID carried through every resend. Queue message IDs generated by the
facade do not replace that identity. Define retry, alerting, reconciliation, and
retention for stuck intents instead of deleting them after an arbitrary send
error. Arkoren's queue facade does not supply an atomic database-and-broker
transaction or an outbox relay API.

## Payload Changes

Keep payloads small: send identifiers rather than full records, credentials, or
large file contents. Broker readers can inspect queued data, and provider message
limits include the envelope. Cloudflare's 128 KB limit includes base64-encoded
payload bytes, so a 128 KB source payload does not fit unchanged.

Messages outlive deployments. Add fields with useful zero values; do not rename
or retype fields while older messages can remain queued. Deploy compatible
workers before producers start emitting new names or payload shapes. A payload
that cannot decode fails terminally without ordinary retries; unknown names get
a bounded [deployment grace period](https://arkoren.dev/docs/guides/queues/delivery-and-retries.md#poison-messages).
Changing consumer policy affects later deliveries of existing messages because
the message does not contain that policy.

For a machine-readable description of jobs without changing execution, use
[Queues and Operations](https://arkoren.dev/docs/guides/asyncapi/queues-and-operations.md). For every queue
function and option, see the [`queue` API](https://pkg.go.dev/arkoren.dev/core/service/queue).

---

Source: https://arkoren.dev/docs/guides/queues/consuming-jobs.md
[HTML documentation](https://arkoren.dev/docs/guides/queues/consuming-jobs)

# Consuming Jobs

A producer can request a report without knowing how to build it. The worker
supplies that implementation by registering a handler on a `*queue.Consumer`.
The consumer finds the handler by job name, decodes the payload, executes it, and
asks the driver to settle the delivery according to the result.

Start with a [configured driver](https://arkoren.dev/docs/guides/queues/drivers.md) and the shared `BuildReportJob` and
`BuildReport` payload from [Producing Jobs](https://arkoren.dev/docs/guides/queues/producing-jobs.md#defining-jobs).
The consumer owns execution policy, not the driver's connection lifecycle.

## Registering Handlers

Register handlers before starting the worker. With `context` imported, `queue`
from `arkoren.dev/core/service/queue`, and your application's
`reports.Build(ctx, reportID) error` operation, connect the job to its implementation:

```go
consumer := queue.NewConsumer(driver)
err := consumer.Handle(BuildReportJob, func(ctx context.Context, payload BuildReport) error {
	return reports.Build(ctx, payload.ReportID)
})
```

Each attempt gets a fresh decoded payload. The context carries consumer
cancellation, the job's execution deadline when configured, and propagated trace
context when telemetry is enabled. Pass it to database, storage, and HTTP calls.
A timeout cancels the context; it does not stop application code by force.

Handle `err` before running the worker. Invalid definitions return
`queue.ErrInvalidName` wrapping the definition error. A second registration for
the same job name returns `queue.ErrDuplicateName`, even if its payload type
differs. A nil handler is rejected without registration.

`Run` discovers queues once when it starts. A later handler on an already
consumed queue can handle subsequent deliveries, but adding a new queue does not
start another consume call until a later run. Register everything first to avoid
timing-dependent coverage.

## Selecting Queues

By default, a consumer reads every queue that has a registered handler. If report
generation needs dedicated workers, restrict an alternative consumer construction:

```go
consumer := queue.NewConsumer(driver, func(worker *queue.ConsumerBuilder) {
	worker.Queues("reports")
})
```

Register handlers on that consumer as above. Other handlers may remain registered,
but their queues are skipped. This restriction neither changes driver routing nor
provisions broker resources. A selected queue without a handler is not consumed;
if none remain, `Run` returns `queue.ErrNoQueues`.

Each `Queues` call replaces the previous restriction; `Queues()` restores all
handled queues. Inputs and the published restriction are copied. Construction
callbacks run once in order, and a retained draft cannot reconfigure the published
consumer. Telemetry and event buses remain shared services.

## Worker Lifetime

`Handle` does not begin consumption. Register `Run` as a supervised runner in an
existing application composition callback, with `application` imported from
`arkoren.dev/core/application`:

```go
compose := func(composer *application.Composer) {
	composer.Run(consumer.Run)
}
```

Pass the callback to your existing `app.Run(ctx, compose)`. Include the driver's
startup, readiness, and cleanup hooks from [Drivers](https://arkoren.dev/docs/guides/queues/drivers.md) when required.
RabbitMQ and NATS need those hooks; memory needs cleanup but no readiness probe;
SQS and Cloudflare have no driver hooks. Construction alone does not connect
RabbitMQ or NATS.

`Run` starts one driver `Consume` call per selected, handled queue. Shipped
drivers execute handlers sequentially within each call, including fetched
batches. Different queues run concurrently. For more concurrency on one queue,
run additional workers sharing the same physical work queue or JetStream durable;
they compete for deliveries rather than receiving a broadcast copy each.
Prefetch and batch size reserve work ahead, not parallel handler slots.

If any consume call returns an error, `Run` cancels its sibling queue consumers,
waits for them, and returns the first error. As an application runner, that error
stops the service; let your deployment supervisor restart a worker after broker
failure rather than silently leaving it idle. Ordinary handler errors whose
retry or failure settlement succeeds do not themselves end `Run`.

### Stopping Workers

Cancel the consumer and wait for it to return before stopping its driver or
closing a borrowed connection. The active handler must cooperate with cancellation;
the consumer cannot kill it. A handler that completes successfully still needs
its acknowledgement, so allow shutdown time for execution and settlement.

Settlement has a fresh 30-second budget after execution and is detached from
consumer cancellation. It can still fail if transport cleanup happens first or
the backend is unavailable. If the application's shutdown budget expires,
cleanup can release a driver while a handler is still running. See
[shutdown settlement](https://arkoren.dev/docs/guides/queues/delivery-and-retries.md#shutdown-settlement) for the retry
exception and its attempt-count implications.

Memory consumer cancellation preserves work not yet admitted; it does not drain
the backlog or revoke the active delivery. An active handler can acknowledge or
retry with the detached settlement context. `memoryqueue.Stop` is different: it
discards ready jobs and delayed retries, prevents new admissions, and waits for
active work only until its context ends. Retrying into that stopped driver fails.
Use a broker-backed driver if pending work must outlive the process.

## Handler Results

Return `nil` only when the work is complete; swallowing an error requests
acknowledgement of unfinished work. Return an ordinary error for a transient
failure. The consumer requests retry using the registered job's attempt limit
and backoff, or terminal failure if attempts are exhausted.

Use `queue.Permanent(err)` when another attempt cannot help. `queue.RetryAfter`
requests a specific delay while still respecting the attempt cap.
`queue.Reject` delegates unwanted-message handling to backend rejection policy;
it is not a portable delete or dead-letter operation. Choose among these using
[Delivery and Retries](https://arkoren.dev/docs/guides/queues/delivery-and-retries.md#handler-outcomes).

Make `reports.Build` repeat-safe with the stable report ID. A successful report
write followed by a lost acknowledgement can run again, and a lost lease can
overlap another attempt. For database-only effects, record completion and the
effect in one transaction. For remote effects, use provider idempotency and
reconcile uncertain results; a local completion flag cannot make a remote call
atomic. Required ordering also belongs in application state.

Job timeout and broker reservation are separate. NATS and SQS try to renew
reservations, but renewal failure can permit duplicates. Cloudflare cannot renew
its lease: all earlier handlers in a fetched batch consume the same lease time,
so keep batches small and finish execution plus settlement before expiry.

## Testing Handlers

Test the report-building operation directly for its application behavior. To
exercise JSON decoding and consumer settlement too, use
`arkoren.dev/core/service/queue/queuetest`. In a test with `t *testing.T`, construct
both facades around the same fake:

```go
fake := queuetest.New()
dispatcher := queue.NewDispatcher(fake)
consumer := queue.NewConsumer(fake)
```

Register your handler using `Handle` and check its registration error. Dispatch
a report payload, then drain due work through that consumer:

```go
if err := dispatcher.Dispatch(t.Context(), BuildReportJob, BuildReport{ReportID: "rpt-204"}); err != nil {
	t.Fatal(err)
}
if err := fake.Drain(t.Context(), consumer); err != nil {
	t.Fatal(err)
}
```

Assert the report's observable stored result using your application's test
fixtures. Successful draining proves pending due work was processed without a
settlement error, not that every handler succeeded: a terminally failed job also
drains. Inspect the fake's `Failed()` or `Rejected()` records when testing those
outcomes.

Delayed retries remain pending by default. `queuetest.WithScheduled()` includes
them by skipping waits, not advancing time; use it when testing attempt policy,
not elapsed-time behavior. A per-message delivery limit catches endless retry
loops. See [running dispatched jobs](https://arkoren.dev/docs/guides/testing/fakes.md#separate-execution)
for assertions and fake options.

See the [`queue` API](https://pkg.go.dev/arkoren.dev/core/service/queue) and
[`queuetest` API](https://pkg.go.dev/arkoren.dev/core/service/queue/queuetest).

---

Source: https://arkoren.dev/docs/guides/queues/delivery-and-retries.md
[HTML documentation](https://arkoren.dev/docs/guides/queues/delivery-and-retries)

# Delivery and Retries

A worker can finish a report and still fail to acknowledge its message. Another
worker can then receive the same work. Retry policy controls what the consumer
requests after an attempt; it cannot make the handler's effects and the broker's
state one atomic operation.

This page assumes the shared report job from [Producing Jobs](https://arkoren.dev/docs/guides/queues/producing-jobs.md)
and a [running consumer](https://arkoren.dev/docs/guides/queues/consuming-jobs.md). A **delivery** is a message reserved
for handling. **Settlement** is the consumer's request to acknowledge, retry,
reject, or terminally fail that delivery. Backend storage, retention, and native
delivery limits still apply; compare [Drivers](https://arkoren.dev/docs/guides/queues/drivers.md#compatibility-matrix).

## Acceptance Boundaries

Successful dispatch confirms driver acceptance, not handler execution. A dispatch
error after transmission starts can leave acceptance uncertain: the backend may
have stored the message before its reply was lost. Repeating the facade call
creates a new message ID and can enqueue another copy.

The consumer requests acknowledgement only after the handler returns `nil`.
If an external effect succeeds before the handler returns an error, retry can
repeat it. Even a successful handler can run again if acknowledgement fails or
the reply is uncertain. Losing a lease can allow another worker to start while
the first handler is still running.

There is no unconditional portable at-least-once delivery guarantee and no
exactly-once handler guarantee. Memory loses pending work at stop or process exit.
Broker retention, delivery limits, and retry-forwarding policy can remove work
before completion. RabbitMQ's retry confirmation covers its intermediate TTL
queue, not later forwarding; its [forwarding protections and expiry limit](https://arkoren.dev/docs/guides/queues/drivers.md#retry-forwarding)
matter even when the work queue is durable.

## Duplicate Effects

Use a stable business identity such as `BuildReport.ReportID`, not a transport ID,
to make repeats harmless. Typed handlers do not receive the message ID, and the
facade creates a fresh one on every dispatch. JetStream's duplicate window or SQS
FIFO deduplication does not deduplicate repeated facade payloads automatically.

For database-only work, enforce uniqueness and record completion in the same
transaction as the effect. For a remote effect, use the provider's idempotency key
when available and reconcile uncertain outcomes before repeating the operation.
A local flag written before the remote call can suppress work that never happened;
a flag written afterward leaves a crash window for duplicate effects. Neither
placement makes the remote call atomic.

Repeats and ordering are different concerns. Concurrent workers and retries can
reorder effects even when each effect is idempotent. Enforce sequence or version
checks in application state if order matters. A [durable outbox](https://arkoren.dev/docs/guides/queues/producing-jobs.md#durable-handoff)
protects dispatch intent across database commits but still needs duplicate-safe
consumption.

## Handler Outcomes

Return the result that matches what the application knows, not what you hope the
broker will do. The consumer converts it into a settlement request:

| Handler Result | Consumer Request |
| --- | --- |
| `nil` | Acknowledge completed work. |
| Ordinary error | Retry with job backoff, or fail terminally at the attempt cap. |
| `queue.RetryAfter(err, delay)` | Retry with this delay instead of backoff, still subject to the cap. |
| `queue.Permanent(err)` | Fail terminally regardless of remaining attempts. |
| `queue.Reject(err)` | Reject without a facade retry; backend rejection policy decides redelivery. |

For an upstream report service that returns a rate-limit delay, your handler can
override backoff. Here `upstream.Build` and `rateLimited` belong to your integration;
the latter recognizes an error with a usable `response.RetryAfter` duration:

```go
response, err := upstream.Build(ctx, payload.ReportID)
if rateLimited(err) {
	return queue.RetryAfter(err, response.RetryAfter)
}
```

Handle the remaining result normally. The explicit delay does not bypass the
attempt limit. Negative delays become zero, selecting driver timing; positive
delays are lower-bound requests that a driver can round up or reject, not silently
shorten. Backend availability and retention still determine whether delivery occurs.

If loading a report shows that its source record no longer exists, retrying cannot
repair that absence. With `errors` imported and your application's
`reports.ErrNotFound` sentinel, classify that branch:

```go
if errors.Is(err, reports.ErrNotFound) {
	return queue.Permanent(err)
}
```

`Permanent` preserves the cause and matches `queue.ErrPermanent` under
`errors.Is`. Check the operation's error before wrapping it: `Permanent(nil)` is
still a non-nil terminal outcome, not success. `RetryAfter` and `Reject` also
accept nil causes while retaining their explicit outcome.

Rejection is deliberately backend-specific. Prefer `Permanent` for irreparable
application failures unless you intentionally configured native rejection/redrive:

| Driver | Accepted Rejection | Accepted Terminal Failure |
| --- | --- | --- |
| Memory | Drops the message. | Logs and drops it. |
| RabbitMQ | Rejects without requeue; drops unless broker dead-letter policy applies. | Transfers to configured failure queue before acknowledging; otherwise rejects without requeue. |
| NATS | Terminates delivery, not a dead-letter transfer. | Publishes to configured failure subject before acknowledging; otherwise logs and acknowledges. |
| Amazon SQS | Leaves the message for visibility expiry and native redrive; can return repeatedly without redrive. | Sends to configured failure queue before deleting; otherwise logs and deletes. |
| Cloudflare | Makes no remote request; lease expiry and native retry/dead-letter policy still apply. | Logs and requests deletion; no failure archive. |

These are successful settlement paths, not unconditional guarantees after an RPC
error. The facade exposes no portable dead-letter inspection or replay API.

## Retry Policy

Set attempts and backoff on a fresh job declaration, using `queue` from
`arkoren.dev/core/service/queue` and `time`. This alternative report definition
allows one initial run and four retries:

```go
var BuildReportJob = queue.NewJob[BuildReport]("build-report", func(job *queue.JobBuilder) {
	job.OnQueue("reports")
	job.Retry(5)
	job.Backoff(2*time.Second, 2, 10*time.Second)
})
```

The four retries request delays of 2, 4, 8, and 10 seconds. `Retry(n)` counts
the first attempt. Zero, the default, imposes no facade cap; native caps and
retention can still end delivery. All shipped drivers report attempt counts.
A custom driver reporting zero cannot support facade attempt-limit enforcement.

`Backoff` takes an initial delay, multiplier, and maximum. A multiplier below one
or NaN becomes one; a zero maximum removes the cap. Negative durations are
invalid. Without backoff, the driver's default timing applies: memory can retry
immediately without a native attempt cap, so always choose a finite policy for
work that may keep failing.

For a custom curve, use an alternative construction with `BackoffFunc`:

```go
var BuildReportJob = queue.NewJob[BuildReport]("build-report", func(job *queue.JobBuilder) {
	job.OnQueue("reports")
	job.Retry(4)
	job.BackoffFunc(func(attempt int) time.Duration {
		return time.Duration(attempt-1) * 5 * time.Second
	})
})
```

The callback receives the attempt about to run, at least two, so these retries
request 5, 10, then 15 seconds. A negative result becomes zero; a nil callback
restores driver timing. It replaces any earlier `Backoff`, and vice versa.
The callback's captured state remains shared and must support concurrent use.

Check [driver bounds](https://arkoren.dev/docs/guides/queues/drivers.md) before choosing delays. RabbitMQ buckets delays
and has TTL/forwarding limits; NATS uses delayed negative acknowledgement; SQS
rounds up to seconds and allows at most 12 hours subject to remaining visibility;
Cloudflare rounds up to seconds and allows at most 24 hours. Unsupported-delay
settlement can fail even though the application correctly requested a retry.

## Execution Deadlines

Set `job.Timeout(duration)` in the definition callback to bound one attempt's
context. Zero means no deadline. Execution is synchronous: passing the deadline
does not force a return. If the handler returns an error after the deadline,
normal outcome and attempt policy apply. If it returns `nil`, the consumer still
acknowledges success. Panics are recovered as ordinary errors with stack text and
follow retry/cap policy; avoid secrets in panic values and logs.

Execution timeout is not the broker's reservation deadline. NATS renews every half
`AckWait`; SQS renews every half visibility duration, with a 12-hour limit from
the receive request. Failed renewal can permit concurrent redelivery. RabbitMQ
uses unacknowledged deliveries rather than a renewable lease; connection loss can
release unsettled work. Cloudflare has no renewal: budget for batch waiting,
execution, and settlement within visibility, and make execution honor its shorter
timeout. Increasing batch size does not increase concurrency within one consume
call.

## Shutdown Settlement

When the consumer's context ends, an active handler receives cancellation. If it
returns an error matching that context error, such as `ctx.Err()` or a `%w`
wrapper, the consumer requests retry with zero delay, bypassing the immediate
attempt-cap and backoff checks. A different error follows ordinary policy.
The reported delivery count is **not discounted**: a later ordinary failure can
therefore exhaust the job's cap. Job timeout alone does not select this shutdown
exception.

After decoding and execution finish, settlement gets a fresh 30-second context
detached from consumer cancellation, retaining context values. Unknown names
receive that same fresh budget without execution. Acknowledgement for completed
work can therefore finish after shutdown begins, but the context cannot keep a
closed connection alive or make a non-cooperative SDK interruptible.

Cancel and join consumers before stopping drivers or closing borrowed clients.
Allow shutdown time for both handler completion and settlement. If the application
budget expires first, driver cleanup can interrupt settlement. In memory,
canceling a consumer preserves its unadmitted backlog; stopping the driver instead
discards pending work and prevents a final retry from being enqueued.

An unsuccessful settlement is returned to the driver, not converted into
successful retry or failure. RabbitMQ ends consumption and closes its channel to
release unsettled jobs. NATS, SQS, and Cloudflare log the error and can keep
consuming while unsettled work becomes eligible for redelivery. A settlement
error does not prove that the backend ignored the request.

## Poison Messages

A **poison message** cannot complete with the current handler or envelope. A
payload that fails typed JSON decoding requests terminal failure immediately;
another attempt cannot repair those bytes. Drivers also handle malformed native
envelopes before the typed handler is called, using their own failure path.
Do not expect facade events for a message the facade never receives.

An unknown job name may instead mean an older worker received newly deployed
work. The consumer requests a 30-second retry until the reported delivery count
reaches ten, then requests terminal failure with `queue.ErrUnknownName`.
For a longer deployment window, construct a consumer with a different cap:

```go
consumer := queue.NewConsumer(driver, func(worker *queue.ConsumerBuilder) {
	worker.UnknownNameAttempts(20)
})
```

Zero restores ten; a negative value removes the facade cap, not broker limits.
The cap relies on reported attempts. Native delivery-limit exhaustion, such as
JetStream `MaxDeliver`, does not itself invoke the facade's terminal failure path
or create its failure record. Deploy compatible workers first and investigate
unknown-name failures instead of using unlimited retries as deployment policy.

## Failure Records

Configure archives on [Drivers](https://arkoren.dev/docs/guides/queues/drivers.md) before relying on terminal jobs being
retained. RabbitMQ, NATS, and SQS can transfer a failure envelope to a configured
destination before acknowledging/deleting the source. Publication failure leaves
the source unsettled. Transfer and source acknowledgement are not atomic: a lost
reply or crash can duplicate work or failure records. Archive access and retention
must protect job data, raw malformed bodies, and unsanitized error text.

With an event bus configured, `queue.JobFailed` is emitted only after the driver
accepts terminal settlement. It does not prove archive durability, undo remote
effects, or indicate a native dead-letter destination. Rejection emits its own
event rather than `JobFailed`; failed settlement emits no accepted-outcome event.
[Monitoring](https://arkoren.dev/docs/guides/queues/metadata-and-telemetry.md#outcome-events) shows event wiring and how
to distinguish handler telemetry from settlement evidence.

See the [`queue` API](https://pkg.go.dev/arkoren.dev/core/service/queue).

---

Source: https://arkoren.dev/docs/guides/queues/metadata-and-telemetry.md
[HTML documentation](https://arkoren.dev/docs/guides/queues/metadata-and-telemetry)

# Monitoring

A report request and its background execution happen at different times, often
in different processes. Queue telemetry connects those operations through trace
context carried beside the payload. Consumer events answer a different question:
which retry, rejection, or terminal-failure settlement did the driver accept?

Neither signal is a report-completion database or a durable failure archive.
Use [Delivery and Retries](https://arkoren.dev/docs/guides/queues/delivery-and-retries.md) to interpret ambiguous sends,
acknowledgement gaps, and backend failure handling.

## Trace Continuity

With a configured `driver` and your application's telemetry handle
`app.Telemetry()`, construct the dispatcher with explicit telemetry. Import
`arkoren.dev/core/service/queue` as `queue`:

```go
dispatcher := queue.NewDispatcher(driver, func(producer *queue.DispatcherBuilder) {
	producer.Telemetry(app.Telemetry())
})
```

Configure the worker independently; producer configuration does not reach another
process or configure its consumer:

```go
consumer := queue.NewConsumer(driver, func(worker *queue.ConsumerBuilder) {
	worker.Telemetry(app.Telemetry())
})
```

Without an explicit handle, both follow the [process default](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle)
at each operation, initially a no-op. Configure an exporter through the
[telemetry guide](https://arkoren.dev/docs/guides/observability/telemetry.md). Telemetry is not required for
dispatching or consuming and does not change the selected settlement outcome.

Dispatch records a producer span named `send reports`; each delivery records a
consumer span named `process reports`. The dispatcher injects trace context into
message metadata and the consumer extracts it, so attempts can appear as siblings
under the producer span. The handler receives the consumer span through its
context; context-aware logging can correlate its work with that trace.

Spans identify the logical destination with `messaging.destination.name`, the
transport with `messaging.system`, and the operation with
`messaging.operation.name` and `messaging.operation.type`. They also carry
`messaging.message.id`, `arkoren.queue.job`, and `arkoren.queue.attempt` when
available. Shipped system names are `memory`, `rabbitmq`, `nats`, `aws_sqs`, and
`cloudflare_queues`. `System("name")` on either construction builder overrides
driver discovery; unnamed custom drivers use `_OTHER`.

## Attempt Metrics

The queue records these instruments, also used by broadcasting:

| Instrument | Observation |
| --- | --- |
| `messaging.client.sent.messages` | Number of dispatch attempts reaching driver instrumentation, including failures. |
| `messaging.client.consumed.messages` | Number of delivery attempts, not distinct jobs. |
| `messaging.client.operation.duration` | Dispatch operation duration in seconds. |
| `messaging.process.duration` | Decode, execution, and settlement duration in seconds. |

Metric keys are `messaging.system`, `messaging.operation.name`,
`messaging.destination.name`, and `error.type` on errors. Message IDs, job names,
and attempt numbers are span attributes, not metric labels. A job succeeding on
its third attempt contributes three consumed observations. Do not interpret the
consumed counter as the number of successfully completed reports.

The consumer span and error classification describe decoding or the handler
result, **not settlement acceptance**. A `nil` handler result with a failed
acknowledgement still has successful handler telemetry. An ordinary error whose
retry settles successfully still records a failed attempt. Context errors use
`context.Canceled` or `context.DeadlineExceeded`; unknown jobs use
`queue.ErrUnknownName`; other causes use their Go type. Merely exceeding a
deadline does not mark a handler returning `nil` as failed.

## Outcome Events

To observe accepted settlements, supply an existing `*event.Bus` named `bus` from
`arkoren.dev/core/event` when constructing the consumer:

```go
consumer := queue.NewConsumer(driver, func(worker *queue.ConsumerBuilder) {
	worker.Events(bus)
})
```

This is an alternative construction; combine `Events` and `Telemetry` in one
callback if you need both. Nil events publish nothing. The bus is process-local,
not a broker archive or durable notification channel.

| Topic | Meaning |
| --- | --- |
| `queue.JobRetried` (`queue.job.retried`) | Driver accepted a policy retry; carries the requested delay. |
| `queue.JobRejected` (`queue.job.rejected`) | Driver accepted rejection, not proof of native dead-letter transfer. |
| `queue.JobFailed` (`queue.job.failed`) | Driver accepted terminal failure handling. |
| `queue.JobTimedOut` (`queue.job.timed-out`) | Handler returned an error after its execution deadline; emitted before settlement. |

For example, register a failure listener during application composition. Here
`composer` is an existing `*application.Composer` and
`alerts.Record(ctx, jobName, err) error` is your application operation. Import
`context` and `arkoren.dev/core/application` for the surrounding composition code:

```go
unsubscribe, err := bus.Listen(queue.JobFailed, func(ctx context.Context, outcome queue.JobFailedEvent) error {
	return alerts.Record(ctx, outcome.Name, outcome.Err)
})
composer.Require(err)
composer.Defer(unsubscribe)
```

`Require` prevents registering a nil cleanup after listener-registration
failure; `Defer` releases the subscription with its owner. Listeners run
synchronously, so keep them short and honor their context. Listener failures do
not change already accepted settlement.

Outcome payloads identify the message ID, job name, logical queue, and reported
attempt count. Retry, rejection, and failure carry error causes, but no job
payload. `JobFailed` confirms accepted terminal handling, not archive durability
or reversal of external effects. Rejection never also emits `JobFailed`.

Failed settlement emits no accepted-outcome event. An errored timed-out attempt
can emit `JobTimedOut` without a follow-up if settlement fails. Shutdown retries
and unknown-name retries do not emit `JobRetried`; the shutdown retry path also
skips timeout notification. Successful handlers emit none of these events.
Missing events therefore do not prove absence of attempts or broker side effects.
See [built-in queue events](https://arkoren.dev/docs/guides/events/built-in-events.md#queues) for their types.

## Message Metadata

`queue.Message.Metadata` is a string map traveling beside the JSON payload.
Broker drivers carry it inside the body, not as native message attributes, so
SQS's native attribute-count limit does not constrain it. It never selects the
queue or handler. A message without metadata arrives with a nil map; nil and
empty are equivalent.

Portable round trips cover at most 16 entries, keys of 1–64 bytes from
`A-Za-z0-9._-` (case-sensitive), valid UTF-8 values, and at most 4096 bytes of
keys plus values. Messages outside those bounds may be rejected. Constants
`MaxMetadataEntries`, `MaxMetadataKeyLength`, and `MaxMetadataSize` expose those
bounds.

`Dispatch` has no caller-metadata argument. With enabled telemetry it creates a
fresh propagation map, including `traceparent` and `tracestate` when supplied by
the propagator. Raw driver callers can supply metadata themselves, but then own
validation and encoding. Typed handlers receive the decoded payload and context,
not the raw envelope or an arbitrary metadata-map argument.

## Privacy Boundaries

Automatic telemetry records names, IDs, durations, attempt counts, systems, and
error classes. It does not record payloads, metadata values, or error text.
Choose names safe to export rather than embedding personal data in destinations
or job names. Anyone who can read broker messages can still inspect their payload
and propagated trace metadata.

Driver logs, handler logs, event error causes, native SDK diagnostics, and failure
archives are separate surfaces; they can contain unsanitized data and error text.
Keep secrets out of errors, apply logger redaction, and restrict broker/archive
access and retention. [Telemetry Privacy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md)
explains deployment-level controls.

## Testing Signals

Use `telemetrytest` to inspect recorded spans without an exporter. In a test with
`t *testing.T`, import `arkoren.dev/core/service/queue`, its `memoryqueue`
subpackage, `arkoren.dev/core/service/telemetry`, and its `telemetrytest`
subpackage. Use the report definition from
[Producing Jobs](https://arkoren.dev/docs/guides/queues/producing-jobs.md#defining-jobs):

```go
signals := telemetrytest.New()
dispatcher := queue.NewDispatcher(memoryqueue.New(), func(producer *queue.DispatcherBuilder) {
	producer.Telemetry(signals.Telemetry(t))
})
```

Dispatch one report and check the producer observation:

```go
if err := dispatcher.Dispatch(t.Context(), BuildReportJob, BuildReport{ReportID: "rpt-204"}); err != nil {
	t.Fatal(err)
}
signals.AssertSpan(t, "send reports", telemetry.String("arkoren.queue.job", "build-report"))
signals.AssertCounter(t, "messaging.client.sent.messages", 1,
	telemetry.String("messaging.destination.name", "reports"))
```

This checks producer instrumentation, not worker execution or broker retention.
To inspect `process reports`, give the consumer the same fake telemetry and
[drain a fake-backed job](https://arkoren.dev/docs/guides/queues/consuming-jobs.md#testing-handlers). Assert event
publication separately from spans when testing settlement behavior.

See the [`queue` API](https://pkg.go.dev/arkoren.dev/core/service/queue) and
[`telemetrytest` API](https://pkg.go.dev/arkoren.dev/core/service/telemetry/telemetrytest).

---

Source: https://arkoren.dev/docs/guides/scheduled-tasks/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/scheduled-tasks/overview)

# Overview

Session cleanup and daily reports need to run even when no request arrives.
The `schedule` service connects those application functions to a calendar.
It also accepts one-off work, such as sending a reminder ten minutes after a
user action.

## Scheduling Model

An **entry** is a named function with a recurring calendar specification. A
**task** is a named, typed payload definition for one-off jobs. Both register
on a `*schedule.Scheduler`; its driver decides when to deliver the work.

The scheduler owns registration, payload decoding, cooperative entry timeouts,
panic recovery, and run observations. The driver owns timers, persistence,
overlap coordination, and missed-tick behavior. Choose and construct one on
[Drivers](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md) before using the snippets below.

Memory runs independent timers in every process and loses deferred jobs on
restart. Temporal and Restate retain engine state and coordinate calendar
delivery across workers. Neither shared timing nor an overlap policy guarantees
exactly-once application effects: make repeat deliveries harmless.

Focused snippets keep error results visible but omit routine propagation checks.
Complete application code must handle those errors.

## Registering Work

With an existing `scheduler`, define a cleanup entry using your application's
`pruneSessions` function, which has type `func(context.Context) error`. Import
`time` and `arkoren.dev/core/service/schedule`:

```go
prune := schedule.NewEntry("prune-sessions", pruneSessions, func(entry *schedule.EntryBuilder) {
	entry.When(schedule.UTC().Every(5 * time.Minute))
	entry.Timeout(time.Minute)
})
```

This matches UTC minutes :00, :05, :10, and so on, rather than five minutes
after construction. The timeout cancels the function's context; it cannot stop
code that ignores cancellation. Pass that context to database and network calls.

Register the definition before starting the scheduler:

```go
err := scheduler.Add(prune)
```

`Add` validates the complete batch and adds all entries or none. Names are
1–64 lowercase letters, digits, or hyphens, with no leading or trailing hyphen;
entries and tasks share one namespace. [Calendar Schedules](https://arkoren.dev/docs/guides/scheduled-tasks/calendar-schedules.md)
explains calendar fields and overlap policy. For payload-bearing work created
by an application action, use [Deferred Tasks](https://arkoren.dev/docs/guides/scheduled-tasks/deferred-tasks.md) instead.

## Running Schedulers

During application composition, register the configured scheduler as a runner.
With an existing `composer` from `arkoren.dev/core/application`:

```go
composer.Run(scheduler.Run)
```

`Run` hands the registered entries and dispatcher to the driver, then blocks
until cancellation or driver failure. Once it starts, `Add` and `Handle` return
`schedule.ErrRunning`. The facade is single-use: a second `Run` also returns
that error, even after the first finishes.

Every worker must register matching names and compatible handlers before
`Run`; only names and JSON payloads travel through the engine, not Go closures.
Keep captured services alive for the full execution lifetime.

Cancellation requests shutdown, not forced callback termination. In particular,
memory stops admitting timer work but drains admitted callbacks without a
driver timeout, on contexts detached from runner cancellation. Bound entry and
task work explicitly. The application's runner-wait budget can expire and let
cleanup proceed while callbacks still use their dependencies. See
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md) and the
[driver-specific shutdown rules](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md).

## Manual Runs

For an operator action or a focused test, run a registered entry without
waiting for a tick:

```go
err := scheduler.RunNow(ctx, "prune-sessions")
```

`RunNow` waits for the function and applies its timeout, panic recovery,
logging, events, and telemetry. It works independently of `Run` and preserves
caller cancellation. It bypasses the driver's calendar and overlap guards,
so it can overlap a scheduled run even with `WithoutOverlapping`. Use
application-level coordination if that concurrency would be unsafe.

For remote operations, `scheduler.MCPTools("jobs")` supplies an `entries` tool
for definitions and next calendar matches, and a `run` tool that waits for
`RunNow`. Select tools explicitly on a
[protected MCP endpoint](https://arkoren.dev/docs/guides/mcp/securing-mcp.md); omit `run` for read-only callers.
Availability policy does not replace resource authorization in application code.

## Inspecting Entries

To review configured execution bounds without running work, filter the
registered definitions:

```go
limited := scheduler.Entries().Filter(func(_ int, entry schedule.Entry) bool {
	return entry.TimeoutValue() > 0
})
```

`Entries` returns an eager `collection.Slice[schedule.Entry]` snapshot in
registration order, not execution history. Replacing its elements cannot edit
the scheduler, and later registrations do not update it. Entries and specs are
immutable; their execution functions and captured dependencies remain shared.

## Events and Telemetry

If you need run notifications, select an event bus when constructing the
scheduler. With an existing `driver` and application `app`:

```go
scheduler := schedule.New(driver, func(clock *schedule.Builder) {
	clock.Events(app.Events())
	clock.Telemetry(app.Telemetry())
})
```

Events are off by default. Configured events report failed and deliberately
skipped runs, accepted deferrals, and accepted cancellations. They do not prove
delivery or that cancellation removed pending work. See
[Error Handling](https://arkoren.dev/docs/guides/scheduled-tasks/error-handling.md) and
[Built-in Events](https://arkoren.dev/docs/guides/events/built-in-events.md#scheduled-tasks).

Without explicit telemetry, the scheduler follows `telemetry.Default()` when
each run starts. Enabled telemetry emits a `schedule run` span and count and
duration metrics labeled by name, entry/task kind, and `ok`, `skipped`, or
`failed` outcome. Calls made with the handler context become child spans.
Payloads, job IDs, and error text are excluded from automatic telemetry;
failure logs and events still carry errors. Configure the shared service using
[Telemetry](https://arkoren.dev/docs/guides/observability/telemetry.md).

For every function and option, see [the `schedule` API reference](https://pkg.go.dev/arkoren.dev/core/service/schedule).

---

Source: https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/scheduled-tasks/drivers)

# Drivers

Choose memory for independent process-local timers and disposable deferred
work. Choose Temporal or Restate when calendar delivery must be coordinated
across workers or deferred jobs must survive worker restarts. Prefer an engine
your team already operates, then check its calendar and failure behavior below.

## Compatibility Matrix

This compares behavior exposed through Arkoren, not the engines' full feature
sets. Restart survival depends on retained engine state. Shared calendar timing
does not guarantee exclusive application effects or exactly-once execution.

| Driver | Shared Calendar Timing | Deferred Tasks | Worker Restart Survival | Non-UTC Entries | Missed Ticks | Workflow Host |
| --- | --- | --- | --- | --- | --- | --- |
| [Memory](#memory) | No | Yes | No | Yes | No | No |
| [Temporal](#temporal) | Yes | Yes | Yes | Partial — Explicit named zones work; Local is rejected. | Partial — Recovery is limited by CatchupWindow, one minute by default. | Yes |
| [Restate](#restate) | Yes | Yes | Yes | Partial — Explicit named zones work; Local is rejected. | Partial — The pending tick runs once, not once per missed calendar match. | Yes |

Memory runs every entry in every replica. Memory and Restate use the core
calendar; Temporal evaluates its own calendar and runs an autumn repeated time
twice. See [Time Zones](https://arkoren.dev/docs/guides/scheduled-tasks/time-zones-and-dst.md) before using local wall-clock times.
Memory does not retry failed handlers; Temporal configures one attempt; Restate
treats ordinary handler errors as terminal but retries missing dispatch and
errors matching `schedule.ErrUnknownName`. Design effects to tolerate repetition.

Engine drivers attach to the vendor's [Worker Hosts](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md).
Construct the schedule driver before starting the host, register matching
entries and handlers on every worker, and run the scheduler using
[the ordinary runner setup](https://arkoren.dev/docs/guides/scheduled-tasks/overview.md#running-schedulers). Host connection,
credentials, endpoint security, and cleanup belong in that shared guide.

Run installation commands in your application's module directory. Constructor
snippets use `schedule` from `arkoren.dev/core/service/schedule`; timing examples
also import `time`. Handle constructor errors before using the returned driver.

`scheduler.Driver()` exposes the raw driver, and `DriverAs[T]()` finds a concrete
type through wrappers exposing `Unwrap() schedule.Driver`. Raw operations bypass
the facade's registration and observations; use ordinary scheduler methods for
application work.

## Memory

The bundled `memoryschedule` driver uses Go timers and needs no separate
installation. Use it for tests, development, or a single replica when losing
deferred jobs is acceptable.

### Creating Timers

Construction starts nothing and cannot fail:

```go
import "arkoren.dev/core/service/schedule/memoryschedule"

driver := memoryschedule.New()
scheduler := schedule.New(driver)
```

`scheduler.Run` starts timers; the driver has no separate `Start` or `Stop`.
Add a [calendar entry](https://arkoren.dev/docs/guides/scheduled-tasks/calendar-schedules.md#defining-entries), or register a
[deferred handler](https://arkoren.dev/docs/guides/scheduled-tasks/deferred-tasks.md#registering-handlers) and call `After`.
Deferrals accepted before `Run` remain pending; overdue jobs run at its next
opportunity. There is no missed-calendar-tick recovery.

The only optional driver setting is `Builder.Logger`, which defaults to
`slog.Default()` and receives driver failure and overlap-skip records.

### Shutdown Boundaries

Runner cancellation stops timer admission and discards unstarted work belonging
to that run. Process exit loses all jobs. Already admitted callbacks retain
context values but not the runner's cancellation or deadline; `Run` waits for
them without a drain timeout.

Set positive entry timeouts and bound task I/O. A callback waiting only for
its detached context can prevent the driver from returning forever. Application
shutdown budgets limit runner waiting, not callback lifetime, so cleanup can
start while callbacks still hold dependencies. No timeout kills Go code.

Deferrals accepted after admission closes, including during draining, remain
pending for a later raw-driver run. The raw driver rejects concurrent `Run`
calls through the drain with `memoryschedule.ErrAlreadyRunning`, and can run
again after it returns. The scheduler facade itself remains single-use.

The driver copies submitted payload bytes. An already-canceled submission
returns the context error and no ID; an already-canceled cancellation leaves
pending work unchanged. Canceling a submission's context after acceptance does
not withdraw its job. `Cancel` removes pending work but cannot stop an admitted
dispatch.

For timer tests, use [Controlling Time](https://arkoren.dev/docs/guides/testing/time.md). `RunNow` checks a
handler without testing timer delivery.

For every function and option, see [the `memoryschedule` API reference](https://pkg.go.dev/arkoren.dev/core/service/schedule/memoryschedule).

## Temporal

Temporal stores recurring entries as Schedules and deferred tasks as delayed
workflows. It retains them across worker restarts while its engine state is
retained. Install the module:

```sh
go get arkoren.dev/driver/temporal
```

### Connecting Workers

With a [configured Temporal host](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#temporal),
construct the driver before `host.Start`. It registers a workflow and activity
without making network calls.

> [!WARNING]
> `Run` deletes Temporal Schedules beginning with `<prefix>-` that are absent
> from your declarations. Give each application a unique prefix on a shared
> namespace. Workers sharing that prefix must declare the same schedule set;
> conflicting deployments can overwrite or remove each other's schedules.

```go
import "arkoren.dev/driver/temporal/temporalschedule"

driver, err := temporalschedule.New(host, func(clock *temporalschedule.Builder) {
	clock.Prefix("accounts")
})
scheduler := schedule.New(driver)
```

The prefix defaults to `arkoren` and names workflows, activities, and schedules.
`Run` creates or updates `<prefix>-<entry-name>` schedules, prunes undeclared
ones, then waits. Deferred workflows named `<prefix>-deferred-*` are not pruned.
Register [calendar entries](https://arkoren.dev/docs/guides/scheduled-tasks/calendar-schedules.md) and
[task handlers](https://arkoren.dev/docs/guides/scheduled-tasks/deferred-tasks.md) before running this scheduler.

A dialing host must connect first: live-context `Run`, `Defer`, and `Cancel`
otherwise return `temporalschedule.ErrNotStarted`. An already-canceled driver
`Run` returns nil without connecting. A host wrapping an existing native client
can submit deferrals before local `Run`. A client-only host needs matching
workers elsewhere.

Existing activities can arrive between host startup and scheduler startup.
They wait for dispatch readiness within their activity timeout, so start
`Run` promptly: readiness waiting consumes that execution budget.

### Timing Policy

If a brief outage must recover more than the default minute of missed ticks,
construct the driver with an explicit catch-up window. This is an alternative
construction using the same host, not a second registration on it:

```go
driver, err := temporalschedule.New(host, func(clock *temporalschedule.Builder) {
	clock.Prefix("accounts")
	clock.CatchupWindow(10 * time.Minute)
	clock.MaxRunTime(20 * time.Minute)
})
```

`MaxRunTime` defaults to one hour and sets the activity bound when an entry has
no positive timeout; deferred tasks use that fallback too. A positive entry
timeout overrides it. Nonpositive `MaxRunTime` and `CatchupWindow` restore their
defaults. Driver `Logger` defaults to `slog.Default()`. Native timing bounds do
not forcibly terminate application callbacks.

Each run has one activity attempt. `WithoutOverlapping` maps to Temporal's
skip policy; otherwise all overlaps are allowed. Temporal has approximately
second-level schedule resolution, and actual firing can lag. Its own calendar
skips spring-gap times and fires autumn-overlap times twice.

### Shutdown Boundaries

Every driver `Run` exit removes its dispatcher and cancels active dispatch
contexts, including on reconciliation failure. It does not join callbacks or
stop code that ignores cancellation. Later activities wait for a new raw-driver
run instead of reusing the stopped dispatcher; a facade cannot be restarted.

The host's worker-stop grace is native policy, not a hard callback deadline.
Its early stop notification does not immediately cancel active callbacks.
See [host shutdown](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#shutdown-boundaries) for
grace, native option precedence, and client ownership.

### Native Access

Use the shared host's `Client()` for SDK operations outside the facade:

```go
native := host.Client()
```

A dialing host exposes it only after connection; `temporal.NewFrom` borrows a
supplied client and exposes it immediately. Configure native TLS, authentication,
and logger policy through [the host](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#connection-policy).
Keep independent native schedules outside the driver's prefix: reconciliation
can overwrite native edits to managed schedules.

### Local Engine

Use the [Temporal local engine](https://arkoren.dev/docs/guides/workflows/drivers.md#local-engine) in your
application. Construct this driver before starting that host and register
`scheduler.Run`. Check a handler's effect after a calendar tick to verify
engine delivery; `RunNow` verifies only the handler. The linked profile includes
health verification and disposable-engine cleanup.

For every function and option, see [the `temporalschedule` API reference](https://pkg.go.dev/arkoren.dev/driver/temporal/temporalschedule).

## Restate

Restate stores each recurring entry in a virtual object whose tick schedules
the next tick before dispatching the current work. Deferred tasks use delayed
sends. Choose it when you operate a Restate server with retained state:

```sh
go get arkoren.dev/driver/restate
```

### Connecting Workers

Configure the [Restate host](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#restate), including
a reachable endpoint, request-identity verification, and deployment registration.
Construct this driver before `host.Start` because it binds native services.

> [!WARNING]
> Use a distinct ownership prefix per application. Startup reconciliation removes
> undeclared entries from that prefix's index. Service names drop separators and
> capitalize the following letter, so prefixes must also have distinct normalized
> names; different strings alone do not ensure isolation.

```go
import "arkoren.dev/driver/restate/restateschedule"

driver, err := restateschedule.New(host, func(clock *restateschedule.Builder) {
	clock.Prefix("accounts")
})
scheduler := schedule.New(driver)
```

The default prefix `arkoren` creates `ArkorenSchedule`, `ArkorenScheduleIndex`,
and `ArkorenRunner`. The index key preserves the exact prefix; ingress requests
escape it as one path segment. Run `scheduler.Run` alongside `host.Serve` using
the shared host's lifecycle setup. `Run` retries initial ingress publication
every second until accepted or canceled, including while the deployment is not
yet registered. It does not fail immediately on an unreachable ingress.

Register [calendar entries](https://arkoren.dev/docs/guides/scheduled-tasks/calendar-schedules.md) and
[task handlers](https://arkoren.dev/docs/guides/scheduled-tasks/deferred-tasks.md) before `Run`. Deferrals can be accepted before
local `Run` if ingress knows the deployment; delivery retries when dispatch is
not ready. Unlike initial publication, `Defer` does not retry submission errors.
The returned job ID is a Restate invocation ID accepted by `Cancel`.

### Timing Policy

For a different native handler budget, select `MaxRunTime` during construction:

```go
driver, err := restateschedule.New(host, func(clock *restateschedule.Builder) {
	clock.Prefix("accounts")
	clock.MaxRunTime(20 * time.Minute)
})
```

This alternative construction sets abort and inactivity timeouts on native Tick
and Run handlers. Zero restores the default hour; negative values leave server
defaults, not an unlimited-execution guarantee. Driver `Logger` defaults to
`slog.Default()`. Entry timeouts remain cooperative application-context bounds.

`WithoutOverlapping` runs inline in the entry's serialized tick invocation;
ordinary entries dispatch through the runner service. A delayed pending tick
runs once and computes its next match after the current time, rather than
replaying every missed calendar match. Ordinary application failures are terminal;
missing dispatch and errors matching `schedule.ErrUnknownName` are retryable.
Runs can repeat, so keep application effects idempotent.

### Deployment Safety

Changing an expression, zone, or overlap flag starts a new schedule generation.
Removing an entry clears its active spec but retains its generation. Adding the
same name later advances the generation so old queued ticks cannot revive it.
Preserve that retained generation if you manage native state manually; it does
not prevent duplicate application effects.

> [!WARNING]
> Before upgrading existing schedule state, let in-flight `Remove` invocations
> finish on the old deployment. Removal records individual state clears rather
> than `ClearAll`; replaying an incompatible recorded history can fail. If an
> older deployment already removed a name, drain its pending ticks before
> reintroducing it because its deleted generation history cannot be recovered.

Task JSON values, including explicit `null`, are preserved. Upgrade serving
workers before relying on null-payload preservation; older workers discard
those bytes. Recurring messages still dispatch with nil payload bytes,
including legacy recurring messages carrying `Payload:null`.

Each rollout creates a deployment. Keep handlers compatible with in-flight
work and use the host's [deployment and shutdown policy](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#deployment-policy).
`Cancel` requests native invocation cancellation; it cannot forcibly stop
application code already executing. Keep captured dependencies alive until
callbacks finish.

### Native Access

The host's `Query` method exposes the admin SQL API for inspection. Keep admin
credentials restricted and do not expose arbitrary queries to untrusted callers.
For an existing host and operation context:

```go
rows, err := host.Query(ctx, "SELECT * FROM sys_invocation LIMIT 10")
```

This reads native invocation records, not the facade's registered definitions.
The workflow driver can bind to the same host; its separate
[cancellation retention](https://arkoren.dev/docs/guides/workflows/drivers.md#cancellation-retention) policy
is not a prerequisite for recurring schedules.

### Local Engine

Use the [Restate local engine](https://arkoren.dev/docs/guides/workflows/drivers.md#local-engine-1), construct
this driver before host startup, and run both `host.Serve` and `scheduler.Run`.
Check the handler's effect after a tick to test durable delivery. The linked
isolated development profile disables request-identity verification; do not use
that setting on an exposed endpoint. Follow its cleanup instructions to remove
the disposable server and schedule state.

For every function and option, see [the `restateschedule` API reference](https://pkg.go.dev/arkoren.dev/driver/restate/restateschedule).

## Hatchet

Hatchet integration is pending and unpublished. It is not an installable
scheduler option and is excluded from the available-driver matrix. Use memory,
Temporal, or Restate instead.

For the portable contract, see [the `schedule` API reference](https://pkg.go.dev/arkoren.dev/core/service/schedule).

---

Source: https://arkoren.dev/docs/guides/scheduled-tasks/calendar-schedules.md
[HTML documentation](https://arkoren.dev/docs/guides/scheduled-tasks/calendar-schedules)

# Calendar Schedules

A stock refresh that runs every weekday belongs on a calendar, rather than
being deferred again by each request. A calendar **entry** combines a stable
name, your execution function, and an immutable `schedule.Spec` describing
matching times. It does not start a timer until the scheduler runs.

Use an existing [configured scheduler](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md) for the examples below.
For work triggered once by a user action, use [Deferred Tasks](https://arkoren.dev/docs/guides/scheduled-tasks/deferred-tasks.md).

## Defining Entries

With your application's `refreshStock` function of type
`func(context.Context) error`, create a weekday refresh at 06:15 UTC. Import
`time` and `arkoren.dev/core/service/schedule`:

```go
refresh := schedule.NewEntry("refresh-stock", refreshStock, func(entry *schedule.EntryBuilder) {
	entry.When(schedule.UTC().Weekdays().Hour(6).Minute(15))
})
```

The callback edits a fresh draft; publication copies its settings and retains
the execution function without calling it. Configure entries only during
construction. Retaining the draft cannot change the published definition, but
the function's captured services remain shared and must outlive executions.

`When` is required. An omitted schedule, nil execution function, or invalid spec
is a declaration error reported by `Err` and `scheduler.Add`, not an inferred
default. Names contain 1–64 lowercase letters, digits, or hyphens, with no edge
hyphen. They share a namespace with deferred tasks.

## Calendar Fields

Start with `schedule.UTC()` unless the job must follow a person's local clock;
[Time Zones](https://arkoren.dev/docs/guides/scheduled-tasks/time-zones-and-dst.md) covers named zones and daylight saving time.
Unset time fields finer than the coarsest selected field become zero, while
coarser fields match every value. For example:

```go
daily := schedule.UTC().Hour(6)
hourly := schedule.UTC().Minute(15)
monthly := schedule.UTC().Day(1)
```

These mean 06:00 daily, minute :15 of every hour, and midnight on the first day
of each month. Date-only specs use midnight; a zone with no fields is invalid.
`Day(31)` skips months without that date rather than rolling into the next month.

If refreshes are needed on selected weekdays, pass several values to `On`:

```go
twice := schedule.UTC().On(time.Monday, time.Thursday).Hour(6)
```

This matches 06:00 on Monday and Thursday. `Day` cannot be combined with `On`,
`Weekdays`, or `Weekends`. Field methods return independent specs, sort and
deduplicate values, and keep the first error. Setting the same field twice is
an error rather than an override.

## Aligned Cadences

For repeated maintenance within each hour, choose an aligned calendar cadence:

```go
cadence := schedule.UTC().Every(15 * time.Minute)
```

This matches :00, :15, :30, and :45, not fifteen minutes after startup or the
previous completion. `Every` accepts positive whole seconds, minutes, or hours
under 24 hours. The step must divide its unit's cycle evenly: 15 minutes works,
7 minutes does not. Finer fields become zero; coarser fields remain unrestricted
unless you select them, such as with `Weekdays()`.

Calendar matches are intended run times, not guaranteed delivery times. Memory
does not recover missed ticks; engine recovery differs. Check
[the driver comparison](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md#compatibility-matrix) before using ticks as
a record of business work. Persist business progress separately if every period
must eventually be processed.

## Cron Expressions

If an existing operational schedule is already expressed as cron, use `Cron`
instead of the field methods:

```go
overnight := schedule.UTC().Cron("0 2 * * 1-5")
```

This matches 02:00 on weekdays. Expressions have five fields, or six with
seconds first. Fields accept numeric values, lists, ranges, wildcards, and
steps; weekday numbers are 0–6, starting with Sunday. Named months, weekday
names, and cron macros are not part of this parser.

`Cron` cannot be combined with field methods, including `Every`. If both
day-of-month and day-of-week fields are restricted in raw cron, either can
match; that differs from combining restrictions as an AND condition.

## Overlap Policy

By default, another scheduled run can start while the previous one remains
active. If overlapping refreshes would compete for the same resource, construct
the entry with `WithoutOverlapping` and a positive timeout:

```go
refresh := schedule.NewEntry("refresh-stock", refreshStock, func(entry *schedule.EntryBuilder) {
	entry.When(schedule.UTC().Every(5 * time.Minute))
	entry.WithoutOverlapping()
	entry.Timeout(4 * time.Minute)
})
```

This is an alternative to the earlier definition. Memory skips a tick while
that entry is busy in the same process; it does not coordinate other replicas.
Temporal uses its native skip policy. Restate executes inline within its
serialized tick invocation, so a pending tick waits rather than overlapping
that invocation. These are driver policies, not a cross-replica lock around
all application effects.

`RunNow` bypasses those guards. A retried or duplicated delivery can repeat
effects even without concurrent runs. Make the function idempotent, and use
application-level locking or fencing when correctness requires it.

`Timeout` defaults to zero, meaning no facade execution limit; negative values
are invalid. A positive value cancels the supplied context after that duration.
The function must observe the context or pass it to I/O: no scheduler timeout
forcibly stops code. Engine drivers also have native execution policies on
[Drivers](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md). A function returning nil after cancellation still counts
as successful; see [Error Handling](https://arkoren.dev/docs/guides/scheduled-tasks/error-handling.md#timeout-boundaries).

## Registering Entries

After constructing the definitions, register them before `Run`:

```go
err := scheduler.Add(refresh)
```

`Add` accepts multiple entries atomically. An invalid definition or duplicate
name rejects the whole batch, leaving existing registrations unchanged. Starting
`Run` closes registration permanently for that facade.

Construction callbacks run synchronously once in order, and later callbacks can
replace valid timing or timeout settings. The first invalid setting cannot be
repaired by a later callback. A supplied nil callback or callback panic propagates;
runtime handler panic recovery does not apply during construction.

To inspect a future match without starting work, use `Next` with an existing
time named `after`:

```go
next := refresh.Next(after)
```

The result is strictly after `after`, or zero if the entry is invalid or no
match exists within five years. This uses the core calendar, not an engine
delivery prediction. Start work with [the scheduler runner](https://arkoren.dev/docs/guides/scheduled-tasks/overview.md#running-schedulers).

For every function and option, see [the `schedule` API reference](https://pkg.go.dev/arkoren.dev/core/service/schedule).

---

Source: https://arkoren.dev/docs/guides/scheduled-tasks/deferred-tasks.md
[HTML documentation](https://arkoren.dev/docs/guides/scheduled-tasks/deferred-tasks)

# Deferred Tasks

A reminder created after a user action needs one future execution, not a
recurring calendar entry. A **task** fixes the name and payload type for that
work. Register its handler on a scheduler, then submit individual jobs with
their payloads and due times.

The examples assume a [configured scheduler](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md) named `scheduler`.
Memory loses jobs on restart; Temporal and Restate retain them in engine state.
Choose that durability boundary before accepting work from users.

## Defining Tasks

Keep the reminder's payload and task definition in a package shared by producers
and workers. Import `arkoren.dev/core/service/schedule`:

```go
type Reminder struct {
	AccountID string `json:"account_id"`
}

var RemindAccount = schedule.NewTask[Reminder]("remind-account")
```

The definition does not register a handler or schedule anything. Names share
the entry namespace and contain 1–64 lowercase letters, digits, or hyphens,
without edge hyphens. `Handle` validates the name; definitions using one name
must agree on their payload type.

Payloads round-trip through `encoding/json/v2`. Prefer a small identifier over
a snapshot of account data: the handler can load current state and decide
whether the reminder is still appropriate. Keep pending payloads compatible
with future worker deployments; a decode failure occurs before the handler runs.

## Registering Handlers

Before `scheduler.Run`, register the application function that sends the
reminder. Here, `sendReminder` has type `func(context.Context, string) error`.
Import `context` and `time` to bound its execution:

```go
err := scheduler.Handle(RemindAccount, func(ctx context.Context, reminder Reminder) error {
	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
	defer cancel()

	return sendReminder(ctx, reminder.AccountID)
})
```

Tasks have no entry-style `Timeout` setting, so bound the handler's work and
pass its context into I/O. This deadline is cooperative, not forced termination.
In particular, memory drains admitted tasks on contexts detached from runner
cancellation and has no driver drain timeout. Keep captured services alive until
handlers finish, even if the application's shutdown wait expires.

`Handle` rejects nil handlers and duplicate names. Registration closes when
`Run` starts. The facade also requires this registration before `At` or `After`,
including on a process that only submits jobs; otherwise it returns
`schedule.ErrUnknownName`. Register compatible names and handlers on all
execution workers. Only the name and JSON payload travel, never the closure.

Make the handler idempotent and recheck current business state before effects.
One-off means one submitted job, not a guarantee that its effects happen once.
[Driver retry and delivery policies](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md#compatibility-matrix) differ.

## Deferring Work

After the application has validated and authorized an account action, defer its
reminder relative to the current time:

```go
id, err := scheduler.After(ctx, RemindAccount, Reminder{AccountID: "acct-42"}, 10*time.Minute)
```

Success returns a driver-chosen job identifier and means accepted for later
execution, not that the handler ran or a message was delivered. Retain the ID
if you might cancel the job. A zero or negative delay makes it due now or in
the past; the driver runs it at its next opportunity, not synchronously.

If the due instant is already known, use `At` with a nonzero `time.Time`
named `due` instead:

```go
id, err := scheduler.At(ctx, RemindAccount, Reminder{AccountID: "acct-42"}, due)
```

These are alternative submissions; calling both creates two jobs. `At` rejects
zero time. Encoding and driver errors retain their causes, and the facade does
not retry failed submissions automatically. A context or transport failure is
not proof that an engine accepted nothing; reconcile uncertain outcomes before
blindly submitting another job when duplicate effects matter.

Deferring a job and committing an application database change are separate
operations. If that boundary must be reliable, use an application-owned durable
handoff or reconciliation design rather than assuming the scheduler joins the
transaction.

Temporal needs a connected host for submission. Restate needs a registered
deployment known to ingress; its initial calendar-publication retry loop does
not apply to deferred submissions. Memory holds pre-run submissions until `Run`
and loses them on process exit. See [Drivers](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md) for those readiness
and shutdown boundaries.

With events configured, successful `At` and `After` calls publish
`schedule.task.deferred` with name, ID, and due time, never the payload. This
is a submission observation, not a delivery receipt.

## Canceling Jobs

When a reminder is no longer needed, pass its returned identifier to `Cancel`:

```go
err = scheduler.Cancel(ctx, id)
```

Unknown and finished IDs succeed. A successful call therefore means the
cancellation request was accepted, not proof that pending work existed or was
removed. The configured `schedule.task.canceled` event has that same boundary.

Memory cannot interrupt an admitted dispatch. Temporal terminates the native
deferred workflow; Restate requests invocation cancellation. Neither can
forcibly stop arbitrary Go code already executing. Recheck business state in
the handler when a stale reminder would cause harm.

## Testing Submissions

To test that application code submits the right reminder without sending mail,
use `scheduletest.New()` from
`arkoren.dev/core/service/schedule/scheduletest`. It records jobs without firing
them. In a test with `t *testing.T`, construct a separate scheduler and register
an inert handler:

```go
fake := scheduletest.New()
scheduler := schedule.New(fake)
err := scheduler.Handle(RemindAccount, func(context.Context, Reminder) error {
	return nil
})
if err != nil {
	t.Fatal(err)
}
```

Invoke the application operation under test with this scheduler. For a focused
submission assertion, call `After` directly:

```go
_, err = scheduler.After(t.Context(), RemindAccount, Reminder{AccountID: "acct-42"}, 10*time.Minute)
if err != nil {
	t.Fatal(err)
}

fake.AssertDeferred(t, RemindAccount, func(reminder Reminder) bool {
	return reminder.AccountID == "acct-42"
})
```

This proves the submitted payload, not handler execution or mail delivery.
Test the handler separately with controlled application dependencies. For real
timer behavior, see [Controlling Time](https://arkoren.dev/docs/guides/testing/time.md).

To execute due fake jobs through the scheduler, start `scheduler.Run` and then
call `fake.Drain(ctx)`. Before `Run` installs dispatch, `Drain` returns
`scheduletest.ErrNotRunning`; inside a `synctest` bubble, wait with
`synctest.Wait()` before draining. Future-due jobs remain pending until due.

For every function and option, see [the `schedule` API reference](https://pkg.go.dev/arkoren.dev/core/service/schedule).

---

Source: https://arkoren.dev/docs/guides/scheduled-tasks/time-zones-and-dst.md
[HTML documentation](https://arkoren.dev/docs/guides/scheduled-tasks/time-zones-and-dst)

# Time Zones

A daily account digest should follow its recipients' local clock, while
background cleanup usually needs only a predictable machine schedule. Choose
the zone as part of each calendar spec: UTC avoids seasonal clock changes;
a named location follows that region's time rules.

This page assumes you can [define a calendar entry](https://arkoren.dev/docs/guides/scheduled-tasks/calendar-schedules.md).
Time zones describe matching calendar instants, not guaranteed execution or
arrival times.

## Choosing Zones

Use `schedule.UTC()` unless the wall-clock time matters to people. For example,
`schedule.UTC().Hour(6)` matches 06:00 UTC each day without a daylight saving
transition. Driver outages and missed-tick policies still affect delivery.

For a local digest at 09:00, use `schedule.In(location)`. Avoid `time.Local`
for shared scheduling: its meaning depends on the machine, and Temporal and
Restate reject the `Local` zone with `schedule.ErrUnsupported`. Load an explicit
named zone instead.

## Loading Locations

Load the desired location using the standard library before constructing a spec.
Import `time` and `arkoren.dev/core/service/schedule`:

```go
location, err := time.LoadLocation("Europe/Amsterdam")
```

Handle the load error before using `location`. `schedule.In(nil)` retains an
invalid-spec error rather than selecting a default zone.

With your application's `sendDigest` function of type
`func(context.Context) error`, construct the entry:

```go
digest := schedule.NewEntry("account-digest", sendDigest, func(entry *schedule.EntryBuilder) {
	entry.When(schedule.In(location).Hour(9))
})
```

The match is 09:00 Amsterdam time, with minutes and seconds defaulting to zero.
Register it with `scheduler.Add(digest)` before starting the scheduler.

Zone loading needs available timezone data. If a minimal container lacks zone
files, embed the standard library's database with a blank import in your program:

```go
import _ "time/tzdata"
```

Deploy timezone data consistently across workers and engines when local-clock
behavior matters; `Next` cannot predict a remote engine using different rules.

## Clock Changes

Daylight saving time creates a spring **gap**, when a range of local times does
not exist, and an autumn **overlap**, when a range occurs twice. The core
calendar used by memory and Restate skips nonexistent times and chooses only
the first occurrence of repeated times.

In Amsterdam on March 29, 2026, the clock jumps from 02:00 to 03:00. A 02:30
entry has no matching instant that day and is skipped, not moved to 03:30.
On October 25, 2026, the hour beginning at 02:00 repeats. The core calendar
matches 02:30 only on its first occurrence.

If a business process must run for every date, do not depend on a tick inside
those transition hours. Choose UTC or a local time outside the transition,
and persist business progress so missed delivery can be reconciled separately.

## Inspecting Matches

Before choosing a local time, inspect the core calendar without running any
handler. Reuse the loaded `location` and import `fmt` for this inspection:

```go
spec := schedule.In(location).Hour(2).Minute(30)
before := time.Date(2026, time.March, 28, 12, 0, 0, 0, location)
fmt.Println(spec.Next(before))
```

The result skips March 29:

```text
2026-03-30 02:30:00 +0200 CEST
```

`Spec.Next` and `Entry.Next` return the first matching instant strictly after
the supplied time. Invalid specs and searches with no match within five years
return zero time. These methods calculate core calendar matches, not remote
engine delivery times.

## Registration Warnings

`scheduler.Add` examines the next year for a transition intersecting a non-UTC
entry's matching times. If it finds one, it logs one warning for that entry
with its name, zone, date, wall time, and `skipped` or `run once` effect.

The warning does not reject registration or change the spec. It describes the
core calendar, including when Temporal's own overlap behavior differs. If it
identifies a risky business time, choose another time or zone; do not treat
successful registration as proof that every date will have an execution.

## Driver Calendars

Memory and Restate use the core calendar. Temporal uses its engine calendar:

| Driver | Spring Gap | Autumn Overlap |
| --- | --- | --- |
| Memory | Skips the nonexistent time. | Matches the first occurrence only. |
| Temporal | Skips the nonexistent time. | Matches both occurrences. |
| Restate | Skips the nonexistent time. | Matches the first occurrence only. |

A Temporal entry in a repeated hour must tolerate two calendar firings, in
addition to ordinary duplicate-delivery risks. Move it outside the overlap if
two intended firings are not appropriate. See [Drivers](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md) for
missed-tick recovery, overlap policy, and engine configuration. Hatchet remains
pending and is not an available alternative.

For every function and option, see [the `schedule` API reference](https://pkg.go.dev/arkoren.dev/core/service/schedule).

---

Source: https://arkoren.dev/docs/guides/scheduled-tasks/error-handling.md
[HTML documentation](https://arkoren.dev/docs/guides/scheduled-tasks/error-handling)

# Error Handling

Scheduled work has no request caller waiting to recover a failure. The
scheduler records what your handler returned; the driver decides whether the
engine retries or moves on. Keep those responsibilities separate when designing
alerts and repeatable business operations.

## Run Outcomes

Every executed entry or task produces one scheduler outcome:

| Outcome | Handler Result | Driver Receives |
| --- | --- | --- |
| `ok` | Returns nil. | Nil. |
| `skipped` | Returns an error matching `schedule.ErrSkipped`. | Nil. |
| `failed` | Returns another error or panics. | An error. |

An overlap tick suppressed by a driver never calls the handler, so it is not
the same as a handler returning `ErrSkipped`. Likewise, reaching a deadline
does not independently classify the run: a handler returning nil after
cancellation still reports `ok`.

## Skipping Work

If a stock refresh has nothing new to process, return `ErrSkipped` rather than
an ordinary failure. With application functions `stockChanged` and `refreshStock`,
import `context` and `arkoren.dev/core/service/schedule`. The first function has
type `func(context.Context) (bool, error)`; the second has type
`func(context.Context) error`:

```go
func refreshIfNeeded(ctx context.Context) error {
	changed, err := stockChanged(ctx)
	if err != nil {
		return err
	}
	if !changed {
		return schedule.ErrSkipped
	}

	return refreshStock(ctx)
}
```

The scheduler logs a skip, labels telemetry `skipped`, and returns nil to the
driver. With an event bus configured, it emits `schedule.run.skipped`, not
`schedule.run.failed`. Wrapped skips work because classification uses
`errors.Is`.

You can also skip when application coordination denies a lock, but a skip does
not itself establish exclusive execution. Choose appropriate
[locking and fencing](https://arkoren.dev/docs/guides/cache/locks.md) if concurrent effects would violate
correctness.

## Timeout Boundaries

To bound a recurring refresh, configure a positive `Timeout` during entry
construction. Import `time`:

```go
refresh := schedule.NewEntry("refresh-stock", refreshIfNeeded, func(entry *schedule.EntryBuilder) {
	entry.When(schedule.UTC().Every(5 * time.Minute))
	entry.Timeout(4 * time.Minute)
})
```

This cancels the handler context after four minutes. Pass it into database and
network calls, or explicitly check it during loops. If the function returns
`ctx.Err()`, the run is failed; returning nil instead reports success even if
the deadline passed. The default zero timeout imposes no facade bound, and
negative values are declaration errors.

No timeout forcibly terminates Go code. Driver-native activity or handler
timeouts can record an engine timeout while application code continues.
Memory detaches admitted callbacks from runner cancellation and drains without
a driver timeout; an application's shutdown wait budget is not a callback
lifetime bound. Bound task handlers themselves as shown in
[Deferred Tasks](https://arkoren.dev/docs/guides/scheduled-tasks/deferred-tasks.md#registering-handlers), and keep their
dependencies alive until execution ends. See [Drivers](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md) for native
timeout and shutdown policies.

## Failure Events

For an ordinary error, the scheduler logs the failure and duration, marks the
run span as failed, and returns an error wrapped as `run <name>: ...`. Execution
panics are recovered into errors carrying the panic value and stack, with
`schedule.panic` as telemetry's error type. Invalid task JSON fails during
decoding before the handler runs.

To notify an operator without embedding alert delivery in every handler,
configure events when constructing the scheduler. With an existing application
`app` and selected `driver`:

```go
scheduler := schedule.New(driver, func(clock *schedule.Builder) {
	clock.Events(app.Events())
})
```

Before execution, register a typed listener. Here, `notifyOnCall` is your
application's `func(context.Context, string) error` notification function:

```go
_, err := app.Events().Listen(schedule.RunFailed, func(ctx context.Context, failure schedule.RunFailedEvent) error {
	return notifyOnCall(ctx, failure.Name)
})
```

The event contains the name, entry/task kind, duration, and original error.
It also fires for `RunNow` failures. Listener errors and panics are logged by
the event bus and do not change the run result; remaining listeners are skipped
after that failure. Alert delivery is not a transaction with execution.

Automatic telemetry excludes error text, payloads, and job IDs, but failure logs
and events carry the error. Do not put secrets in errors. See
[Built-in Events](https://arkoren.dev/docs/guides/events/built-in-events.md#scheduled-tasks) for payload details
and [scheduler telemetry](https://arkoren.dev/docs/guides/scheduled-tasks/overview.md#events-and-telemetry) for instrumentation.

## Driver Recovery

A failed callback does not, by itself, end `Scheduler.Run`. The driver receives
the dispatch error and applies its execution policy:

- Memory logs the failure without retrying it; the next calendar tick is independent.
- Temporal configures one activity attempt per firing; a failed run does not
  prevent the next schedule firing.
- Restate makes ordinary application errors terminal. Missing dispatch and
  any error matching `schedule.ErrUnknownName` remain retryable, including an
  application error wrapping that sentinel.

Do not add submission retries as a substitute for understanding those policies.
The facade does not automatically retry `At` or `After`. A repeated submission
creates another job, and an uncertain transport result can require reconciliation.

Make effects idempotent even when failed handlers are not retried: repeated
delivery and interrupted execution can still repeat work. Overlap policy is
not deduplication and does not cover manual `RunNow`; see
[Overlap Policy](https://arkoren.dev/docs/guides/scheduled-tasks/calendar-schedules.md#overlap-policy).

Runner startup failures are separate. Temporal returns reconciliation errors
from `Run`, allowing the application runner supervisor to stop the application.
Restate instead retries initial publication every second until ingress accepts
it or the context is canceled. An unreachable server can therefore leave
publication waiting rather than immediately failing the runner.

## Registration Errors

Handle returned registration and submission errors using `errors.Is` rather
than matching their text:

| Error | Meaning |
| --- | --- |
| `schedule.ErrInvalidName` | An entry or task name violates the portable naming rule. |
| `schedule.ErrInvalidEntry` | A required spec or handler is missing, timing is invalid, a timeout is negative, or `At` received zero time. |
| `schedule.ErrDuplicateName` | An entry or task already uses that name. |
| `schedule.ErrUnknownName` | `At` or `After` has no registered handler, or `RunNow` cannot find an entry. Application errors can also wrap this identity. |
| `schedule.ErrRunning` | Registration followed `Run`, or the facade was run a second time. |
| `schedule.ErrUnsupported` | The driver cannot perform the operation, such as an engine entry using the Local zone. |

`Add` validates its entire batch before changing registration. Entry definitions
retain their first declaration error; later callbacks cannot repair it.
`When` is required even if construction callbacks are omitted or empty.

Construction callbacks run once in order, including for invalid declarations.
A nil execution function is a returned declaration error, but a supplied nil
construction callback or callback panic propagates. Runtime handler panic
recovery does not apply during construction.

For every function and option, see [the `schedule` API reference](https://pkg.go.dev/arkoren.dev/core/service/schedule).

---

Source: https://arkoren.dev/docs/guides/workflows/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/workflows/overview)

# Workflows Overview

A signup can send a welcome message now and wait days for confirmation. A
**workflow** keeps that process together as a named Go function. Each execution
is a **run**, with your identifier, a status, and a typed result.

With a durable driver, recorded history lets the engine rebuild an interrupted
run on another worker. The memory driver executes the same definitions locally
but loses runs when the process exits. Choose and configure a
[driver](https://arkoren.dev/docs/guides/workflows/drivers.md) before using the client examples.

## Process Boundaries

A [queue job](https://arkoren.dev/docs/guides/queues/overview.md) is one unit of background work. A
[scheduled task](https://arkoren.dev/docs/guides/scheduled-tasks/overview.md) starts work at a time. A workflow
coordinates several actions and decisions over its lifetime: send a message,
wait for confirmation, then activate an account or let the invitation expire.

A **step** performs an external effect, such as sending mail or updating a
database. A **signal** supplies a typed value from outside the run. The workflow
function coordinates those operations; it does not perform their I/O itself.
Durable drivers can suspend between operations without keeping your application
goroutine alive throughout the wait.

Durability is not an atomic transaction with your database or mail service.
An effect can succeed before its step result is recorded, causing another
attempt after recovery. Make step effects idempotent. Likewise, committing a
database change and starting a workflow are two separate operations unless your
application provides a durable handoff, such as an outbox.

## Defining Work

For onboarding, pass a small payload that both client and worker can encode.
These examples use `arkoren.dev/core/service/workflow`:

```go
type User struct {
	ID    int    `json:"id"`
	Email string `json:"email"`
}
```

Your application supplies `sendWelcome`, with signature
`func(context.Context, User) (string, error)`. It sends through your mail service,
honors cancellation, and tolerates repeated calls for the same user. Declare it
as a step so the driver can record its result:

```go
var SendWelcome = workflow.NewStep("send-welcome", sendWelcome)
```

A workflow can return that result directly. Give every workflow a positive
`MaxDuration`; registration rejects a definition without it.

```go
var Onboard = workflow.New("onboard", func(c *workflow.Context, user User) (string, error) {
	return c.Step(SendWelcome, user)
}, func(definition *workflow.Builder) {
	definition.MaxDuration(96 * time.Hour)
})
```

This baseline ends after sending the welcome message. To keep the run open for
confirmation, replace its body with the version in
[Signals and Waiting](https://arkoren.dev/docs/guides/workflows/signals-and-waiting.md#awaiting-confirmation). The whole-run
limit must exceed that legitimate wait as well as the steps around it.

Definitions are immutable once built. Builder callbacks run synchronously, once
in order, on a fresh draft; retaining that draft cannot reconfigure the published
definition. A supplied nil callback or a callback panic propagates. Expanded nil
or empty callback slices supply no edits.

Focused snippets in this category show error results but omit routine
propagation. Complete function bodies and examples about recovery keep their
checks; application code must handle the remaining errors.

## Replay Rules

An engine can **replay** the workflow function from the beginning, supplying
recorded operation results to reconstruct its state. Given those same results,
the function must make the same calls in the same order.

- Put network, database, file, and other I/O in [workflow steps](https://arkoren.dev/docs/guides/workflows/steps.md).
  Do not read mutable global state in coordination code.
- Use `c.Now()` and `c.UUID()` instead of reading the clock or random numbers
  directly. The driver makes their values stable across replay.
- Do not start goroutines or use your own channels in the workflow function.
  Use [durable waits](https://arkoren.dev/docs/guides/workflows/signals-and-waiting.md) instead. These rules also apply to
  `Select` callbacks and to code using the native context from `c.Raw()`.
- Inputs and outputs must round-trip through the shared codec. The default is
  `encoding/json/v2`; a `time.Duration` field does not marshal, so represent
  payload durations as an integer number of seconds.

A workflow change can invalidate histories still in flight. A new workflow name
keeps old and new coordination separate. An input version works only when you
retain the old branch and route existing runs to compatible workers; changing a
version field alone does not migrate history. Check the
[provider rollout limits](https://arkoren.dev/docs/guides/workflows/drivers.md) before replacing workers.

`MaxDuration` ends an overlong run with `workflow.ErrTimeout`; it cannot forcibly
terminate arbitrary Go code. `Retention` controls how long finished records stay
available where the provider supports it. Retained records, rather than the
identifier alone, provide duplicate-start protection.

## Registering Definitions

A worker needs the workflow and every step it can execute. Validate them together
before constructing the driver or starting its host:

```go
registry, err := workflow.Handlers(Onboard, SendWelcome)
```

Handle `err` at startup. Workflows and steps share a name namespace: names contain
1–64 lowercase letters, digits, or inner hyphens. Registration reports duplicate
names, nil functions, invalid declarations, and missing run limits. Constructors
retain the first declaration error; `NewSignal` instead panics for an invalid
signal name, so declare signals at startup too.

The registry snapshots definitions and policies, not the services captured by
their functions. Client and worker must agree on names, payload types, and codec.
Custom codecs must produce valid JSON and support concurrent use. Set them on
both `RegistryBuilder.Codec` and `ClientBuilder.Codec`; configuring one role does
not configure the other.

Native drivers register work through a [worker host](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md);
their construction belongs on [Drivers](https://arkoren.dev/docs/guides/workflows/drivers.md).

Once the driver is ready, [start and inspect a run](https://arkoren.dev/docs/guides/workflows/running-workflows.md).
For declarations and options, see the
[`workflow` API reference](https://pkg.go.dev/arkoren.dev/core/service/workflow).

---

Source: https://arkoren.dev/docs/guides/workflows/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/workflows/drivers)

# Workflow Drivers

Use memory for process-local work and tests. Choose Temporal or Restate when
runs must survive worker replacement, provided the engine retains their history.
An existing engine is a useful starting point, but check the operations and
retention your process needs before committing to it. Hatchet is pending and
unpublished; it is not an available integration.

Examples use the [onboarding definitions](https://arkoren.dev/docs/guides/workflows/overview.md#defining-work) and import
`arkoren.dev/core/service/workflow`. Handle registry and constructor errors
before using their results. A **runner** registers definitions for execution;
a **driver** starts and controls runs. Native drivers share a
[worker host](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md) with scheduler drivers.

## Compatibility Matrix

These capabilities describe Arkoren's adapters, not the vendors' entire SDKs.
Restart survival assumes retained engine data, not protection against engine
data loss. Memory clients must share one driver in one process.

| Driver | Restart Survival | Owned Child Runs | Detached Runs | Start Delay | Client-Only Process |
| --- | --- | --- | --- | --- | --- |
| [Memory](#memory) | No | Yes | Yes | Yes | No |
| [Temporal](#temporal) | Yes | Yes | Yes | Yes | Yes |
| [Restate](#restate) | Yes | No | Yes | Yes | Yes |

`Child` owns cancellation only when it creates a run; joining an existing run
must never acquire that ownership. Memory and Temporal preserve this distinction
atomically. Restate returns `workflow.ErrUnsupported` before submitting child
work. `Spawn` is detached and is not a cancellation-equivalent fallback.

| Driver | Finished Retention Setting | Retained Pre-Execution Cancellation | Zero-Backoff Retries |
| --- | --- | --- | --- |
| [Memory](#memory) | Yes — One hour by default, in this process only. | Yes — While the process and retained record remain. | Yes |
| [Temporal](#temporal) | Partial — Namespace policy; workflow Retention is ignored. | Yes — Subject to native history retention. | No — Multiple attempts with zero initial backoff are rejected. |
| [Restate](#restate) | Yes — One day by default. | Partial — Requires positive retention and persisted preflight-retention activation. | Partial — Explicit zero is supported with native millisecond precision. |

Cancellation acceptance alone does not establish a retained outcome or permission
to reuse an identifier. See [Restate retention](#cancellation-retention) before
depending on duplicate protection for delayed or queued runs.

All three drivers support [client and worker observations](https://arkoren.dev/docs/guides/workflows/running-workflows.md#events-and-telemetry).
Their propagation boundaries differ:

| Driver | Client Calls | Step Attempts | Start-To-Attempt Carrier | Child/Spawn Carrier |
| --- | --- | --- | --- | --- |
| [Memory](#memory) | Yes | Yes | Yes — Copied start carrier. | Yes |
| [Temporal](#temporal) | Yes | Yes | Yes — Dedicated input envelope. | Yes |
| [Restate](#restate) | Yes | Yes | Yes — Invocation headers extracted in actual attempts. | Partial — Spawn only; Child is unsupported. |

## Memory

Memory is included in `arkoren.dev/core`. It runs workflows in local goroutines;
replicas share neither runs nor signals, and restart loses both. It uses the
system clock, so [synctest](https://arkoren.dev/docs/guides/workflows/testing.md#workflow-decisions) can advance waits and
backoff without real delays. It does not replay coordination code.

### Connecting Locally

Build a registry before constructing the driver:

```go
import "arkoren.dev/core/service/workflow/memoryworkflow"

registry, err := workflow.Handlers(Onboard, SendWelcome)
```

After handling `err`, wrap the driver in the ordinary facade:

```go
driver := memoryworkflow.New(registry)
client := workflow.NewClient(driver)
```

There is no startup hook. Register `driver.Stop` with application cleanup and
give cleanup a deadline. Stop cancels runs and joins workflow and step-attempt
goroutines until they return or its context ends. It cannot kill code that
ignores cancellation; a later Stop can continue joining it. After Stop, new
starts return `memoryworkflow.ErrStopped`. A nil registry executes nothing.

### Local Policy

The only driver setting is `memoryworkflow.Builder.Logger`; nil selects
`slog.Default()`. Step retry and timeout policies belong to the definitions,
not the driver. Finished runs remain for their `Retention`, one hour when zero;
expiry removes results and duplicate-start protection.

Use [Starting Runs](https://arkoren.dev/docs/guides/workflows/running-workflows.md#starting-runs) with this `client`.
With the confirmation-aware definition, signaling then reading `Result` returns
the supplied token. Signals buffer at most 1024 undelivered values per run and
signal name; another send returns `memoryworkflow.ErrQueueFull` without enqueuing
that value. Unknown or finished runs return `workflow.ErrNotFound`.

See the [`memoryworkflow` API reference](https://pkg.go.dev/arkoren.dev/core/service/workflow/memoryworkflow).

## Temporal

Temporal executes workflow functions as native workflows and steps as activities.
Choose it when you use Temporal's workers and tooling, or need parent-owned child
runs. From your application's module directory, install the driver:

```sh
go get arkoren.dev/driver/temporal
```

### Connecting Workers

Create a [Temporal host](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#temporal) and register your runner
before starting that host. Construction does no network I/O:

```go
import "arkoren.dev/driver/temporal/temporalworkflow"

runner, err := temporalworkflow.NewRunner(host, registry)
```

After checking the constructor error, use its client driver:

```go
client := workflow.NewClient(runner.Driver())
```

For a separate producer process, set the host's `NoWorker(true)` and use
`NewDriver`, supplying the registry to derive native execution timeouts and
reject unknown workflow names:

```go
driver, err := temporalworkflow.NewDriver(host, func(driver *temporalworkflow.DriverBuilder) {
	driver.Registry(registry)
})
```

Without that optional registry, the runner still enforces `MaxDuration`, but the
client sets no server execution-timeout backstop. A dialing host needs `Start`
before calls; a host made with `temporal.NewFrom` exposes the borrowed client
immediately without replacing worker startup or taking ownership of that client.

### Matching Processes

Client and worker must agree on namespace, task queue, codec, and prefix. Both
`DriverBuilder.Prefix` and `RunnerBuilder.Prefix` default to `arkoren`; both also
accept a `Logger`, defaulting to `slog.Default()` for adapter records. The
runner's required positional registry also supplies its driver registry; there
is no `RunnerBuilder.Registry` setting.

Workflow logging uses Temporal's replay-aware logger and native level filtering.
Bound and grouped `slog.LogValuer` attributes resolve before forwarding, preserving
their redaction behavior. Host connection, TLS, and native SDK options belong to
[Worker Hosts](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#temporal).

The workflow type is `<prefix>-wf-<name>`, its step activity is
`<prefix>-step-<name>`, and the facade run ID is the native workflow ID. IDs are
unique across workflow names in one namespace while history remains. Result,
status, cancel, and signal calls describe and verify the workflow type first;
a wrong workflow name returns `ErrNotFound` without issuing a command or returning
output. Grant describe permission as well as command permissions. These checks
do not replace application authorization.

Use the [run-control examples](https://arkoren.dev/docs/guides/workflows/running-workflows.md) after host startup. With
matching routing, a signaled onboarding run returns its confirmation token.
Metadata is stored in memo; tracing uses a separate input envelope and needs no
host propagator hook. Deploy carrier-aware workers before traced producers:
older workers cannot decode traced envelopes. Pause producers if worker-first
rollout is not possible. Legacy untraced histories keep their argument shape.

### Retry Limits

Positive step timeouts become activity start-to-close timeouts. `Timeout(0)`
removes that attempt limit; the workflow's `MaxDuration` supplies the native
schedule-to-close bound. A permanent failure becomes non-retryable.
The worker logs an error for each failed step attempt, including attempts that
will be retried; an attempt log does not by itself mean the run failed.

Positive backoff caps apply from the first retry. An uncapped declaration asks
for Go's largest duration rather than the native implicit cap, but server limits
can reduce it. The tested two-attempt policy on native server 1.29.1 retained
`876000h`; this is evidence for that server and policy, not a universal maximum.
Inspect the persisted policy if this limit matters.

Multiple attempts with zero initial backoff return `workflow.ErrUnsupported`
before activity encoding or scheduling, even if the handler would succeed first
time. The driver does not substitute a positive interval or retry the whole
workflow. Single attempts and explicitly positive backoff remain supported.

> [!WARNING]
> Older zero-backoff histories recorded an activity command that replacement
> workers now reject; replay can fail with `TMPRL1100`. Keep affected runs on
> their original workers until completion, or drain them before replacement.
> Mixed workers on an unversioned task queue are not safe routing. There is no
> automatic migration or added replay marker; replay retained histories before
> rollout.

### Run Lifetimes

Sleep uses a native timer; waits select over signals, timers, and cancellation.
An internal timer enforces `MaxDuration`, with the native execution timeout as a
backstop. Step, child-result, existing-child join, and Spawn-acknowledgment waits
end on the parent's cancellation or duration limit without waiting for their
native futures to finish.

Only an atomically created child is owned by the parent; it receives cancellation
or parent-close termination. Joining an existing ID observes its original run
and ignores new input, without canceling that run when the parent ends. This
ownership decision survives replay without another history command or marker.
Spawn is detached but waits for start acknowledgment within the parent's budget.
An acknowledgment error does not prove the child never started: reconcile that
same ID before replacing work.

Terminal errors preserve native chain order: an outer timeout remains
`ErrTimeout`, and an outer application error is not replaced by a timeout cause.
Application error kinds remain matchable; unrelated lookup and transport errors
remain errors. Workflow panics fail the run rather than retrying a workflow task.
`Retention` and `Description` are ignored; retention belongs to the namespace.
Poll `c.Done()` non-blockingly, never by a blocking native channel read.

### Native Access

`host.Client()` exposes `go.temporal.io/sdk/client.Client`, and `c.Raw()` exposes
`go.temporal.io/sdk/workflow.Context`. Native operations bypass facade observations
and do not relax the [replay rules](https://arkoren.dev/docs/guides/workflows/overview.md#replay-rules).

### Local Engine

For a disposable development engine, save this in your application's `compose.yaml`:

```yaml
services:
  temporal:
    image: temporalio/temporal:1.5.1
    command: ["server", "start-dev", "--ip", "0.0.0.0", "--headless"]
    ports:
      - "127.0.0.1:17233:7233"
```

Run `docker compose up -d`, then run
`docker compose exec temporal temporal operator cluster health --address 127.0.0.1:7233`
until it succeeds. Configure the host's `HostPort("127.0.0.1:17233")` and
`TaskQueue("onboarding")`, then register the runner and
[lifecycle hooks](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#temporal). Start a run and read its result
to verify delivery, not merely server health. `docker compose down --volumes`
removes the disposable engine and history. The CLI image tag does not establish
the native server's version or duration limits.

See the [`temporalworkflow` API reference](https://pkg.go.dev/arkoren.dev/driver/temporal/temporalworkflow).

## Restate

Restate calls an HTTP endpoint in your application. Workflow functions become
workflow services, and steps run inline as durable actions. Choose it for durable
execution through that serving model when you do not need parent-owned `Child`
runs. Install from your application's module directory:

```sh
go get arkoren.dev/driver/restate
```

### Connecting Workers

Configure a reachable endpoint and request-identity keys using the
[Restate host](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#restate). Before `host.Start`, bind the
workflow services and construct a separate client driver:

```go
import "arkoren.dev/driver/restate/restateworkflow"

runner, err := restateworkflow.NewRunner(host, registry)
```

After handling that error, create the facade:

```go
driver, err := restateworkflow.NewDriver(host)
client := workflow.NewClient(driver)
```

Binding does not listen, serve, or register a deployment. Register host `Start`
and `Stop` as startup/cleanup hooks and `Serve` as a runner, as described in
[Worker Hosts](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md#restate). A producer-only process uses
`ClientOnly(true)` and `NewDriver`, which binds no services.

### Matching Processes

Use the same codec and prefix in every client and worker. Both role builders
expose `Prefix` (default `arkoren`) and `Logger` (nil selects `slog.Default()`).
Service naming removes prefix separators and uppercases the following letter;
the default workflow `onboard` becomes `ArkorenWf_onboard`. `runner.Services()`
returns the bound service names in workflow-name order.

After deployment registration, use [run control](https://arkoren.dev/docs/guides/workflows/running-workflows.md) to start
and signal the confirmation-aware onboarding workflow. Reading the token from
`Result` verifies endpoint delivery. Duplicate starts return `ErrAlreadyStarted`;
an undeployed workflow returns `ErrNotFound`.

Signals resolve durable promises, keeping the first value. For accepted runs
still queued or delayed, the adapter forces delivery only after proving pending
state. Lookup and output-probe failures remain errors rather than being treated
as missing runs. Cancel needs access to admin SQL introspection as well as ingress.

Trace headers use `arkoren-trace-*`, separate from `arkoren-meta-*` metadata and
native attempt headers. Upgrade old workers before expecting trace continuity.

### Retry Limits

The adapter explicitly supplies initial delay, multiplier, attempts, and maximum
interval. Zero initial delay stays zero; zero maximum requests Go's largest
duration instead of the SDK default cap. Native delays have millisecond precision
and the multiplier is `float32`.

Step execution is cooperative. Positive `Timeout` supplies a context deadline;
after the body returns, late success becomes `workflow.ErrTimeout`. No adapter
goroutine abandons the body to escape that deadline, so noncooperative code can
prevent return. Zero supplies no attempt deadline. The SDK logs an error for
each failed attempt it retries.

### Run Lifetimes

`MaxDuration` starts a durable timer when execution begins, not at submission.
Blocking runtime operations observe it; it does not forcibly interrupt running
Go code. Native handler inactivity and abort timeouts are fixed at 24 hours.
Cancel closes a local attempt's Done and requests native cancellation; a running
step or another process observes it at a durable-operation boundary.

`Child` returns `ErrUnsupported` before send, attach, or cancellation. A durable
receipt identifies an invocation but cannot atomically distinguish creating it
from joining another owner's run across replay. A lookup then send is not safe
ownership. Spawn remains supported for explicitly detached work only.

> [!WARNING]
> Drain runs that called the earlier `Child` implementation on their original
> deployment before replacing it. Rejection changes their journal sequence;
> forced registration does not migrate that history. Completed retained outputs
> remain readable. Spawn is not an equivalent cancellation fallback.

### Cancellation Retention

`Retention` keeps finished outputs and promises for the configured window; zero
uses Restate's one-day default. Retained cancellation of delayed or queued runs
additionally requires positive effective retention and persisted activation of
`preflight_invocation_termination_retention` on the relevant partitions.

With that profile, pre-execution cancellation keeps `ErrCanceled`, `Canceled`
status, and duplicate protection for the retention window. Without it, the legacy
profile discards the invocation: Result and Status return `ErrNotFound`, another
Cancel can return `ErrNotFound`, and Start can immediately reuse the key. That
native behavior is not permission to repeat business work without reconciliation.

The tested Restate 1.7.12 includes the opt-in feature introduced in 1.7.8. Operators
request it with
`RESTATE_EXPERIMENTAL_ENABLE_PREFLIGHT_INVOCATION_TERMINATION_RETENTION=true`
and must ensure persisted activation. Activation establishes a minimum server
version: plan the compatibility change before applying it to an existing cluster,
and do not downgrade below 1.7.8 after the feature barrier applies. It cannot
recover already discarded records. See the
[native release notes](https://github.com/restatedev/restate/blob/v1.7.12/release-notes/v1.7.8.md#opt-in-retention-for-terminated-pre-flight-invocations).

The driver neither enables nor verifies this profile. Health proves HTTP
readiness; `/version` reports requested flags, not partition-barrier activation.
Public admin SQL on 1.7.12 cannot expose the internal feature state needed to
prove activation. None is a retained-cancellation readiness gate, and there is
no local tombstone or silent fallback. VQueues are not required for the feature.

### Native Access

`c.Raw()` exposes `restate.WorkflowContext` from `github.com/restatedev/sdk-go`;
keep its calls [replay-safe](https://arkoren.dev/docs/guides/workflows/overview.md#replay-rules). `host.Query` accesses admin SQL:
restrict credentials and never expose arbitrary queries to untrusted callers.
Each rollout is a new deployment, so preserve compatibility with runs in flight.

### Local Engine

For a fresh disposable server, save this as your application's `compose.yaml`.
The flag requests preflight retention; it is not proof of persisted activation
or a safe upgrade procedure for an existing cluster.

```yaml
services:
  restate:
    image: docker.restate.dev/restatedev/restate:1.7.12
    environment:
      RESTATE_EXPERIMENTAL_ENABLE_PREFLIGHT_INVOCATION_TERMINATION_RETENTION: "true"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "127.0.0.1:18080:8080"
      - "127.0.0.1:19070:9070"
```

Run `docker compose up -d`, then `curl --fail http://localhost:19070/health`
until it succeeds. This checks HTTP health only. For an application on your
machine, configure the host to reach ingress/admin and let the server reach back:

> [!WARNING]
> The following endpoint skips request-identity verification. Anyone who can
> reach port 9080 can invoke handlers. Use it only on an isolated development
> machine; production requires identity keys and network access controls.

```go
import "arkoren.dev/driver/restate"

host := restate.New(func(host *restate.Builder) {
	host.Addr(":9080")
	host.IngressURL("http://localhost:18080")
	host.AdminURL("http://localhost:19070")
	host.AdvertisedURL("http://host.docker.internal:9080")
	host.Register(true)
	host.AllowInsecure(true)
})
```

Bind the runner, wire the host hooks, and read an onboarding result to confirm
registration and execution. `docker compose down --volumes` deletes the disposable
server and history.

See the [`restateworkflow` API reference](https://pkg.go.dev/arkoren.dev/driver/restate/restateworkflow).

---

Source: https://arkoren.dev/docs/guides/workflows/steps.md
[HTML documentation](https://arkoren.dev/docs/guides/workflows/steps)

# Workflow Steps

A workflow decides what happens next; a **step** performs work outside the
engine, such as sending a welcome message. The driver records the step's result
and can supply that result during replay instead of repeating completed work.
Use steps for every external effect in a
[workflow function](https://arkoren.dev/docs/guides/workflows/overview.md#replay-rules).

## Defining Steps

An effect can succeed before its result reaches engine history. Retrying or
recovering the step can therefore repeat that effect. Use a stable business key
with the external service, or another application-owned idempotency mechanism;
recording the step is not an exactly-once guarantee.

For the overview's `User`, supply your application's cancelable, idempotent
`sendWelcome(context.Context, User) (string, error)` function. Declare it with
`arkoren.dev/core/service/workflow`. Policy examples also use the standard
`time` package:

```go
var SendWelcome = workflow.NewStep("send-welcome", sendWelcome)
```

Register it alongside the workflow with `workflow.Handlers(Onboard, SendWelcome)`.
Inside that workflow, call the published definition:

```go
messageID, err := c.Step(SendWelcome, user)
```

The result is a string, matching the step's output type. A failure is wrapped
with the step name. A declaration with a non-nil `Err()` fails without running;
registration catches these errors before workers start.

## Retry Policy

A step runs once by default, with a one-minute timeout for that attempt. Enable
more attempts when failures are transient and repeating the effect is safe:

```go
var SendWelcome = workflow.NewStep("send-welcome", sendWelcome, func(step *workflow.StepBuilder) {
	step.Retry(5)
	step.Backoff(time.Second, 2, time.Minute)
	step.Timeout(30 * time.Second)
})
```

This replaces the baseline declaration rather than registering another step with
the same name. It allows five executions total, with retry delays of one, two,
four, and eight seconds. Timeout bounds each attempt, not all five together;
the body must honor its `context.Context` so cancellation can stop its work.

Without an explicit Backoff, enabled retries start at one second, multiply by
two, and cap at 30 seconds. `Retry` values below one become one. A zero backoff
maximum removes the cap; zero initial delay requests no deliberate positive
delay, not an engine default. Factors below one or nonfinite factors become one.
Negative initial delay, cap, or timeout is a declaration error. `Timeout(0)`
removes the per-attempt limit, not the workflow's duration limit.

Native policies are not interchangeable. Temporal rejects multi-attempt calls
with zero initial backoff before scheduling work; Restate retry delays have
millisecond precision. Check [driver limits](https://arkoren.dev/docs/guides/workflows/drivers.md) before relying on these
settings, and do not bypass a rejection with an application retry loop in the
workflow function.

Return `workflow.Permanent(err)` when another attempt cannot succeed. After
attempts stop, the workflow decides whether to return the error or compensate
for earlier work. [Error Handling](https://arkoren.dev/docs/guides/workflows/error-handling.md) covers that decision and
stable error kinds.

## Local Overrides

When one call needs a different policy, derive a variant rather than mutating
the registered step. For example, a caller can deliberately allow only one attempt:

```go
singleAttempt := SendWelcome.Derive(func(step *workflow.StepBuilder) {
	step.Retry(1)
})
messageID, err := c.Step(singleAttempt, user)
```

Derive keeps the name, typed body, inherited policy, and first declaration
error, then applies edits. It does not reapply defaults or change `SendWelcome`.
The registered handler still supplies the work; the derived definition supplies
this call's policy. Workflows have no Derive method.

## Child Ownership

When a larger process needs another workflow's result, use `Child` only on a
[compatible driver](https://arkoren.dev/docs/guides/workflows/drivers.md#compatibility-matrix). Memory and Temporal support
it; Restate rejects it before submission.

Assume your application has a registered `FulfillOrder` workflow with an `Order`
input and a typed receipt output. Inside a parent workflow, derive its ID from
the order's stable string ID:

```go
receipt, err := c.Child(FulfillOrder, "fulfillment-"+order.ID, order)
```

If the call creates the run, it acquires **cancellation ownership**: canceling
the parent or reaching its `MaxDuration` also stops that child. If the ID already
exists, Child joins the existing run and ignores the new input. Joining never
acquires ownership, even when another parent created the run; ending the parent
ends only its wait, leaving the existing child with its original owner.

Creation versus joining must be atomic and stable across replay. A lookup
followed by a separate start cannot establish ownership. Do not replace an
unsupported Child with an ordinary client Start inside coordination code: that
is external I/O and does not preserve this contract.

## Detached Work

If work must continue independently, use Spawn instead. Given a registered
`SendSurvey` workflow accepting the same order:

```go
id, err := c.Spawn(SendSurvey, "survey-"+order.ID, order)
```

Spawn returns the ID without waiting for the workflow's result. The run outlives
its parent, so this is not a substitute for owned Child cancellation. An existing
ID is treated as success while its record remains retained; use
[a client handle](https://arkoren.dev/docs/guides/workflows/running-workflows.md#reading-results) to inspect the result later.

Both calls reject empty IDs, IDs containing ASCII control characters, and the
parent's own ID for the same workflow name with `workflow.ErrInvalidID`. Stable
business IDs avoid a replay starting a different child, but protection expires
with the run's retained history. On Temporal, Spawn waits for start acknowledgment
within the parent's budget: an error from that wait does not prove no child
started. Reconcile the same ID before replacing work.

For declarations and options, see the
[`workflow` API reference](https://pkg.go.dev/arkoren.dev/core/service/workflow).

---

Source: https://arkoren.dev/docs/guides/workflows/signals-and-waiting.md
[HTML documentation](https://arkoren.dev/docs/guides/workflows/signals-and-waiting)

# Signals and Waiting

An onboarding run can send mail immediately, then wait days for the user to
confirm. A **signal** carries that external confirmation into the run. Durable
drivers retain accepted signals and waits in engine state; the memory driver
keeps them only while its process lives.

These examples use `arkoren.dev/core/service/workflow` and the overview's
[`User` and `SendWelcome`](https://arkoren.dev/docs/guides/workflows/overview.md#defining-work). Client-side sending is
covered by [Signaling Runs](https://arkoren.dev/docs/guides/workflows/running-workflows.md#signaling-runs).

## Declaring Signals

Declare a shared name and payload type for senders and waiters:

```go
var EmailConfirmed = workflow.NewSignal[string]("email-confirmed")
```

The signal carries a string token. A sender and waiter using the same name with
incompatible types is an application error. Names follow the workflow identifier
rule: 1–64 lowercase letters, digits, or inner hyphens. An invalid signal name
panics at declaration, so declare signals during startup rather than from input.

A signal resolves once per run. The first value is remembered: later Wait or
Select calls for that name return it immediately, even if more values were sent.
Signals accepted before the workflow reaches its wait are retained too. This is
not a message stream; use separate runs or another messaging capability when
every event needs processing. Workflow signals are also distinct from the
payload-less signals on the [event bus](https://arkoren.dev/docs/guides/events/overview.md).

## Awaiting Confirmation

Replace the overview's short workflow body with this function when confirmation
belongs to the process. The function uses `errors` and `time` from the standard
library:

```go
func onboard(c *workflow.Context, user User) (string, error) {
	if _, err := c.Step(SendWelcome, user); err != nil {
		return "", err
	}

	token, err := c.Wait(EmailConfirmed, workflow.Timeout(72*time.Hour))
	if errors.Is(err, workflow.ErrTimeout) && c.Err() == nil {
		return "expired", nil
	}

	return token, err
}
```

Publish that function under the same definition shape, retaining a maximum that
exceeds the confirmation wait:

```go
var Onboard = workflow.New("onboard", onboard, func(definition *workflow.Builder) {
	definition.MaxDuration(96 * time.Hour)
})
```

After the step succeeds, a confirmation returns its token. If only the 72-hour
wait expires, the result is `"expired"`. The `c.Err() == nil` check prevents a
whole-run timeout from being mistaken for that ordinary expiry branch: both
use `workflow.ErrTimeout`. Cancellation or a run limit still ends the operation.

No Timeout, zero, or a negative Timeout disables only the per-wait limit. The
run's `MaxDuration` and cancellation remain in force. A remembered signal returns
without scheduling another wait timer.

Keep engine history and signal state for the full interval during which the run
must recover. Finished-run retention controls later inspection and duplicate
protection, not permanent delivery. See [Drivers](https://arkoren.dev/docs/guides/workflows/drivers.md) for provider
retention and queue limits; the memory driver cannot survive process loss.

## Racing Alternatives

If the deadline should trigger another action rather than return a timeout,
select between the confirmation and a timer inside your workflow function:

```go
var (
	token   string
	expired bool
)

err := c.Select(
	workflow.On(EmailConfirmed, func(value string) {
		token = value
	}),
	workflow.After(72*time.Hour, func() {
		expired = true
	}),
)
```

On success, exactly the chosen callback runs. Use `expired` to choose your next
step, or `token` to continue confirmation. Handle `err` before using either;
cancellation, decoding failures, or the whole-run timeout can prevent selection.
Callbacks run synchronously on the coordination path and must follow the same
[replay rules](https://arkoren.dev/docs/guides/workflows/overview.md#replay-rules) as the workflow body. Set local state in
them; perform external effects in subsequent steps.

An already remembered signal wins immediately, in case order if several are
remembered. Otherwise the runtime chooses the signal or timer that fires; do
not infer a portable ordering for simultaneous arrivals. A nonpositive After
delay is immediately eligible, not guaranteed to beat every available signal.
Select without cases returns `workflow.ErrNoCases`; On and After panic for nil
callbacks.

## Pausing Work

When no external message is needed, use a durable sleep instead of `time.Sleep`:

```go
err := c.Sleep(24 * time.Hour)
```

An engine can suspend the run during that pause. Sleep returns early when the run
is canceled or times out. Memory uses process-local timers instead.

Do not block on your own channel or on `<-c.Done()` inside coordination code.
Temporal cannot schedule a native Go channel block and reports a deadlock.
Use Sleep, Wait, or Select to block, and inspect `c.Err()` when you need the run's
cancellation reason. [Error Handling](https://arkoren.dev/docs/guides/workflows/error-handling.md) explains how that reason
differs from a caller abandoning a result wait.

For declarations and options, see the
[`workflow` API reference](https://pkg.go.dev/arkoren.dev/core/service/workflow).

---

Source: https://arkoren.dev/docs/guides/workflows/running-workflows.md
[HTML documentation](https://arkoren.dev/docs/guides/workflows/running-workflows)

# Running Workflows

Use a workflow client to start a process and return to it from later requests or
another service. These examples assume a [configured driver](https://arkoren.dev/docs/guides/workflows/drivers.md), an
operation context `ctx`, and the confirmation-aware
[`Onboard` definition](https://arkoren.dev/docs/guides/workflows/signals-and-waiting.md#awaiting-confirmation).

The facade is safe for concurrent use and does no I/O during construction.
The driver owner remains responsible for startup and cleanup:

```go
import "arkoren.dev/core/service/workflow"

client := workflow.NewClient(driver)
```

## Starting Runs

Choose a stable business identifier so duplicate requests refer to the same
process. For an already validated signup represented by `User`:

```go
run, err := client.Start(ctx, Onboard, "user-42", user)
if err != nil && !errors.Is(err, workflow.ErrAlreadyStarted) {
	return err
}
```

This fragment uses the standard `errors` package. Success returns a typed handle,
not the workflow's result. A duplicate returns that existing handle together
with `ErrAlreadyStarted`; it does not replace the original input or restart the
run. Protection lasts only while the provider retains the execution record.
Temporal IDs are unique across workflow names in a namespace, so include enough
business context to avoid collisions.

An empty ID or one containing ASCII control characters fails with
`workflow.ErrInvalidID` before submission. Invalid workflow declarations and
payload encoding can also fail before driver I/O. A transport failure after
submission can leave the outcome uncertain: the engine may have accepted work.
Reconcile the same logical ID rather than starting a replacement with a new ID.

Starting a run is not atomic with committing your application database. If the
handoff must survive a crash between those operations, persist the intent through
an application-owned outbox or equivalent durable mechanism. A successful Start
also does not make every external effect exactly once; steps still need
[idempotency](https://arkoren.dev/docs/guides/workflows/steps.md#defining-steps).

For a deferred start, add a delay without sleeping in the caller:

```go
run, err := client.Start(ctx, Onboard, "user-42", user,
	workflow.Delay(time.Minute),
	workflow.Metadata(map[string]string{"source": "signup"}))
```

The driver schedules the delay. Metadata copies the outer map and is visible
through provider operational interfaces where supported; keep secrets and
personal data out of it. Native timing, metadata, and retention limits are on
[Drivers](https://arkoren.dev/docs/guides/workflows/drivers.md).

## Reading Results

A later request can recreate a typed handle without a network call:

```go
run := client.Run(Onboard, "user-42")
status, err := run.Status(ctx)
```

The handle itself does not establish existence. Status queries the driver and
returns `workflow.Running`, `Completed`, `Failed`, or `Canceled`; unknown or
expired records return `ErrNotFound`. Running includes a delayed run that has
not executed yet.

To obtain the typed output, wait for the run to finish:

```go
token, err := run.Result(ctx)
```

Result returns the confirmation token, `"expired"` for the workflow's ordinary
confirmation deadline, or the run's failure. The context limits only this wait:
if `ctx` ends, Result returns its context error and the run continues. Ask again
with a fresh context and the same ID. [Error Handling](https://arkoren.dev/docs/guides/workflows/error-handling.md) explains
which error identities survive the worker boundary.

## Signaling Runs

After verifying a confirmation request and authorizing it for the target user,
send the token through the shared signal definition:

```go
err := client.Signal(ctx, Onboard, "user-42", EmailConfirmed, token)
```

This accepts delivery to the identified run; it does not wait for application
processing or completion. Unknown or finished runs return `ErrNotFound`. A signal
accepted before the workflow reaches Wait is retained, and the first value for
that name resolves the run's signal. See
[Signals and Waiting](https://arkoren.dev/docs/guides/workflows/signals-and-waiting.md#declaring-signals) for repeated waits
and retention boundaries.

The workflow name and ID are routing information, not authorization. Keep
untrusted callers away from raw engine submission, and resolve which run they
may signal in application code rather than trusting an arbitrary request ID.

## Canceling Runs

To request that the process stop, cancel the run rather than the context of a
Result call:

```go
err := run.Cancel(ctx)
```

Canceling a known finished run succeeds without replacing its result. An unknown
run returns `ErrNotFound`. A successful call observes cancellation acceptance,
not forced termination of application code or proof that all cleanup completed.
Bodies must cooperate with cancellation, and external effects already performed
are not undone.

Inspect the resulting status or result when the terminal outcome matters.
[Restate pre-execution cancellation](https://arkoren.dev/docs/guides/workflows/drivers.md#cancellation-retention) can lose
the record without its required retention profile. Neither cancel acceptance
nor a missing lookup authorizes reusing a business ID without reconciliation.
Worker shutdown is a separate [host lifecycle](https://arkoren.dev/docs/guides/workflows/shared-worker-host.md) concern.

## Inspecting Definitions

To inspect registered worker policy rather than live runs, query the registry
you constructed at startup:

```go
retried := registry.Steps().Filter(func(_ int, handler workflow.StepHandler) bool {
	return handler.Options.Attempts > 1
})
```

Steps and Workflows return eager `collection.Slice` snapshots sorted by name;
filtering preserves that order. You can group workflows by their run budget:

```go
byDuration := registry.Workflows().GroupBy(func(_ int, handler workflow.WorkflowHandler) time.Duration {
	return handler.Options.MaxDuration
})
```

Each group preserves name order. These records are not execution histories or
live queries. Editing a returned record cannot replace a registered handler or
policy. Functions, their captures, and services remain shared; collection
operations do not create ownership or replay isolation.

## Events and Telemetry

Client events describe successful local calls, not engine transitions. Given
your application `app`, enable them when constructing its client:

```go
client := workflow.NewClient(driver, func(client *workflow.ClientBuilder) {
	client.Events(app.Events())
})
```

Start emits `workflow.run.started`, Signal emits `workflow.run.signaled`, and
Cancel emits `workflow.run.canceled`. Payloads contain the workflow name and run
ID, plus the signal name where relevant; they omit input, output, and signal
values. Failed calls emit nothing. A duplicate Start is silent; a successful
Cancel of a finished run still emits. There is no portable terminal engine
subscription, and polling emits no completed or failed event. See
[Built-in Events](https://arkoren.dev/docs/guides/events/built-in-events.md#workflows).

Configure [telemetry](https://arkoren.dev/docs/guides/observability/telemetry.md) independently in clients and
workers. Given a started service `observations`, replace ordinary registry
construction with worker registration:

```go
registry, err := workflow.NewRegistry([]workflow.Definition{Onboard, SendWelcome},
	func(worker *workflow.RegistryBuilder) {
		worker.Telemetry(observations)
	})
```

Pass the validated registry to the driver or native runner. Configure the client
separately, even if both roles share a process:

```go
client := workflow.NewClient(driver, func(client *workflow.ClientBuilder) {
	client.Telemetry(observations)
	client.Events(app.Events())
})
```

Nil telemetry follows the current process's `telemetry.Default()` on every call
or actual attempt; explicit services remain shared. Export every process to a
common backend: a local event bus or memory capture does not collect observations
from other pods.

Client spans are `workflow start`, `workflow signal`, `workflow cancel`,
`workflow status`, and `workflow result`. Each call records
`arkoren.workflow.operation.count` and `arkoren.workflow.operation.duration` in
seconds, including validation failures. Duplicate starts retain their error and
handle but use the expected `already_started` telemetry outcome. Result duration
measures the caller's wait and decoding, not durable execution time.

Workers observe `workflow step_attempt` only when a step handler actually runs.
Every retry is another attempt; replay, recorded results, suspension, signals,
and sleeps are not attempts. Dependency calls using the step's ordinary context
can become child spans. An attempt records the handler's return, not the engine's
terminal decision; a successful attempt is not proof of successful run completion.
Use engine history for durable run timing and state, and do not add telemetry
side effects to replayable coordination code.

The start span travels in a separate persisted carrier, not in metadata or a Go
context. Actual step attempts and supported child/spawn runs receive it; joining
an existing run preserves its original parent. Signals observe submission and
do not replace that parent. Retries and new child attempts retain the original
start parent rather than a fabricated orchestration span. Check
[driver propagation and rollout](https://arkoren.dev/docs/guides/workflows/drivers.md#compatibility-matrix) before enabling
traced producers. Carriers enter engine inputs or headers: allow only trusted
submitters and keep sensitive baggage out of your propagator.

Automatic telemetry omits payloads, values, outputs, metadata, run IDs, and raw
error messages. Metric labels use only operation, outcome, and classified error
type. Client-call panics still propagate unchanged; a started observation closes
with `workflow.panic`, not success or the raw panic value. Application logging
through `c.Logger()` needs its own redaction policy.

Native SDK operations bypass these observations and local client events. Custom
native scheduling outside the adapter must explicitly carry propagation.

For declarations and options, see the
[`workflow` API reference](https://pkg.go.dev/arkoren.dev/core/service/workflow).

---

Source: https://arkoren.dev/docs/guides/workflows/shared-worker-host.md
[HTML documentation](https://arkoren.dev/docs/guides/workflows/shared-worker-host)

# Worker Hosts

When one application uses an engine for workflows and scheduled tasks, a **host**
owns their shared connection and execution endpoint. Pass one Temporal or Restate
host to both drivers, then wire its lifecycle once. Memory drivers have no shared
host; Hatchet is pending and unpublished.

This page owns lifecycle for both [workflow drivers](https://arkoren.dev/docs/guides/workflows/drivers.md) and
[scheduler drivers](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md). Driver-specific registration
belongs on those pages. Examples assume an existing application composer and a
validated workflow registry.

## Registration Order

Construct the host, register or bind all work, then start the host. Construction
does no network I/O. A workflow `NewRunner` and a scheduler constructor add work
to the host; a workflow `NewDriver` only controls runs. Late registration returns
the vendor's `ErrStarted`.

Inside a composer, register cleanup at the acquiring owner, before later
construction can fail. Use `composer.Must` for fallible construction there;
startup callbacks complete before application runners begin. See
[Lifecycle and Ownership](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md).

Builder callbacks run synchronously once, in order, on one fresh draft. Nil
entries panic; expanded nil or empty slices supply no edits. Publication detaches
owned settings and containers from the retained draft, not native SDK internals,
hooks, clients, or captured services. Finish configuring shared native objects
before publishing them.

## Temporal

The `arkoren.dev/driver/temporal` host shares a client, task queue, and worker.
Within your composer, create and register it before either driver:

```go
import "arkoren.dev/driver/temporal"

host := temporal.New(func(host *temporal.Builder) {
	host.HostPort("temporal:7233")
	host.TaskQueue("app")
})
composer.Cleanup(host.Stop)
```

Register workflow and schedule work on that same host. Here `registry` already
contains your workflows and steps:

```go
import (
	"arkoren.dev/driver/temporal/temporalschedule"
	"arkoren.dev/driver/temporal/temporalworkflow"
)

runner := composer.Must(temporalworkflow.NewRunner(host, registry))
schedules := composer.Must(temporalschedule.New(host))
```

The runner's Driver supplies the workflow client; `schedules` supplies the
scheduler driver. Neither starts a worker. Add host lifecycle and readiness hooks:

```go
composer.Startup(host.Start)
composer.Check("temporal", host.Ping)
```

Start dials, applies registrations, and starts the worker. Ping checks native API
reachability, not worker liveness or scheduler dispatch readiness. A wrong address
fails startup rather than waiting for the first client request. The scheduler's
own run lifecycle is separate and uses the already-started host.

### Connection Policy

Defaults are `HostPort("localhost:7233")`, namespace `default`, and task queue
`arkoren`. Configure TLS, credentials, interceptors, and other native settings
through `ClientOptions` and `WorkerOptions`. When dialing, host address and
namespace override matching native fields, including when using host defaults.
The host adapts `slog.Default()` when no logger is supplied; an explicit native
client logger takes precedence. SDK option records are value copies, not
recursive snapshots.

If you already own a native client, wrap it instead:

```go
host := temporal.NewFrom(client, func(host *temporal.Builder) {
	host.TaskQueue("app")
})
```

Stop leaves this borrowed client open. Only TaskQueue, WorkerOptions,
WorkerStopTimeout, and NoWorker apply; host logger and connection settings do
not. Configure the borrowed client's logger before wrapping it: the worker
inherits that logger. `host.Client()` is available immediately for NewFrom,
but is nil before Start for a dialing host. A nil borrowed client fails startup.

### Shutdown Boundaries

Serialize Start and Stop; call Stop after Start returns. An already-canceled
startup context acquires nothing. Native worker startup has no context parameter,
so cancellation during that call is observed only after it returns, followed by
resource cleanup. Registration callbacks and SDK hooks must return. Registration
panics propagate after owned-client cleanup.

`WorkerStopTimeout` defaults to ten seconds; zero restores the default and a
negative host value disables grace. A nonzero native
`WorkerOptions.WorkerStopTimeout` wins. Native grace can apply separately to
internal workers before activity contexts are canceled: it is not a total
shutdown deadline and cannot terminate arbitrary callbacks.

Stop starts worker cleanup even with an expired context, then waits only until
its context ends. Cleanup continues afterward. The owned client closes only
after worker cleanup returns; a context error does not prove cleanup finished.
Repeated Stop calls wait on the same cleanup using their own contexts. Borrowed
clients remain yours to close.

See the [`temporal` host API](https://pkg.go.dev/arkoren.dev/driver/temporal).

## Restate

Restate reaches an endpoint in your application rather than polling a worker.
The host therefore has three lifecycle operations: Start opens a listener, Serve
handles requests, and Stop releases it.

Configure ingress/admin URLs that your process can reach, and an advertised URL
the Restate server can reach. Before constructing the host, obtain the server's
request-identity public keys (`publickeyv1_...`):

```go
import "arkoren.dev/driver/restate"

host := restate.New(func(host *restate.Builder) {
	host.Addr(":9080")
	host.IngressURL("http://restate:8080")
	host.AdminURL("http://restate:9070")
	host.AdvertisedURL("http://app:9080")
	host.IdentityKeys(identityKeys...)
	host.Register(true)
})
composer.Cleanup(host.Stop)
```

Here `identityKeys` is your configured `[]string` of server public keys. A serving
host without keys fails Start with `restate.ErrNoIdentityKeys`.
`AllowInsecure(true)` skips verification and permits anyone who can reach the
listener to invoke handlers; use it only for
[isolated local development](https://arkoren.dev/docs/guides/workflows/drivers.md#local-engine-1), never as a production
startup repair. The endpoint supports HTTP/1.1 and unencrypted HTTP/2; establish
production network access controls and transport protection separately.

Bind workflow and scheduler services to this host before startup, using their
constructors from the respective Drivers pages. Then add all three hooks:

```go
composer.Startup(host.Start)
composer.Run(host.Serve)
composer.Check("restate", host.Ping)
```

Start builds the endpoint and opens the listener without serving. Serve begins
request handling, optionally registers the deployment, and blocks until the
application stops. Stop also owns an opened listener if Serve never runs, such
as during startup rollback. Canceling Serve or a registration failure does not
replace cleanup.

Ping checks the admin API's health, not ingress reachability, registration,
successful workflow delivery, or persisted activation of the
[preflight-retention profile](https://arkoren.dev/docs/guides/workflows/drivers.md#cancellation-retention). Verify an actual
run when delivery readiness matters. The default advertised URL is
`http://localhost` plus the listen address; that only works when the server can
reach the application there.

### Deployment Policy

`Register(true)` asks Serve to register a deployment; otherwise operators must
register it. Each rollout creates a new deployment. `RegisterForce` overwrites
a deployment at the same URL and is appropriate only when none of the old work
is in flight. `RegisterBreaking` permits registration that removes services.
Neither option makes incompatible journals replay-safe. Use `AdminAuthKey` for
a protected admin API and `AuthKey` for protected ingress.

Drain workflow runs that used the earlier Child implementation on their original
deployment before upgrading to its rejection; there is no automatic history
migration. See [workflow rollout limits](https://arkoren.dev/docs/guides/workflows/drivers.md#run-lifetimes-1).

When the same host serves schedules, preserve retained generations and finish
in-flight Remove invocations on the old deployment before replacing an
incompatible journal. If an older deployment removed an entry, drain its pending
ticks before reusing its name. The
[Restate scheduler guidance](https://arkoren.dev/docs/guides/scheduled-tasks/drivers.md#restate) owns those
schedule-specific constraints.

### Shutdown Boundaries

Stop waits for active HTTP requests until its context expires. It then closes
remaining connections and returns the shutdown error. Closing connections cancels
request contexts, not arbitrary Go code that ignores cancellation. The shared
HTTP client remains caller-owned and is not closed by endpoint cleanup.

`host.Query` runs admin SQL; restrict its credentials and never accept arbitrary
queries from untrusted callers. See the
[`restate` host API](https://pkg.go.dev/arkoren.dev/driver/restate).

## Client-Only Hosts

A web process can start work without serving it. For Temporal, set
`temporal.Builder.NoWorker(true)` and construct `temporalworkflow.NewDriver`.
For Restate, set `restate.Builder.ClientOnly(true)` and construct
`restateworkflow.NewDriver`. These host settings skip the worker or listener;
native connection and authentication still belong to the host.

Use the workflow driver's [client-only setup](https://arkoren.dev/docs/guides/workflows/drivers.md), not NewRunner.
Temporal rejects a runner on a NoWorker host. Scheduler drivers register work,
so construct those in the serving process instead. Match codec, prefix, and
native routing across producers and workers.

A host with no registered work also behaves client-only. NoWorker defaults false;
setting it false later restores permission but never forces a worker without
registrations. Temporal's WorkerDisabled reports the stored flag, not liveness.

Continue with [Running Workflows](https://arkoren.dev/docs/guides/workflows/running-workflows.md) once the host is started,
or return to [Scheduled Tasks](https://arkoren.dev/docs/guides/scheduled-tasks/overview.md) for scheduler use.

---

Source: https://arkoren.dev/docs/guides/workflows/error-handling.md
[HTML documentation](https://arkoren.dev/docs/guides/workflows/error-handling)

# Error Handling

A temporary gateway failure can justify another step attempt; a permanently
declined payment cannot. The step reports that distinction. After attempts end,
the workflow chooses whether to compensate, return a different result, or fail.
Returning a workflow failure does not retry the whole run. Engine replay rebuilds
coordination state; it is not a retry policy for failed workflows.

Examples use `arkoren.dev/core/service/workflow`, standard `context` and `errors`,
and application-owned `Order`, `Receipt`, and gateway services. Configure attempt
counts and delays on [Workflow Steps](https://arkoren.dev/docs/guides/workflows/steps.md#retry-policy); native limits belong
on [Drivers](https://arkoren.dev/docs/guides/workflows/drivers.md).

## Stopping Retries

Use a stable application error for a condition callers must recognize. For
example, classify a gateway's permanent decline inside the step body:

```go
var ErrCardDeclined = workflow.NewError("billing.card-declined", "card declined")

func charge(ctx context.Context, order Order) (Receipt, error) {
	receipt, err := gateway.Charge(ctx, order)
	if errors.Is(err, gateway.ErrDeclined) {
		return Receipt{}, workflow.Permanent(ErrCardDeclined)
	}

	return receipt, err
}
```

The gateway and its ErrDeclined are your application's provider adapter.
Permanent stops further attempts even when the configured budget has attempts
left. Other failures remain eligible for the step's retry policy. A single
attempt is the default; a timed-out attempt can retry like another failure.
`errors.Is(err, workflow.ErrPermanent)` recognizes the permanent marker, and
`workflow.Permanent(nil)` remains nil.

## Compensating Work

A successful reservation is not undone when a later charge fails. Given
registered Reserve, Charge, and Release steps accepting an Order, explicitly
attempt release before returning the charge failure:

```go
func checkout(c *workflow.Context, order Order) (Receipt, error) {
	if _, err := c.Step(Reserve, order); err != nil {
		return Receipt{}, err
	}

	receipt, err := c.Step(Charge, order)
	if err != nil {
		if _, releaseErr := c.Step(Release, order); releaseErr != nil {
			return Receipt{}, errors.Join(err, releaseErr)
		}

		return Receipt{}, err
	}

	return receipt, nil
}
```

Publish this function with `workflow.New` and a suitable positive MaxDuration,
then register the workflow and all three steps. Release is itself recorded and
retried according to its policy. Compensation is another fallible effect, not a
database rollback or a guarantee that cancellation will leave time to undo work.
If the run's context has ended, later runtime operations can fail immediately.
Provide application-owned reconciliation for effects that remain unresolved.

The step failure reaching coordination code has already crossed the driver's
error representation. Match its stable kind rather than relying on a Go type
that exists only inside the step. Joining errors preserves local reporting;
do not expect every arbitrary Go cause to survive a wire round-trip.

## Matching Failures

Callers obtain a run's terminal failure from Result and use `errors.Is`:

```go
_, err := run.Result(ctx)
if errors.Is(err, ErrCardDeclined) {
	// Ask the customer for another card.
}
```

`workflow.Error` values have stable kinds. Kind, message, and permanent marker
survive encoding; arbitrary wrapped types and values do not. A worker-specific
`errors.As` target therefore cannot recover its original type in another
process. Nonempty equal kinds match; kinds beginning `workflow.` are reserved.

Distinguish errors from client commands from errors that end a run:

| Error | Meaning |
| --- | --- |
| `workflow.ErrAlreadyStarted` | Start found an existing run and also returned its handle; the existing run did not fail. |
| `workflow.ErrNotFound` | A workflow/run is unknown or forgotten; signaling a finished run also returns this error. |
| `workflow.ErrTimeout` | A step attempt, signal wait, or whole-run duration limit expired. |
| `workflow.ErrCanceled` | A run or its blocking operation was canceled. |
| `workflow.ErrUnsupported` | The driver cannot perform the operation or policy; retrying the same request does not add support. |

Provider lookup/transport errors remain errors rather than proof that work never
started. Reconcile uncertain outcomes by the same ID as described in
[Starting Runs](https://arkoren.dev/docs/guides/workflows/running-workflows.md#starting-runs).

## Timeout Boundaries

Three clocks answer different questions. A step Timeout limits one attempt.
`workflow.Timeout` limits one signal wait. MaxDuration limits the workflow run.
Each can produce `workflow.ErrTimeout`, but the workflow can handle an individual
step or wait failure without returning a failed run.

When branching on an ordinary wait expiry, also check that the run itself is
still alive:

```go
token, err := c.Wait(EmailConfirmed, workflow.Timeout(72*time.Hour))
if errors.Is(err, workflow.ErrTimeout) && c.Err() == nil {
	return "expired", nil
}
```

`c.Err()` is nil for a live run and reports `ErrCanceled` or `ErrTimeout` once
the run ends. Done closes when that state is observed. Never block on the native
Go channel inside a workflow function; Temporal reports a deadlock. Use durable
Wait, Sleep, or Select for blocking.

Separately, `run.Result(ctx)` returns a context error if the caller stops waiting
first. That does not cancel the run; read its result again with a fresh context.
Cancellation is cooperative, and native completion/retention boundaries remain
[provider-specific](https://arkoren.dev/docs/guides/workflows/drivers.md#compatibility-matrix).

## Panic Boundaries

A panic in a step body becomes a permanent failure rather than repeating the
same bug through retries. A workflow-body panic fails the run with a message
beginning `panic:`. Its stack goes to the logger, not the returned error. Fix the
bug; do not use panic as a retry mechanism.

Engine control-flow panics, including Restate suspension and abort, pass through
the adapter so native replay keeps working. Recovery of application bodies does
not promise recovery of every native SDK goroutine. Constructor callbacks and
client-call panics propagate; client telemetry records the bounded panic class
without changing the panic value.

## Safe Messages

> [!WARNING]
> Errors returned by steps or workflows can enter engine history, backups, and
> Result responses in other processes. Do not include secrets, personal data,
> or raw sensitive provider messages. Return a stable safe message instead.

For an unexpected provider failure, separate classification from private
diagnostics:

```go
if err != nil {
	logger.ErrorContext(ctx, "charge failed", "order", order.ID, "error", err)
	return Receipt{}, workflow.NewError("billing.charge-failed", "charge failed")
}
```

The application logger needs its own access controls and redaction; even the
order identifier or provider error can be sensitive. This generic error does not
misclassify an outage as a declined card. Mark it permanent only if you know
another attempt cannot succeed. Panic messages can also be persisted, so never
panic with sensitive data.

## Registration Errors

Validate definitions at startup with `workflow.Handlers` before using the registry.
Definitions keep their first declaration error. Registration reports
`ErrInvalidName`, `ErrNilFunc` (including typed-nil definitions),
`ErrInvalidOptions`, `ErrMissingMaxDuration`, or `ErrDuplicateName` as appropriate.
Negative definition durations are invalid, and MaxDuration must be positive.
An invalid signal name instead panics when NewSignal is called.

Runtime ID validation rejects empty IDs and ASCII control characters; Child and
Spawn also reject starting the current run under its own workflow name. These
fail with `ErrInvalidID`, not a backend submission error. `c.Step` rejects a
definition whose own Err is already set without executing it.

For declarations and options, see the
[`workflow` API reference](https://pkg.go.dev/arkoren.dev/core/service/workflow).

---

Source: https://arkoren.dev/docs/guides/workflows/testing.md
[HTML documentation](https://arkoren.dev/docs/guides/workflows/testing)

# Testing Workflows

Test the request to start work separately from the work itself. A signup handler
test should prove that it submits onboarding for the right user, without sending
mail. A workflow test should exercise the process's decisions. Replay and native
retention need an engine test: neither a fake nor memory executes durable history.

The following are fragments inside an existing `Test...` function with
`t *testing.T`. They use `arkoren.dev/core/service/workflow` and the
[onboarding definitions](https://arkoren.dev/docs/guides/workflows/overview.md#defining-work). Use the
[confirmation-aware body](https://arkoren.dev/docs/guides/workflows/signals-and-waiting.md#awaiting-confirmation) for timeout
and signal tests.

## Submission Tests

Create a fake driver and pass its ordinary client to the application code under
test:

```go
import "arkoren.dev/core/service/workflow/workflowtest"

fake := workflowtest.New()
client := workflow.NewClient(fake)
```

Here `signup` is your application function: it accepts a client, persists or
validates signup according to your application, and starts Onboard. Exercise it,
then assert the typed payload:

```go
if err := signup(t.Context(), client, User{ID: 42}); err != nil {
	t.Fatal(err)
}

fake.AssertStarted(t, Onboard, func(user User) bool {
	return user.ID == 42
})
```

The assertion proves submission of that input, not that onboarding executed or
the signup/database handoff was atomic. The fake never calls workflow or step
bodies. Every workflow name is considered registered, so validate your real
registry in separate tests. Runs stay Running until you call Complete, Fail,
or Cancel.

Use the fake's assertions for no starts, start counts, signals, and cancellation
when those are the behavior under test. Failed driver calls are not recorded.

## Scripted Results

If application code waits for a result, script the outcome before calling it:

```go
fake.Complete(Onboard.Name(), "user-42", "confirmed")
```

Completing a run before Start stores a script; the later start returns a finished
run. Completing an already-started run releases its waiters. A repeated Start
still returns `workflow.ErrAlreadyStarted` with the existing handle.

To exercise an application error branch, script a stable error kind:

```go
var ErrInvitationRevoked = workflow.NewError("onboarding.revoked", "invitation revoked")

fake.Fail(Onboard.Name(), "user-42", ErrInvitationRevoked)
```

Use a fresh fake or a separate run ID for this alternative. Fail uses the same
wire representation as real workflow failures, so arbitrary worker-side Go types
do not survive. Application code should match the kind with `errors.Is`.

`fake.FailNext(err)` fails the next driver operation; `fake.FailWith(err)` fails
every subsequent operation until cleared with nil. These model command/lookup
failures, not a failure returned by the workflow body. Distinguish the two in
your assertions.

## Workflow Decisions

Use memory to execute the real workflow function, replacing its external
dependencies with fakes. For onboarding, define SendWelcome against a fake mail
sender so this test does not send real mail. Its handler should return promptly
and honor cancellation.

Inside your test, `synctest` advances the process-local clock when all work is
blocked. The three-day confirmation deadline needs no wall-clock wait:

```go
import (
	"context"
	"testing"
	"testing/synctest"
	"time"

	"arkoren.dev/core/service/workflow/memoryworkflow"
)

synctest.Test(t, func(t *testing.T) {
	registry, err := workflow.Handlers(Onboard, SendWelcome)
	if err != nil {
		t.Fatal(err)
	}

	driver := memoryworkflow.New(registry)
	t.Cleanup(func() {
		cleanup, cancel := context.WithTimeout(context.Background(), time.Second)
		defer cancel()
		if err := driver.Stop(cleanup); err != nil {
			t.Error(err)
		}
	})
	client := workflow.NewClient(driver)

	run, err := client.Start(t.Context(), Onboard, "user-42", User{ID: 42})
	if err != nil {
		t.Fatal(err)
	}

	out, err := run.Result(t.Context())
	if err != nil || out != "expired" {
		t.Fatalf("Result = %q, %v; want expired", out, err)
	}
})
```

No confirmation is sent, so the wait expires and the workflow returns
`"expired"`. To test confirmation instead, send EmailConfirmed after Start and
before Result, then assert that token. Add retry tests only for decisions that
belong to your workflow or step policy, with explicit counts on the fake service.
See [Controlling Time](https://arkoren.dev/docs/guides/testing/time.md).

Cleanup deadlines bound the Stop caller's wait, not the lifetime of code that
ignores cancellation. Workflow bodies and fake services must cooperate, and
application-owned background work must be joined. A synctest bubble also cannot
finish while its goroutines remain blocked indefinitely.

## Engine Regressions

Memory never replays the workflow function, so a passing memory test cannot
establish determinism. Review coordination code for clock, random, I/O, goroutine,
and native-channel use, then test representative retained histories against the
actual engine when deploying changes.

Match integration tests to the deployed server version and feature profile.
Native retry precision/caps, delayed or queued cancellation retention, endpoint
registration, startup rollback, and shutdown cleanup are distinct boundaries.
A healthy native API proves none of them. The [driver comparison](https://arkoren.dev/docs/guides/workflows/drivers.md)
identifies Temporal policy rejections and Restate's operator-managed retention
profile.

For joined children, wait until the engine acknowledges the join before ending
the parent. Seeing the parent start does not prove it joined. Verify that an
existing run finishes with its original input and without duplicate work, while
a newly created child stops with its owner. Unsupported Child must reject before
submission and leave independent runs untouched.

## Driver Conformance

If your application supplies a custom driver, use `workflowtest.Run` to exercise
the portable contract. Build the suite's registry inside the factory, not from
your onboarding definitions:

```go
workflowtest.Run(t, func(t *testing.T) workflow.Driver {
	driver := memoryworkflow.New(workflowtest.Registry(t))
	t.Cleanup(func() {
		cleanup, cancel := context.WithTimeout(context.Background(), time.Second)
		defer cancel()
		if err := driver.Stop(cleanup); err != nil {
			t.Error(err)
		}
	})
	return driver
}, workflowtest.WithSynctest())
```

WithSynctest creates a driver inside each separate bubble and runs cases
sequentially; cleanup registered on the supplied test runs before the bubble
ends. Never use it for an external engine.

Without that option, the factory runs once with the parent test, and parallel
subtests share the driver and its dependencies. All shared fixtures must support
concurrent use. Register cleanup on the factory's `t` so it runs after those
subtests; a deferred stop in the parent function runs too early.

Declare unsupported operations with `WithoutChild`, `WithoutSpawn`, or
`WithoutImmediateStepRetries` when applicable. These options assert rejection
before submission or affected-handler execution rather than silently skipping
the contract. Keep every suite definition registered. Conformance complements,
not replaces, deployed-engine replay and regression tests. See
[Driver Conformance Suites](https://arkoren.dev/docs/guides/testing/conformance-suites.md) and
[Integration Tests](https://arkoren.dev/docs/guides/testing/integration-tests.md).

For fake assertions and suite options, see the
[`workflowtest` API reference](https://pkg.go.dev/arkoren.dev/core/service/workflow/workflowtest).

---

Source: https://arkoren.dev/docs/guides/llm/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/overview)

# Language Models Overview

A language model receives a conversation and produces its next message. Use
`*llm.Model` to send those requests through a driver with shared defaults,
validation, events, and telemetry. Configure a hosted or local connection in
[Drivers](https://arkoren.dev/docs/guides/llm/drivers.md); the examples here assume a configured `driver`, a model
ID in `modelID`, and an operation context `ctx`.

The model does not store a conversation or execute requested tools by itself.
Your application owns [conversation history](https://arkoren.dev/docs/guides/llm/conversation-history.md). An
[agent](https://arkoren.dev/docs/guides/llm/running-agents.md) adds the repeated turns needed to execute tools and
validate a final answer. [Embeddings](https://arkoren.dev/docs/guides/llm/embeddings.md) use a separate facade for
vectors rather than generated replies.

Focused examples show error results but omit routine propagation. Handle errors
before using results in your application. Import `llm` from
`arkoren.dev/core/service/llm`; the snippets also use standard `fmt` and `errors`.

## Generating Replies

To summarize a maintenance notice, create a reusable model and send one user
message. Use the incoming request context in an HTTP handler, or a bounded
command context, so cancellation reaches the provider.

```go
model := llm.New(driver, modelID)
reply, err := model.Generate(ctx, llm.Request{
	Instructions: "Summarize the notice in one sentence.",
	Messages: []llm.Message{
		llm.UserText("Search will be unavailable Sunday from 02:00 to 02:15 UTC."),
	},
})
```

`Generate` performs one turn without facade retries. The reply's message can
contain text, reasoning, or tool requests; it is not necessarily a final answer.
To display its text parts:

```go
for _, part := range reply.Message.Parts {
	if text, ok := part.(llm.Text); ok {
		fmt.Println(text.Text)
	}
}
```

The wording depends on the model. A requested tool is data in this response,
not an executed application action. Use [Running Agents](https://arkoren.dev/docs/guides/llm/running-agents.md) when
Arkoren should manage that loop, or [Streaming Replies](https://arkoren.dev/docs/guides/llm/streaming.md) to display
fragments as they arrive.

## Reading Outcomes

Check `reply.StopReason` before treating the reply as complete:

| Reason | Meaning |
| --- | --- |
| `llm.StopEndTurn` | The model ended its turn. |
| `llm.StopToolUse` | The model requested tool execution. |
| `llm.StopMaxTokens` | The output limit stopped the reply. |
| `llm.StopRefusal` | The model refused. |
| `llm.StopContentFilter` | A content filter stopped the reply. |

`Generate` returns truncated and refused replies as successful response data.
Agent runs instead return `llm.ErrTruncated` or `llm.ErrRefused` for those
outcomes, before tool execution or final-output decoding.

## Setting Defaults

If all calls need an output cap, publish it with the model:

```go
model := llm.New(driver, modelID, func(model *llm.ModelBuilder) {
	model.MaxTokens(1024)
})
```

A request's positive `MaxTokens` overrides that fallback; zero leaves it to the
facade default, then the provider. This cap applies to each turn, not the total
tokens of an agent run. Reaching it can leave incomplete output.
`Request.Model` similarly overrides the facade's model ID for one request.

Construction does no I/O. An empty model ID or nil driver is reported when you
call the facade. Share the model when its driver and shared callbacks support
concurrent use; the acquiring owner manages the driver's lifetime separately.
Construction callbacks publish copied settings, not a reconfigurable facade.

## Handling Failures

Drivers preserve provider errors while adding portable sentinels. Match them
with `errors.Is`, rather than parsing a diagnostic:

```go
switch {
case errors.Is(err, llm.ErrContextLength):
	// Trim the conversation before another attempt.
case errors.Is(err, llm.ErrRateLimited), errors.Is(err, llm.ErrUnavailable):
	// Apply your application's retry or fallback policy.
case errors.Is(err, llm.ErrRefused):
	// Return an application-approved refusal response.
}
```

The facade does not retry provider failures. The OpenAI driver's SDK retries
also default to off. Decide whether another attempt is safe, especially after
an agent has performed tool effects; cancellation cannot undo those effects.

## Observing Calls

`Response.Usage` reports input, output, cache, and reasoning tokens. Counts the
provider omits stay zero. `Result.Usage` sums the turns of an agent run.

Set `ModelBuilder.Events(bus)` to publish standalone response outcomes and agent
or tool outcomes. Agent turns do not also publish standalone response events.
An embedder has its own event setting. See [built-in LLM events](https://arkoren.dev/docs/guides/events/built-in-events.md#language-models)
for payloads and topics, and [stream outcomes](https://arkoren.dev/docs/guides/llm/streaming.md#stopping-streams)
for consumer stops.

Telemetry creates `chat {model}` spans, with `invoke_agent` and `execute_tool`
spans for agent execution. Prompts, replies, tool arguments, and tool results
are absent from telemetry and event payloads. Failure events carry bounded
`ErrorType` values rather than raw errors: a sentinel name, `context.Canceled`,
`context.DeadlineExceeded`, or `unknown`. Context classifications take precedence;
execution-policy failures otherwise use `llm.ErrToolAvailability`, not their
`ErrPermanent` wrapper.

A panic still propagates. An observation it unwinds ends with telemetry class
`llm.interrupted`, without panic data or an invented outcome event. Previously
completed observations remain published; this is not panic recovery.

Returned errors and saved history can still contain provider, tool, policy, or
schema diagnostics. Review them before logging or returning them to a client.
Bounded event classifications do not sanitize application-owned data.

For all functions and options, see the [llm API reference](https://pkg.go.dev/arkoren.dev/core/service/llm).

---

Source: https://arkoren.dev/docs/guides/llm/drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/drivers)

# Drivers

Use the OpenAI driver for hosted OpenAI models or a server with a compatible
API. Choose the API mode and model together: a server can accept text requests
without supporting your tools, schemas, or media inputs. The matrix below shows
what Arkoren translates in each mode, so you can check those requirements before
connecting.

The driver handles the vendor's wire protocol and maps its errors onto the
contract's sentinels. Defaults, request validation, agents, telemetry, and events
belong to the facade; your calling code uses the same contract types.

There is one vendor driver today, OpenAI. Compatible servers use that driver,
subject to their API and model limitations. There is no memory driver;
tests use the scripted `llmtest.Fake` ([Testing Language Models](https://arkoren.dev/docs/guides/llm/testing.md)).
[Choosing a Driver](https://arkoren.dev/docs/guides/drivers/choosing-a-driver.md) compares drivers across
services.

## Compatibility Matrix

Compare the two API modes of the shipped OpenAI driver. These are translation
capabilities exposed by Arkoren, not guarantees that every model or compatible
server accepts every request. Ollama, vLLM, and gateways use this same driver;
check their selected API and model rather than assuming OpenAI feature parity.
The scripted fake is for application tests, not a separate provider protocol.

| Driver and mode | Streaming | Tool calls | Structured output | Image input | Document input | Reasoning history | Embeddings |
| --- | --- | --- | --- | --- | --- | --- | --- |
| [OpenAI: Responses](#openai) | Yes | Yes | Yes | Yes | Yes | Partial — Only reasoning items with a signature round-trip. | Yes |
| [OpenAI: Chat Completions](#openai) | Yes | Yes | Yes | Yes | No | No | Yes |

Responses is the default mode. Chat Completions rejects document parts with
`llm.ErrUnsupported` and drops reasoning history, though it still sends the
requested reasoning effort. Responses restores reasoning items only when their
signature contains the item ID and encrypted content. Structured output uses a
JSON Schema response format with no prompt-based fallback if the server rejects
it. Embeddings use the separate embeddings endpoint and model in either mode.
Select a model and server that support your required inputs, tools, and schemas.

## OpenAI

The `openaillm` driver speaks to OpenAI through its Responses API, and through
Chat Completions for servers that only implement that. Choose it for OpenAI's
models, or for a compatible server such as Ollama, vLLM, or a gateway. It
handles text, images, PDFs, tool calls, reasoning, structured output, and
embeddings.

### Installation

Install the OpenAI driver from your application's module directory:

```sh
go get arkoren.dev/core arkoren.dev/driver/openai
```

### Connecting

For hosted OpenAI, obtain an API key and choose a model your account can access.
Set `OPENAI_API_KEY` outside source control. Requests can incur provider charges;
this guide does not require making a live call. With your selected ID in
`modelID`, construct the connection and facade:

```go
import (
	"arkoren.dev/core/service/llm"
	"arkoren.dev/driver/openai/openaillm"
)

driver, err := openaillm.New() // OPENAI_API_KEY, OPENAI_BASE_URL from the environment

model := llm.New(driver, modelID)
```

Handle the construction error before using `driver`. Set `OPENAI_API_KEY`
before calling the hosted service and choose a model your
account can access. `New` does no I/O; connection and credential failures appear
on the first request. Follow the [overview](https://arkoren.dev/docs/guides/llm/overview.md#generating-replies) to get
your first answer. Register `Ping` as a
[health check](https://arkoren.dev/docs/guides/application/health-and-readiness.md) if the server supports listing models.

### Configuration

If your organization routes calls through an OpenAI-compatible gateway, keep
the same calling code and change the connection. Here `key` is the gateway
credential loaded by your application:

```go
driver, err := openaillm.New(func(driver *openaillm.Builder) {
	driver.APIKey(key)
	driver.BaseURL("https://llm-gateway.internal/v1")
})
```

You can also set `OPENAI_BASE_URL` without changing source. Keep SDK retries off
unless you deliberately want retry policy in the connection rather than your
application. They default to zero additional attempts; `MaxRetries(n)` enables
up to `n` SDK retries. Use a request context with a deadline to bound the whole
call. If you supply `HTTPClient`, its timeout and transport remain your
responsibility; `NewFrom` retains the native client's own retry policy.
`New` reads `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `OPENAI_ORG_ID`, and
`OPENAI_PROJECT_ID` through the SDK unless explicitly configured. Servers
without authentication need no API key. Empty API selects Responses; empty
strict mode selects its mode-specific default. These empty values are not
equal to the named constants. Unknown API modes and negative retry or embedding
batch-size settings fail construction.

Embeddings use batches of at most 2048 inputs by default; zero restores that
default. Results are ordered by the response index and input-token usage is
summed across batches. `Purpose` is ignored by this driver. Use
[request hooks](#request-hooks) for native parameters
or [reuse an SDK client](#native-clients) for existing connection policy.

### Generating Replies

With the `model` above, follow [generating replies](https://arkoren.dev/docs/guides/llm/overview.md#generating-replies)
to receive text or [streaming text](https://arkoren.dev/docs/guides/llm/streaming.md#streaming-text) to
display it as it arrives. For vectors, pass the same `driver` to the
[embedding setup](https://arkoren.dev/docs/guides/llm/embeddings.md#embedding-text); the embedding model is
separate from the chat model. No driver `Start` or `Stop` call is required.

### Mode Limits

Use the default Responses mode when the server supports it. If it only offers
Chat Completions, add `driver.API(openaillm.ChatCompletions)` to the construction
callback. Documents and reasoning do not have the same support. The
[Ollama workflow](https://arkoren.dev/docs/guides/llm/local-development-with-ollama.md) shows local setup and
mode selection.
`Ping` needs a working `GET /models` endpoint and reads one page. A server without
that endpoint fails readiness even if generation works.

By default, Responses initializes `store: false` and
`include: ["reasoning.encrypted_content"]` before the native hook. These are
request defaults, not enforced storage restrictions: a hook can override either
field, and an explicit `Store = openai.Bool(true)` reaches both generation and
streaming requests. Neither mode guarantees that the server avoids logging or
retaining data. Review your hooks, gateway configuration, and provider policy
before sending private prompts or tool results.

A compatible server must accept the parameters your hook leaves in the request.
Reasoning signatures retain `<item id>:<encrypted content>` for replay with your
saved conversation. Reasoning without that signature is skipped. Document parts
become `input_file` data URLs.
Tool calls use `call_id`; tool-result error flags have no provider field, so
their content is sent unchanged.

Chat Completions sends single text parts as plain strings for older servers and
requests stream usage. It rejects documents with `llm.ErrUnsupported`, drops
reasoning history/output, and sends only reasoning effort. For conformance tests
in this mode, skip those checks with `llmtest.WithoutDocuments()` and
`llmtest.WithoutReasoning()`.

In both modes, maximum tokens map to the corresponding provider output limit.
Reasoning Off sends `none`, Low/Medium/High use their names, and Default omits the
setting. Input-token usage includes cached tokens; output usage includes
reasoning tokens. Unreported counts remain zero. Stop reasons remain successful
single-turn replies: tool calls, exhausted output limits, refusals, and content
filters have distinct portable reasons. The agent loop interprets those reasons
as described in [Reading Outcomes](https://arkoren.dev/docs/guides/llm/overview.md#reading-outcomes).

Stopping iteration cancels the HTTP request. A stream missing its terminal event
fails with `llm.ErrUnavailable` wrapping `io.ErrUnexpectedEOF`.

### Strict Schemas

Strict schema rewriting defaults on for Responses and off for Chat Completions.
To disable it explicitly, set `StrictSchemas(false)` in the construction
callback. Strict rewriting recursively marks every property required, permits
null for formerly optional properties, and sets `additionalProperties: false`.
It traverses properties, items, composition branches, `$defs`, and definitions.
Open maps without properties remain unchanged; they do not satisfy OpenAI's
strict-object requirements. A compatible server's acceptance depends on its own
schema support. Turning rewriting off does not guarantee that a server accepts
open maps or structured output.

A null optional value decodes to Go zero, which typed validation treats as absent.
With strict mode off, schemas remain unchanged and are sent with `strict: false`.

Tool and output schemas must have JSON-object roots in either mode. JSON `null`,
arrays, and scalar roots fail locally with `llm.ErrInvalidRequest` before the
native hook or an HTTP request. Strict mode does not replace the application's
validation of model output or tool arguments.

Structured output uses `text.format` in Responses or `response_format` in Chat
Completions, with type `json_schema`. Unsupported formats return
`llm.ErrUnsupported`; there is no prompt-based fallback. Use
[Structured Output](https://arkoren.dev/docs/guides/llm/structured-output.md) to declare and validate the Go result.

### Provider Failures

The SDK's `*openai.Error` stays available through `errors.As`. The driver maps
429 to `llm.ErrRateLimited`, server failures to `ErrUnavailable`, missing models
to `ErrNotFound`, context-limit failures to `ErrContextLength`, and policy
refusals to `ErrRefused`. Other client failures normally match `ErrInvalidRequest`.

A 400/422 naming structured-output fields while an output schema was sent can
match `ErrUnsupported`, except `invalid_json_schema`, which remains
`ErrInvalidRequest`. A failed Responses body/stream event uses its error code;
unknown body codes also become `ErrInvalidRequest`. Do not treat an invalid
schema as evidence that the server lacks structured output.

### Response Notifications

This setup is optional; ordinary generation, streaming, and embeddings do not
require a webhook endpoint. When your application creates background work through
OpenAI's provider API, receive its response lifecycle notifications with `openaiwebhook` from the same
module. This adapter neither starts background work nor changes `openaillm`
requests. Copy the endpoint's `whsec_` signing secret from OpenAI's webhook
settings, not your API key.

With imports `arkoren.dev/core/http/webhook` and
`arkoren.dev/driver/openai/openaiwebhook`, supply your
`accept` function of type `webhook.AcceptFunc[openaiwebhook.Response]`:

```go
source, err := openaiwebhook.New(signingSecret)
if err != nil {
	return err
}
handler, err := webhook.Handler(source, func(receiver *webhook.Builder[openaiwebhook.Envelope]) {
	receiver.Handle(openaiwebhook.ResponseCompleted, accept)
})
if err != nil {
	return err
}
r.Post("/webhooks/openai", handler)
```

Here `r` is your application router; check both construction errors before
registering the handler. Configure its public HTTPS URL and event subscriptions
in OpenAI. Core acknowledges after acceptance succeeds; the
[webhook guide](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md) covers queueing and receipt policy.
The adapter also defines failed, canceled, and incomplete events. These are
response references, not generated output: retrieve the response in your handler
or worker.

The official SDK verifies the raw body and a five-minute past/future timestamp
window. Keep clocks synchronized and use deduplication to reduce replays within
that window. `Delivery.ID` is the signed `webhook-id`; `Envelope.ID` is the
separate JSON event ID. For payloads and failure behavior, see the
[OpenAI webhook reference](https://pkg.go.dev/arkoren.dev/driver/openai/openaiwebhook).

`New` requires a canonical `whsec_` base64 secret with at least 24 decoded bytes.
Invalid configuration wraps `webhook.ErrInvalid`; invalid authentication matches
both `openaiwebhook.ErrInvalidSignature` and `webhook.ErrRejected` (400 by
default). Authenticated malformed envelopes or bound response references return
503. Bound events need `data.id` and a positive Unix-seconds `created_at`; an
optional nonempty `object` must be `event`. Authenticated unbound events receive
200 without event-specific decoding.

No response is fetched automatically. `ResponseCanceled` uses native event key
`response.cancelled`; failed, canceled, and incomplete payloads remain thin
references, not output snapshots or failure reasons. OpenAI's
[webhook delivery policy](https://developers.openai.com/api/docs/guides/webhooks#handling-webhook-requests-on-a-server)
describes retries for up to 72 hours, but that is provider policy, not an
exactly-once receiver guarantee. Scope deduplication by project/endpoint and make
acceptance idempotent.

### Request Hooks

If a deployment needs an OpenAI-only setting, use a request hook rather than
changing portable agent code. Import `responses` from
`github.com/openai/openai-go/v3/responses` and add this to the driver's existing
construction callback to keep response storage explicitly disabled:

```go
driver.ResponsesHook(func(params *responses.ResponseNewParams) {
	params.Store = openai.Bool(false)
})
```

This fragment also imports `openai` from `github.com/openai/openai-go/v3`.
It reinforces a request setting, not a guarantee against provider retention.
The hook sees the translated history, tools, output schema, and defaults before
each Responses request, including each agent turn and each streaming request.
It does not run during construction. Its edits are not reset by the driver:
setting `params.Store = openai.Bool(true)` deliberately changes the storage
request. Do not use `store: false` as a guarantee against provider retention.

Keep captures concurrency-safe, do not retain request parameters, and check that
the server supports the fields you set. `ChatHook` provides the corresponding
Chat Completions boundary, including an explicit `Store` override. For a Chat
stream, the driver subsequently sets `stream_options.include_usage` to `true`,
even if the hook disabled it. Neither hook applies to embeddings.

### Native Clients

If your application already configures the official SDK, import `openai` from
`github.com/openai/openai-go/v3` and pass your existing `client openai.Client`
to `openaillm.NewFrom(client)`. The caller retains ownership and connection and
retry settings; the driver's API mode, schemas, batch size, and hooks still
apply.

`NewFrom` ignores connection edits but validates settings just like `New`.
Construction callbacks run once in order on a fresh draft; retained drafts
cannot alter published settings. Native client internals and hook captures remain
shared. A nil hook clears it; hooks must not retain request parameters.

To make SDK calls outside the contract, use the existing driver's client:

```go
client := driver.Client()
models, err := client.Models.List(ctx)
```

Here `ctx` is your request or command context. On success, `models.Data` contains
one page of provider model records. Native operations do not inherit the
facade's validation, events, or telemetry and are not portable contract methods.

### Local Servers

[Local Development](https://arkoren.dev/docs/guides/llm/local-development-with-ollama.md) shows how to
point the driver at a server on your machine.

### API References

For every function and option, see [the `openaillm` API
reference](https://pkg.go.dev/arkoren.dev/driver/openai/openaillm) and the optional
[OpenAI webhook adapter](https://pkg.go.dev/arkoren.dev/driver/openai/openaiwebhook).

---

Source: https://arkoren.dev/docs/guides/llm/conversation-history.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/conversation-history)

# Conversation History

A model sees only the messages you send on that call. Arkoren does not maintain
a session or trim old turns for you. Keep the conversation in your application,
authorize access to it for the current caller, and decide what content may be
sent to the provider. Stored messages can contain private prompts, media, tool
results, and validation feedback.

These examples use `llm` from `arkoren.dev/core/service/llm`, standard
`encoding/json/v2`, a configured `model`, and an operation context `ctx`.
[Drivers](https://arkoren.dev/docs/guides/llm/drivers.md) owns connection setup; [Running Agents](https://arkoren.dev/docs/guides/llm/running-agents.md)
owns the tool loop.

## Continuing Conversations

To ask a follow-up to a single-turn reply, retain the entire assistant message
before adding another user message:

```go
history := []llm.Message{llm.UserText("Explain the Sunday search maintenance.")}
reply, err := model.Generate(ctx, llm.Request{Messages: history})
history = append(history, reply.Message, llm.UserText("Which time zone is that?"))
followup, err := model.Generate(ctx, llm.Request{Messages: history})
```

Use this path only when your application handles any requested tools itself.
`Generate` does not execute them. Check its [stop reason](https://arkoren.dev/docs/guides/llm/overview.md#reading-outcomes)
before treating it as a finished answer.

For an agent run, the returned `Result.Messages` already includes the original
history, assistant messages, and tool results. With an agent named `assistant`:

```go
result, err := model.Run(ctx, assistant, history)
history = append(result.Messages, llm.UserText("Give me a shorter explanation."))
next, err := model.Run(ctx, assistant, history)
```

These fragments assume successful calls. Do not append only `result.Text`:
that discards tool calls, results, and reasoning needed for later turns. An
agent history must end with a user message. System instructions belong in the
request or agent definition, not a synthetic system-role message.

## Saving Messages

To save a successful run in your own storage, encode its complete conversation:

```go
data, err := json.Marshal(result.Messages)
```

The encoded parts have a `"type"` discriminator. Loading the same representation
restores concrete `Text`, `Image`, `Document`, `ToolCall`, `ToolResult`, and
`Reasoning` parts:

```go
var restored []llm.Message
err = json.Unmarshal(data, &restored)
```

Unknown part types fail decoding. Preserve opaque `Reasoning.Signature` values
unchanged; do not interpret or rewrite them. Driver modes differ in whether
they replay reasoning at all; consult the [compatibility matrix](https://arkoren.dev/docs/guides/llm/drivers.md#compatibility-matrix).

When trimming history to fit a model's context limit, keep tool calls and their
matching results together. A result's `CallID` answers a call's `ID`. Removing
one side can leave a conversation the provider cannot accept. Arkoren does not
perform automatic compaction, retention, or storage encryption.

## Adding Media

If the selected model accepts images, include the question and bytes in one
user message. Here `pngBytes` contains the image your application has approved
for provider access:

```go
message := llm.User(
	llm.Text{Text: "Which maintenance window is shown on this notice?"},
	llm.Image{Data: pngBytes, MediaType: "image/png"},
)
```

`Image{URL: imageURL}` is an alternative when the server supports remote images.
URL retrieval policy belongs to that server. Documents use
`llm.Document{Data: documentBytes, MediaType: "application/pdf"}` and require a
driver mode and model that support them.

`llm.User(parts...)` copies the outer parts slice, not nested buffers.
`Image.Data`, `Document.Data`, and `ToolCall.Arguments` remain shared. Configure
them before a call and do not mutate messages or their buffers during a call
or while consuming a stream. A copied message slice is not a deep snapshot.
Serialize or deep-copy content yourself when another owner needs to mutate it.

## Handling Partial History

After model turns have occurred, a failed `Run` returns the conversation and
usage so far; its typed `Output` remains zero. Inspect the error before deciding
whether that history can be resumed.

At `llm.ErrMaxSteps`, final-turn tool calls receive error results saying they
were not run. That repaired history can be passed to a new run. Other failures
can leave a partial turn, including unanswered tool calls. Do not blindly
resubmit it or repeat tool effects. [Run limits](https://arkoren.dev/docs/guides/llm/running-agents.md#limiting-runs)
explain the execution boundary.

`RunStream` does not return `Result.Messages`. Its `Done` events contain model
responses, not the tool-result messages appended inside the run. Use `Run` when
you need the complete conversation to persist or resume.

For message formats and ownership, see the [llm API reference](https://pkg.go.dev/arkoren.dev/core/service/llm).

---

Source: https://arkoren.dev/docs/guides/llm/running-agents.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/running-agents)

# Running Agents

When an answer needs application data, the model can request a tool, read its
result, and then answer. An **agent** defines that repeated-turn behavior:
instructions, available tools, and the final output type. `Model.Run` performs
the loop; the agent itself holds no conversation state.

This is an execution loop in your service, not an installed coding-agent skill
or an [MCP server](https://arkoren.dev/docs/guides/mcp/first-server.md). The examples use `llm` from
`arkoren.dev/core/service/llm`, standard `errors` and `fmt`, a configured `model`,
and a caller-specific `ctx`. [Drivers](https://arkoren.dev/docs/guides/llm/drivers.md) owns provider setup.

## Defining Agents

For a maintenance assistant that returns prose, declare a reusable text agent:

```go
var assistant = llm.NewAgent("maintenance-assistant", func(agent *llm.AgentBuilder[string]) {
	agent.Instructions("Explain maintenance notices briefly. Do not invent schedules.")
})
```

Definitions are immutable values, normally package-level variables. A builder
is a construction draft, not a runtime reconfiguration handle. Check
`assistant.Err()` during application construction: invalid names, duplicate or
invalid tools, and output-schema errors are reported before provider I/O.
The zero agent is invalid. Names use 1–64 bytes of lowercase letters, digits,
and inner hyphens.

Only the exact `string` type works without an output schema. Structs and named
string types need [Structured Output](https://arkoren.dev/docs/guides/llm/structured-output.md).
Publication detaches tool containers and rendered schema bytes, not schema
implementations, handlers, policies, or their captures. Finish configuring
shared dependencies before concurrent use.

## Running Turns

Send a user question with the request's authenticated context. For a plain-text
agent, the final typed output and text are strings:

```go
result, err := model.Run(ctx, assistant, []llm.Message{
	llm.UserText("Explain this notice: search is offline Sunday, 02:00–02:15 UTC."),
})
fmt.Println(result.Output)
```

On success, `result.Steps` counts model turns and `result.Usage` sums their token
usage. `result.Messages` includes the input and messages added by the run; keep
it for [conversation history](https://arkoren.dev/docs/guides/llm/conversation-history.md). The input must end with
a user message and is not modified, but nested message buffers remain shared.

To vary one call without replacing the definition, supply run options:

```go
result, err := model.Run(ctx, assistant, history,
	llm.Instructions("Use plain language for a public status page."),
	llm.MaxTokens(512),
)
```

Here `history` is your caller-owned `[]llm.Message`. Added instructions follow
the stable agent instructions after a blank line. They change the prompt prefix
and can reduce provider prompt-cache reuse. Keep stable guidance in the agent
and per-user facts in messages when possible. `llm.UseModel(modelID)` selects a
different model for one run.

## Adding Tools

When the answer needs a live schedule, attach the `lookupWindow` tool from
[Defining Tools](https://arkoren.dev/docs/guides/llm/defining-tools.md) by replacing the earlier definition:

```go
var assistant = llm.NewAgent("maintenance-assistant", func(agent *llm.AgentBuilder[string]) {
	agent.Instructions("Look up maintenance windows before reporting their times.")
	agent.Tools(lookupWindow)
	agent.ToolConcurrency(2)
})
```

The run advertises allowed tools, calls the model, executes guarded tool
requests, appends results, and repeats until it accepts a final answer or stops.
Availability and schema validation still apply to every attempted execution;
model instructions are not an authorization mechanism.

Calls from one reply run concurrently. `ToolConcurrency(1)` serializes them;
the default `0` is unlimited. Results return in model-request order, not finish
order. Make handlers and their dependencies concurrency-safe, and choose a
limit compatible with shared connections and external rate limits.

## Limiting Runs

Each model turn is one step, including structured-output correction attempts.
The first positive limit wins: run `llm.MaxSteps`, agent `MaxSteps`, model
`ModelBuilder.MaxSteps`, then 10. This bounds turns, not total duration, token
usage, or the number of tools requested in one turn.

```go
result, err := model.Run(ctx, assistant, history, llm.MaxSteps(3))
if errors.Is(err, llm.ErrMaxSteps) {
	history = result.Messages
	// Save the repaired history for an explicitly authorized continuation.
}
```

If the last allowed turn requests tools, none of those calls run. Arkoren adds
an error result for each and returns `llm.ErrMaxSteps`, leaving resumable
history. Other errors can leave a partial turn: inspect history rather than
automatically resuming or repeating side effects.

An output cap is per turn: the first positive run or agent cap overrides the
model fallback. A truncated turn returns `llm.ErrTruncated`; refusal or content
filtering returns `llm.ErrRefused`. Those checks happen before tools or output
decoding. A failed run leaves typed `Output` zero, even when `Text` or history
contains content. Unknown tool requests become error results; if the run later
fails after such a request, its error also matches `llm.ErrUnknownTool`.

## Canceling Work

Use a deadline and pass `ctx` to every blocking dependency. A permanent tool
failure cancels sibling calls and ends the run; the initiating failure or parent
cancellation remains the cause, rather than a later sibling cancellation.
Arkoren waits for admitted calls to finish. Cancellation cannot undo effects
or forcibly terminate a handler that ignores its context.

> [!WARNING]
> Tool panics are not recovered. Tools execute in their own goroutines, so HTTP
> recovery middleware around `Run` cannot catch them. An unrecovered panic
> terminates the process. Return errors for expected failures and recover inside
> a tool when a dependency can panic.

For progress without a returned conversation or typed result, use
[RunStream](https://arkoren.dev/docs/guides/llm/streaming.md#streaming-agents). Stopping its consumer cancels and
joins admitted work; it is not a rollback.

## Observing Runs

With model events configured, returning observed runs publish
`llm.agent.completed` or `llm.agent.failed`; observed tool outcomes publish
`llm.tool.invoked` or `llm.tool.failed`. Invalid declarations and discovery-policy
failures happen before the agent observation begins. Known admitted tool calls
receive observations; unknown names do not create tool spans or events.

Payloads hold names, counts, durations, and bounded failure classifications,
not arguments or results. Consumer stops and panics do not invent an agent
outcome. Already completed tool observations remain published. Apply the
[observation and privacy rules](https://arkoren.dev/docs/guides/llm/overview.md#observing-calls) to returned errors
and saved conversations as well.

For all agent and run options, see the [llm API reference](https://pkg.go.dev/arkoren.dev/core/service/llm).

---

Source: https://arkoren.dev/docs/guides/llm/defining-tools.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/defining-tools)

# Defining Tools

A **tool** is an application function the model can request by name. The model
receives its description and input schema; an [agent run](https://arkoren.dev/docs/guides/llm/running-agents.md#adding-tools)
checks the request before calling the typed handler. Treat model arguments as
untrusted input, even when the provider enforces a schema.

The examples use `llm` from `arkoren.dev/core/service/llm`, `validate` from
`arkoren.dev/foundation/validate`, and standard `context` and `errors`. They
assume an application-owned `windows` service. Its `FindAuthorized` method has
signature `func(context.Context, string) (WindowSummary, error)` and checks the
authenticated caller's access to the selected service before returning data.
Neither that service nor its authorization is supplied by Arkoren.

## Defining Inputs

To answer questions about maintenance, let the model select a service by ID.
Keep the input small and describe the field's meaning in the schema:

```go
type WindowQuery struct {
	ServiceID string `json:"service_id"`
}

var windowQuery = validate.Struct(func(object *validate.StructBuilder[WindowQuery]) {
	object.Field("ServiceID", func(id *validate.StringBuilder[string]) {
		id.Required()
		id.Describe("The service identifier supplied by the user.")
	})
})
```

`NewTool` renders this schema for the provider. At execution, it decodes JSON
with json/v2 and calls the schema's `Parse`, applying defaults before the handler.
Decoding failures and validation violations become model-facing error results;
the handler does not run. Keep schema descriptions and validation diagnostics
free of secrets because the provider can receive them.

The OpenAI driver requires JSON-object schema roots and applies mode-specific
[strict-schema rewriting](https://arkoren.dev/docs/guides/llm/drivers.md#strict-schemas). Provider enforcement does
not replace application validation or permission checks.

## Returning Results

Return a narrow result instead of a complete internal record. This tool exposes
only the approved schedule fields:

```go
type WindowSummary struct {
	StartsAt string `json:"starts_at"`
	EndsAt   string `json:"ends_at"`
}

func findWindow(ctx context.Context, in WindowQuery) (WindowSummary, error) {
	return windows.FindAuthorized(ctx, in.ServiceID)
}
```

`FindAuthorized` must obtain the principal from trusted context, not from a
model-supplied user ID. A valid service ID is not proof of access. Send the
authenticated caller's context into the run and onward to blocking dependencies.

Publish the tool once, normally at package scope:

```go
var lookupWindow = llm.NewTool(
	"lookup-window", "Find the authorized maintenance window for a service.",
	windowQuery, findWindow,
)
```

Strings go to the model unchanged; other results are JSON-encoded. Tools have
no output schema. The result must contain only information the caller, model,
and provider may receive.

Check `lookupWindow.Err()` during construction. Invalid names, nil schemas or
handlers, and schema-rendering failures make the tool unusable; the zero tool
is invalid. Names use 1–64 bytes of lowercase letters, digits, and inner
hyphens. An agent containing an invalid tool fails before provider I/O.

## Offering Capabilities

An account might enable maintenance lookups without granting access to every
service. Use availability to offer that capability, while retaining the
resource check in `FindAuthorized`. Here `canViewMaintenance` is your
observational, concurrency-safe `func(context.Context) (bool, error)`:

```go
var lookupWindow = llm.NewTool(
	"lookup-window", "Find the authorized maintenance window for a service.",
	windowQuery, findWindow, func(tool *llm.ToolBuilder) {
		tool.Available(canViewMaintenance)
	},
)
```

This replaces the earlier declaration. Without a policy, a tool is available;
explicitly supplying a nil policy is a declaration error.

At discovery, `Run` and `RunStream` evaluate each policy once before model I/O,
preserving declaration order. The advertised list remains fixed for that run.
Hidden tools remain guarded in the registry: guessing a name does not bypass
the policy. Execution checks availability again after acquiring a concurrency
slot, before JSON decoding, schema parsing, or the handler. Denial sends only
`tool unavailable` to the model and lets the loop continue.

A newly granted capability appears only in a new run. Revocation blocks later
calls but cannot retract metadata already sent to the provider. Keep sensitive
facts out of names, descriptions, and schemas. `lookupWindow.Available(ctx)`
previews the policy without parsing arguments or invoking the handler; it does
not authorize a specific resource.

A policy error preserves its cause under `llm.ErrToolAvailability` and never
sends its diagnostic to the model, even if it wraps `ToolError`. Discovery
failure prevents model I/O. Execution-policy failure also matches
`llm.ErrPermanent`, cancels siblings, and ends the run. The initiating failure
is not replaced by a sibling's later cancellation.

## Handling Failures

Ordinary handler errors become the generic model-facing text `the tool failed`;
the model can continue or request another call. Use `ToolError` only for feedback
you explicitly approve for publication. For example, when the authorized lookup
returns your application sentinel `ErrNoWindow`:

```go
window, err := windows.FindAuthorized(ctx, in.ServiceID)
if errors.Is(err, ErrNoWindow) {
	return WindowSummary{}, llm.ToolError("No maintenance window is scheduled.")
}
return window, err
```

Do not expose unauthorized resource existence through an error message. Wrapped
`ToolError` text is also visible to the model. Validation feedback follows its
own publication path; an operational schema error is otherwise an ordinary
redacted tool failure, not permission to send its diagnostics.

For a failure that must end the run, return `llm.Permanent(err)`, with a non-nil
cause. The error matches `llm.ErrPermanent`; siblings receive cancellation and
are joined before the run returns. Cancellation does not undo effects or
terminate a noncooperative handler. Follow the [tool panic warning](https://arkoren.dev/docs/guides/llm/running-agents.md#canceling-work)
because handlers run in separate goroutines without panic recovery.

## Sharing Definitions

Tool definitions are immutable, but schemas, handlers, policies, and captured
dependencies remain shared. Rendered schema bytes are copied at construction;
retaining a builder cannot change the published tool. Configure shared values
before concurrent use and avoid request-specific state in package-level captures.

For the same operation exposed through LLM and MCP, share the typed handler,
schema, and context-only policy, then declare separate transport tools. There
is no automatic converter. [MCP authorization](https://arkoren.dev/docs/guides/mcp/availability-and-authorization.md)
explains the same distinction between offering a capability and authorizing a
resource.

For tool declarations and errors, see the [llm API reference](https://pkg.go.dev/arkoren.dev/core/service/llm).

---

Source: https://arkoren.dev/docs/guides/llm/structured-output.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/structured-output)

# Structured Output

A maintenance notice may describe a service and an outage duration in prose,
while your application needs fields it can compare or store. Structured output
asks the model for JSON, then decodes and validates the final reply into a Go
value. The provider's schema support guides generation; application validation
still decides whether to accept the result.

Use a configured `model` and operation context `ctx` from [Drivers](https://arkoren.dev/docs/guides/llm/drivers.md).
The examples import `llm` from `arkoren.dev/core/service/llm`, `validate` from
`arkoren.dev/foundation/validate`, and standard `fmt`. Here `notice` is text your
application has approved for the provider.

## Defining Output

Start with the fields your application actually consumes:

```go
type Maintenance struct {
	Service string `json:"service"`
	Minutes int    `json:"minutes"`
}
```

Describe their meaning and allowed values using a
[typed validation schema](https://arkoren.dev/docs/guides/validation/overview.md):

```go
var maintenanceSchema = validate.Struct(func(object *validate.StructBuilder[Maintenance]) {
	object.Field("Service", func(service *validate.StringBuilder[string]) {
		service.Required()
		service.Describe("The service affected by the notice.")
	})
	object.Field("Minutes", func(minutes *validate.IntBuilder[int]) {
		minutes.Required()
		minutes.Min(1)
		minutes.Describe("The stated outage duration in minutes.")
	})
})
```

`Describe` text travels with the schema to the model. Rules JSON Schema cannot
express, such as cross-field checks, are not sent but still run during local
validation. Do not mistake a valid shape for factual accuracy or permission to
perform an action.

## Running Validation

Attach the schema to an immutable agent definition:

```go
var extractMaintenance = llm.NewAgent("extract-maintenance", func(agent *llm.AgentBuilder[Maintenance]) {
	agent.Instructions("Extract only the service and duration stated in the notice.")
	agent.Output(maintenanceSchema)
})
```

Run the agent with the notice as a user message:

```go
result, err := model.Run(ctx, extractMaintenance, []llm.Message{llm.UserText(notice)})
fmt.Println(result.Output.Service, result.Output.Minutes)
```

On success, `Output` is the decoded value after schema defaults and validation.
`Text` remains the raw final reply. On failure, `Output` stays zero; inspect
the error before consuming fields. Only the exact `string` type can omit an
output schema; named string types also need one.

Check `extractMaintenance.Err()` at application construction. Invalid or
unrenderable schemas fail before provider I/O. Rendered bytes are copied, but
the schema implementation remains shared; follow [definition ownership](https://arkoren.dev/docs/guides/llm/running-agents.md#defining-agents)
before using it concurrently.

## Correcting Replies

If final JSON cannot decode or validation directly returns `validate.Errors`,
the agent appends a user message explaining the violation and asks again.
Custom `Check` callbacks use `validate.Fail` for such correctable violations.
Feedback enters both saved history and a subsequent provider request, so keep
custom diagnostics free of secrets and internal details.

By default, one correction is allowed. To allow two, add this to the existing
agent construction callback:

```go
agent.OutputRetries(2)
```

Zero selects the default of one retry; a negative value disables retries.
Each correction attempt takes another model turn and counts against
[MaxSteps](https://arkoren.dev/docs/guides/llm/running-agents.md#limiting-runs). When retries run out, the error
matches `llm.ErrInvalidOutput` and preserves the decoding or validation cause.
The returned result retains history so far. If the step limit runs out first,
the error is `llm.ErrMaxSteps`; correction feedback may already be in history.

Operational schema errors are different. A failing dependency in `Check`,
cancellation, or an error wrapping rather than directly returning
`validate.Errors` stops output processing immediately. Its cause remains
available without adding `llm.ErrInvalidOutput`, consuming an output retry, or
appending diagnostics to history. Failure events expose only the
[bounded classification](https://arkoren.dev/docs/guides/llm/overview.md#observing-calls), not the diagnostic.

These retries correct output; they do not retry provider failures. Truncation
and refusal stop the run before output decoding.

## Schema Limits

Keep optional fields optional in the portable schema. The OpenAI driver's
[strict rewriting](https://arkoren.dev/docs/guides/llm/drivers.md#strict-schemas) makes optional properties required
but nullable; returned null decodes to Go zero, which validation treats as
absent. Strict mode changes the transmitted schema, not local validation.

OpenAI tool and output schema documents must themselves be JSON objects in
either mode; raw JSON arrays, scalars, and null fail locally before the request
hook or HTTP request. This is not a promise that every schema expressed as an
object is supported by the server. Open maps without properties are
not closed by strict rewriting and do not meet strict-object requirements.
Turning rewriting off does not establish server support for those schemas.

If a server rejects the structured-output format, the driver returns
`llm.ErrUnsupported`. There is no prompt-based fallback, and output correction
does not repair unsupported provider features.

## Single-Turn Output

If your application manages turns itself, request the schema directly without
an agent:

```go
schema, err := maintenanceSchema.JSONSchema()
reply, err := model.Generate(ctx, llm.Request{
	Messages: []llm.Message{llm.UserText(notice)},
	Output:   schema,
})
```

`reply` is still an `llm.Response`, not a validated `Maintenance`. Check its
stop reason, collect text, decode, and validate it yourself. Prefer the agent
when you want Arkoren to own that acceptance and correction loop.

For all options, see the [llm API reference](https://pkg.go.dev/arkoren.dev/core/service/llm)
and [validate API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/llm/streaming.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/streaming)

# Streaming Replies

A long answer leaves the user waiting if you display it only after generation
finishes. Streaming lets you show text as it arrives; it does not promise a
shorter total generation time. Use `Model.Stream` for one turn and
`Model.RunStream` for an agent's turns and tool progress. Both return
`collection.TryLazySlice[llm.Event]`, so you can use `for ... range` or native
collection operations. Breaking out of the loop cancels the request to the vendor.

Construction and lazy operations such as `Filter`, `Map`, and `Take` do no model
I/O. Traversal starts the work. Traversing the same collection again issues new
provider calls, which can be billed again; an agent can also repeat tool effects.
Materialize once if you need to reuse the observed events without another run.

The examples use `llm` from `arkoren.dev/core/service/llm` and the standard
`fmt` package. Reuse `model` and `ctx` from the
[driver setup](https://arkoren.dev/docs/guides/llm/drivers.md#connecting). Use an operation context with a deadline;
the request context is appropriate inside an HTTP handler.

## Streaming Text

If you want to print an answer as it is written, read text fragments from
`Stream` rather than waiting for `Generate` to return the whole response. Use
your existing `model` and a `request` of type `llm.Request`, with the same
shape as [a single-turn request](https://arkoren.dev/docs/guides/llm/overview.md#generating-replies):

```go
for event, err := range model.Stream(ctx, request) {
	if err != nil {
		return err
	}
	if delta, ok := event.(llm.TextDelta); ok {
		fmt.Print(delta.Text)
	}
}
```

The events are `TextDelta` and `ReasoningDelta` fragments, `ToolCallStarted` and
`ToolCallDelta` for a tool call being written, and finally `Done`, which carries
the complete `Response`. A consumer that ignores the deltas still gets the
result from `Done`. A stream that ends without `Done` or an error yields
`llm.ErrUnavailable`.

A terminal error means the stream failed; text already shown to the user does
not mean a complete answer was received.

## Streaming Agents

An agent may pause its answer while it looks up an order or calls another tool.
Showing that progress helps the person waiting understand what is happening.
`RunStream` forwards the text from each turn along with tool-progress events.
Use the `assistant` agent from [Running Agents](https://arkoren.dev/docs/guides/llm/running-agents.md) and your
stored `history`:

```go
for event, err := range model.RunStream(ctx, assistant, history) {
	if err != nil {
		return err
	}
	switch e := event.(type) {
	case llm.TextDelta:
		fmt.Print(e.Text)
	case llm.ToolStarted:
		fmt.Println("running", e.Name)
	case llm.ToolFinished:
		fmt.Println("finished", e.Name, "error:", e.IsError)
	case llm.StepCompleted:
		fmt.Println("step", e.Step, "done")
	}
}
```

`ToolStarted` and `ToolFinished` carry the call ID and name; `ToolFinished` also
carries an error flag. Neither carries arguments or results. `ToolStarted`
follows admission to the concurrency slot, not merely the model's request for a
call. A queued call canceled before admission is not announced. A start does not
guarantee handler execution: availability and argument validation still apply.
If the tool phase completes without a fatal error, `ToolFinished` events arrive
in call order after every admitted call has finished.

`RunStream` cannot return a `Result`; use `Run`
when you need the complete conversation to persist or resume. `Done` contains
each model response, not the tool-result messages appended inside the run,
so collecting `Done` events alone does not reconstruct that conversation.

For a [structured-output agent](https://arkoren.dev/docs/guides/llm/structured-output.md), streamed text can still
be incomplete or fail final validation. Do not treat a text delta as a validated
typed result; keep reading the stream's error value.

## Stopping Streams

To record a normally completed standalone turn, consume `Model.Stream` through
its natural end. Breaking or returning from a successful event, including
`Done`, cancels the driver context and suppresses both `llm.response.generated`
and `llm.response.failed`. The turn span ends without error. A driver failure
is different: its observation is finalized before its error is yielded, so
stopping on that error does not retract the failure event.

Stopping `RunStream` cancels the run, joins admitted tool calls, and suppresses
the agent outcome event. Completed tool observations remain published. Tool
handlers must honor cancellation; stopping the consumer cannot undo effects or
forcibly terminate a blocked handler. A panic also propagates rather than
becoming a successful stop; the
[observation rules](https://arkoren.dev/docs/guides/llm/overview.md#observing-calls) explain its bounded
telemetry and lack of an invented outcome event.

## Browser Delivery

For a chat page, the fragments need to reach the browser as they arrive at your
server. `response.SSEJSON` sends each iterator value as a JSON-encoded
server-sent event. Use `.Iter()` to pass the collection to this standard
`iter.Seq2` consumer. Each event
has a `"type"` tag the browser can use to distinguish text from tool progress.

The iterator must cooperate with cancellation and shutdown; blocking model and
tool work needs a way to finish before the streaming handler can return.

Inside your existing Arkoren handler, with `w`, `req`, and the conversation `history`
already available, import `response` from `arkoren.dev/core/http/response`:

```go
return response.SSEJSON(w, req, model.RunStream(req.Context(), assistant, history).Iter())
```

In the browser, read the stream with `fetch` for a `POST`, or with `EventSource`
for a `GET`, and parse each event's `data` as JSON.

`SSEJSON` stops consuming values on request cancellation, server shutdown, or a
write or flush failure. It makes `yield` return false and waits for the iterator
producer to exit; it does not abandon that goroutine. Producers must return when
`yield` returns false and must not yield further values, even a final `Done`.

Client disconnects cancel `req.Context()`. Graceful shutdown is different: an
Arkoren `server.Server` announces it through `server.ShuttingDown(req.Context())`
without canceling that request context. The shutdown helper is in
`arkoren.dev/core/http/server`. A write or flush failure does not cancel the
request context either.

The streaming helper cannot forcibly interrupt a model call or tool handler
blocked outside `yield`. Arrange for that work to stop on request cancellation
and the shutdown signal, or give it another bounded way to exit. Passing only
`req.Context()` does not guarantee prompt graceful shutdown while work is blocked;
a noncooperative producer can keep `SSEJSON` waiting indefinitely.
[Streaming Responses](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md) explains the write
deadline and cooperative shutdown behavior of streaming responses.

> [!NOTE]
> The headers are sent before the first event. An error that occurs later, such
> as a rate limit, ends the stream and is returned to the handler's caller, but
> cannot change the committed status code. Check what you can before streaming.
> If the browser needs an error event, send an application-approved message from
> a wrapper iterator, not raw provider, tool, or schema diagnostics.

## Collecting Events

To display only text, filter the stream directly; no collection conversion or
additional import is required:

```go
text := model.Stream(ctx, request).Filter(func(_ int, item llm.Event) (bool, error) {
	_, ok := item.(llm.TextDelta)
	return ok, nil
})
for item, err := range text {
	if err != nil {
		return err
	}
	fmt.Print(item.(llm.TextDelta).Text)
}
```

Filtering out `Done` does not redefine completion or bypass the source's
missing-`Done` validation. Reading this filtered stream through its natural end
still completes the source turn. `Take(n)` instead stops early after its limit:
it cancels and joins the source under the same rules as breaking the loop.

For finite streams, `events, err := model.Stream(ctx, request).Items()` collects
events into a Go slice. On failure, `events` contains the successful prefix and
`err` retains the terminal error. That prefix is not a completed response, even
if it contains text or a `Done` before a later error. `RunStream` materialization
does not produce the conversation or typed result returned by `Run`.

Materialization requires a finite source and holds the collected events in
memory. JSON encoding the collection also materializes it; it does not send
events to the browser as they arrive. Use `SSEJSON` with `.Iter()` for that.

[Fallible Collections](https://arkoren.dev/docs/guides/collections/fallible-collections.md) lists what a
`TryLazySlice` can do.

For every function and option, see [the `llm` API reference](https://pkg.go.dev/arkoren.dev/core/service/llm)
and [the `response` API reference](https://pkg.go.dev/arkoren.dev/core/http/response).

---

Source: https://arkoren.dev/docs/guides/llm/embeddings.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/embeddings)

# Embeddings

A search for “planned downtime” should find an article titled “Sunday maintenance”
even if the words differ. An **embedding** represents text as a vector of
numbers so a search system can compare meaning instead of only exact words.
`*llm.Embedder` requests those vectors; it does not store them or run a search.

Use `llm` from `arkoren.dev/core/service/llm`, a configured `driver` from
[Drivers](https://arkoren.dev/docs/guides/llm/drivers.md), and an operation context `ctx`. Choose a model intended
for embeddings, separately from the model that generates replies. Here
`embeddingModelID` is that selected ID.

## Embedding Text

To prepare maintenance articles for comparison, pass their approved text to a
reusable embedder:

```go
embedder := llm.NewEmbedder(driver, embeddingModelID)
vectors, err := embedder.Embed(ctx,
	"Search is unavailable during Sunday maintenance.",
	"The status page lists completed maintenance windows.",
)
```

On success, `vectors` has type `[][]float32`: `vectors[0]` belongs to the first
article and `vectors[1]` to the second. No-input calls fail with
`llm.ErrInvalidRequest` before provider I/O. Nil contexts, nil drivers, empty
model IDs, and negative dimensions are also rejected; the facade rejects a
driver result whose vector count does not match the input count.

The OpenAI driver batches at most 2048 inputs by default and restores input
order using response indexes. Configure its `EmbedBatchSize` for a server with a
different input-count limit. Batching does not split an individual text or
enforce the provider's token budget. Bound text lengths and corpus size in your
application. The facade does not retry failed calls.

## Choosing Dimensions

By default, the provider selects the vector length. If the model supports a
smaller representation and your storage needs it, request the dimensions at
construction:

```go
embedder := llm.NewEmbedder(driver, embeddingModelID, func(embedder *llm.EmbedderBuilder) {
	embedder.Dimensions(512)
})
```

Zero restores the provider default. The model must support the requested size;
evaluate relevance with your own data before reducing it. Use the same model
and dimensions for indexed documents and incoming queries. Vectors from
different models are not interchangeable, even if they have equal lengths.

## Query Purpose

Some providers represent stored documents differently from search queries.
Mark the stored text as `llm.PurposeDocument`; construct a corresponding query
embedder with `llm.PurposeQuery`:

```go
queries := llm.NewEmbedder(driver, embeddingModelID, func(embedder *llm.EmbedderBuilder) {
	embedder.Purpose(llm.PurposeQuery)
})
queryVectors, err := queries.Embed(ctx, "When is planned downtime?")
```

`queryVectors[0]` is the query vector to compare with stored document vectors.
Use the same dimension setting on both embedders if you override it. The OpenAI
driver ignores purpose because it uses the same embedding operation for both;
other drivers can use the hint. Omission selects `llm.PurposeUnspecified`.

## Storing Vectors

Store each article's vector with its source identifier and the model/dimension
configuration that produced it. Arkoren does not prescribe a vector store,
similarity function, normalization rule, or relevance threshold. Before using
cosine similarity, verify that vectors are nonzero and have equal lengths;
those conditions are not guaranteed by the facade's vector-count check.

Embeddings are not cached automatically. Cache repeated work in your storage
layer or the [cache service](https://arkoren.dev/docs/guides/cache/overview.md). Changing the model or
dimensions requires rebuilding comparable stored vectors.

For application tests, the [scripted fake](https://arkoren.dev/docs/guides/llm/testing.md#testing-vectors) produces
deterministic hash-based vectors, not semantic embeddings. It tests wiring and
storage, not search relevance. [Local Development](https://arkoren.dev/docs/guides/llm/local-development-with-ollama.md#local-embeddings)
shows the separate Ollama embedding-model workflow.

## Observing Calls

Embedding calls produce `embeddings {model}` spans. Configure
`EmbedderBuilder.Events(bus)` separately from model events to publish
`llm.embeddings.generated` or `llm.embeddings.failed`. Payloads include model,
input count, and duration; successful events include usage. Failures contain
bounded `ErrorType`, not raw errors or texts.

Apply the [observation and privacy rules](https://arkoren.dev/docs/guides/llm/overview.md#observing-calls), including
panic behavior. Stored vectors, source texts, and returned errors remain your
application's data to protect; telemetry privacy is not a storage policy.

For dimensions, purpose, and embedding requests, see the [llm API reference](https://pkg.go.dev/arkoren.dev/core/service/llm).

---

Source: https://arkoren.dev/docs/guides/llm/local-development-with-ollama.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/local-development-with-ollama)

# Local Development

Use a downloaded Ollama model when you want to check model requests on your own
machine without a hosted API key. Ollama exposes an OpenAI-compatible API, so
you reuse the [OpenAI driver](https://arkoren.dev/docs/guides/llm/drivers.md). A local answer checks connectivity
and prompt behavior for that model, not parity with your hosted deployment.

This workflow uses local inference, not Ollama's cloud service. Cloud models
still need authentication and network access. After installation and model
download, local inference does not require an internet connection.

## Starting Ollama

Install [Ollama](https://ollama.com/download) and start its local server. If
your installation does not start it automatically, run `ollama serve` in
another terminal. Keep the unauthenticated plaintext endpoint on loopback;
do not expose it to an untrusted network.

Download the model before sending a request:

```sh
ollama pull llama3.2
```

The model remains on disk after the server stops. Its identifier is the name
you pulled, not a hosted OpenAI model name.

## Connecting Locally

Install the Go modules in your application's module directory:

```sh
go get arkoren.dev/core arkoren.dev/driver/openai
```

Import `llm` from `arkoren.dev/core/service/llm` and `openaillm` from
`arkoren.dev/driver/openai/openaillm`. Set both the loopback URL and a nonsecret
key value so the SDK cannot inherit a hosted `OPENAI_API_KEY`:

```go
driver, err := openaillm.New(func(driver *openaillm.Builder) {
	driver.APIKey("ollama")
	driver.BaseURL("http://localhost:11434/v1")
})
```

Handle the construction error before creating the facade. Construction performs
no network I/O. With a bounded operation context `ctx`, send a small request:

```go
model := llm.New(driver, "llama3.2")
reply, err := model.Generate(ctx, llm.Request{
	Messages: []llm.Message{llm.UserText("Explain planned maintenance in one sentence.")},
})
```

On success, inspect `reply.Message` and its [stop reason](https://arkoren.dev/docs/guides/llm/overview.md#reading-outcomes).
The request goes to your local server, not OpenAI. A successful reply verifies
basic generation, not tools, structured output, or every media input.
`driver.Ping(ctx)` checks `GET /models`; register it as a
[health check](https://arkoren.dev/docs/guides/application/health-and-readiness.md) if you want readiness to
depend on that endpoint.

You can set `OPENAI_BASE_URL` instead of the callback URL, but still prevent
hosted credentials from leaking to the local endpoint. The driver needs no
`Start` or `Stop` calls.

## Selecting Modes

The default is Responses, supported by
[Ollama from 0.13.3](https://docs.ollama.com/api/openai-compatibility#responses-api).
Ollama supports non-stateful Responses: your application still sends the
[conversation history](https://arkoren.dev/docs/guides/llm/conversation-history.md). For an older server, or a
compatible server offering only Chat Completions, add this to the existing
driver callback:

```go
driver.API(openaillm.ChatCompletions)
```

Chat mode drops reasoning history/output, rejects `Document` parts with
`llm.ErrUnsupported`, and disables strict-schema rewriting by default.
[Mode Limits](https://arkoren.dev/docs/guides/llm/drivers.md#mode-limits) describes the translation boundary.

Responses initializes `store: false`, but [request hooks](https://arkoren.dev/docs/guides/llm/drivers.md#request-hooks)
can override it. That field is not a retention guarantee. Local servers,
gateways, and hosted providers have separate logging and storage policies.

## Local Embeddings

Generation and embedding use different models. Download an embedding model
before constructing the separate facade:

```sh
ollama pull nomic-embed-text
```

Reuse the local driver and embed one approved text:

```go
embedder := llm.NewEmbedder(driver, "nomic-embed-text")
vectors, err := embedder.Embed(ctx, "Search is unavailable during maintenance.")
```

On success, `vectors[0]` is a `[]float32` for that text. Keep the model and
dimensions consistent when comparing it with other vectors; [Embeddings](https://arkoren.dev/docs/guides/llm/embeddings.md)
explains storage and comparison responsibilities.

## Checking Behavior

Server compatibility and model capability are separate checks. Smaller models
can produce invalid arguments or final output; schema validation and
[output correction](https://arkoren.dev/docs/guides/llm/structured-output.md#correcting-replies) handle only the
documented classes of failure. A rejected output format returns
`llm.ErrUnsupported`, without a prompt-based fallback.

Ollama's documented Chat Completions support accepts inline image bytes, not
remote image URLs, and does not support `tool_choice`. With a vision-capable
model, use `llm.Image{Data: pngBytes, MediaType: "image/png"}` instead of assuming
URL input works. Schemas also remain subject to both the
[driver limits](https://arkoren.dev/docs/guides/llm/drivers.md#strict-schemas) and the server's support.

Use the [scripted fake](https://arkoren.dev/docs/guides/llm/testing.md) for repeatable application tests. Keep
local-server smoke tests separate and opt-in: a generated answer does not prove
tool authorization, output accuracy, or semantic search relevance.

## Cleaning Up

Stop a server you started with `ollama serve` using Ctrl+C. Do not stop a server
owned by another application. Remove downloaded models only when no longer
needed:

```sh
ollama rm llama3.2
ollama rm nomic-embed-text
```

Removing a model frees its local storage and requires another download before
you can use that identifier again.

For driver options, see the [openaillm API reference](https://pkg.go.dev/arkoren.dev/driver/openai/openaillm).

---

Source: https://arkoren.dev/docs/guides/llm/testing.md
[HTML documentation](https://arkoren.dev/docs/guides/llm/testing)

# Testing Models

Routine application tests should not depend on a billable, nondeterministic
model. `llmtest.Fake` implements `llm.Driver` with scripted replies, so your
tests can exercise the real facade, agent loop, and validation without provider
I/O. There is no memory driver for this contract; use the fake for this task.

Import `llm` from `arkoren.dev/core/service/llm` and `llmtest` from
`arkoren.dev/core/service/llm/llmtest`. These fragments run inside a test with
`t *testing.T`, `ctx := t.Context()`, and standard `errors` where shown.

## Scripting Replies

Queue the answer before calling your model. With the text `assistant` from
[Running Agents](https://arkoren.dev/docs/guides/llm/running-agents.md#defining-agents):

```go
fake := llmtest.New()
fake.Reply("Search maintenance lasts fifteen minutes.")
model := llm.New(fake, "test-model")
```

Run the real facade and assert the exact accepted output:

```go
result, err := model.Run(ctx, assistant, []llm.Message{
	llm.UserText("How long does search maintenance last?"),
})
if err != nil {
	t.Fatal(err)
}
if result.Output != "Search maintenance lasts fifteen minutes." {
	t.Fatalf("output = %q", result.Output)
}
```

Each `Generate` or `Stream` call consumes one reply. Exhausting the script
returns `llmtest.ErrNoReply`, exposing an unexpected turn. For a structured-output
agent, script JSON text and let the real output schema decode and validate it.
`ReplyParts` scripts multiple parts; `ReplyResponse` sets explicit stop reasons
and usage; `ReplyWith` computes a response or error from the request.

## Testing Tools

For a separate tool-dispatch test, create a fresh fake and model as above, then
script a request followed by the final answer:

```go
fake.ReplyToolCall("lookup-window", `{"service_id":"search"}`)
fake.Reply("Search maintenance starts at 02:00 UTC.")
```

Run an agent containing the `lookupWindow` from [Defining Tools](https://arkoren.dev/docs/guides/llm/defining-tools.md).
The script consumes two model turns with your real handler between them.
Supply that handler with a test service: faking the model does not fake tool
effects, resource authorization, or network dependencies.

Test denial with a caller context your policy rejects, and assert that the
protected handler was not invoked. Test the application's resource check
directly too; hiding a capability is not proof of row-level access control.

## Inspecting Requests

After executing the tool scenario with a fresh fake, assert on the advertised
capability rather than only the final text:

```go
fake.AssertGenerated(t, func(request llm.Request) bool {
	return len(request.Tools) == 1 && request.Tools[0].Name == "lookup-window"
})
fake.AssertNotGenerated(t, func(request llm.Request) bool {
	return request.Model != "test-model"
})
```

All predicates passed to one assertion must match the same request.
`Generated()` returns recorded generation/stream requests for detailed checks;
`Embedded()` returns embedding requests. The fake proves what your code sent,
not provider wire compatibility or model quality.

## Testing Failures

Inject an outage and check the portable error your application handles:

```go
fake.FailNext(llm.ErrUnavailable)
_, err := model.Run(ctx, assistant, []llm.Message{llm.UserText("Show the schedule.")})
if !errors.Is(err, llm.ErrUnavailable) {
	t.Fatalf("err = %v", err)
}
```

An injected failure is not recorded and consumes no scripted reply.
`FailWith(err)` keeps failing calls until cleared with nil; `Reset()` clears
scripts, records, and injected failures. Use `ReplyWith` when you need a
recorded scripted turn to fail instead.

To verify truncation handling, script a response with
`StopReason: llm.StopMaxTokens`: `Generate` returns it as data, whereas `Run`
returns `llm.ErrTruncated`. A refusal or content-filter stop similarly becomes
`llm.ErrRefused` in an agent run. Do not assert success only because partial
text exists.

## Streaming Events

The fake synthesizes word-sized text deltas and ends with `Done`. With a fresh
fake/model and a user-message `request` of type `llm.Request`:

```go
fake.Reply("Maintenance is complete.")
var text string
for event, err := range model.Stream(ctx, request) {
	if err != nil {
		t.Fatal(err)
	}
	if delta, ok := event.(llm.TextDelta); ok {
		text += delta.Text
	}
}
if text != "Maintenance is complete." {
	t.Fatalf("text = %q", text)
}
```

This checks consumption and assembly, not provider timing or HTTP framing.
Consume through natural completion when testing an outcome; breaking at `Done`
has different [observation semantics](https://arkoren.dev/docs/guides/llm/streaming.md#stopping-streams).

## Testing Vectors

Embeddings need no script. The fake returns hash-based unit vectors with the
requested dimensions, or 8 by default:

```go
embedder := llm.NewEmbedder(fake, "test-embedding")
vectors, err := embedder.Embed(ctx, "Search maintenance", "Search maintenance")
if err != nil {
	t.Fatal(err)
}
if !slices.Equal(vectors[0], vectors[1]) {
	t.Fatal("identical texts produced different test vectors")
}
```

This fragment also imports standard `slices`. Determinism tests storage and
retrieval wiring, not semantic relevance. Use `EmbedWith` to supply known
vectors when testing your own ranking logic.

## Driver Conformance

Driver authors can call `llmtest.Run` with a factory returning their driver.
The suite exercises the actual driver, not a fake transport. With a hosted
driver it makes real, potentially billable requests and does not automatically
skip under `testing.Short()`.

Keep that test opt-in: check both short mode and an explicit environment flag
before constructing a live driver. Configure credentials and model IDs your
account can access; pass them with `llmtest.WithModel(modelID)` and
`llmtest.WithEmbeddingModel(embeddingModelID)`. Do not run it as an ordinary
unit test or treat nondeterministic model output as a deterministic fixture.

Use `WithoutEmbeddings`, `WithoutImages`, `WithoutDocuments`,
`WithoutReasoning`, or `WithoutStructuredOutput` only for capabilities the
selected driver/server cannot provide. For OpenAI Chat Completions, documents
and reasoning are unsupported. [Writing a Driver](https://arkoren.dev/docs/guides/drivers/writing-a-driver.md)
explains the broader contract checks. A failing live-model assertion can be
nondeterministic; investigate before assigning the cause to the driver.

For fake methods and conformance options, see the [llmtest API reference](https://pkg.go.dev/arkoren.dev/core/service/llm/llmtest).

---

Source: https://arkoren.dev/docs/guides/mcp/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/mcp/overview)

# Overview

An assistant requesting a shipment quote needs a named operation, an argument
schema, and a result it can interpret. The **Model Context Protocol (MCP)**
provides that connection between an AI application's client and your application.
Your service still owns the calculation, access decisions, and any effects.

## Request Path

The AI application is the **host**. Its MCP client discovers tools and calls
them with JSON arguments; your MCP server dispatches those calls to application
operations:

```text
AI host → MCP client → HTTP endpoint → typed operation
```

`arkoren.dev/core/mcp` defines immutable tools without depending on a protocol
SDK. `arkoren.dev/driver/mcp/mcpserver` registers selected definitions with the
official Go MCP SDK and returns an `http.Handler`. Your application mounts that
handler and owns the listener, authentication, deadlines, and shutdown.

Serving tools does not require a model API key or a model conversation in your
Go process. If your application needs to run the model itself, use
[Running Agents](https://arkoren.dev/docs/guides/llm/running-agents.md) instead. MCP serving, LLM agents, and
the [documentation skill](https://arkoren.dev/docs/guides/agent-skills/working-with-agents.md) are separate tasks.

## Typed Operations

Start with a narrow application operation rather than exposing arbitrary service
methods. For a shipment quote, define its JSON argument and result shapes:

```go
type QuoteRequest struct {
	WeightGrams int `json:"weight_grams"`
}

type Quote struct {
	PriceCents int `json:"price_cents"`
}
```

With `quote` as your existing
`func(context.Context, QuoteRequest) (Quote, error)`, import
`arkoren.dev/core/mcp` and publish its tool definition:

```go
tool := mcp.NewTool("shipping.quote", quote, func(action *mcp.ToolBuilder[QuoteRequest, Quote]) {
	action.Description("Quote domestic shipping in cents for a parcel's weight.")
	action.ReadOnly(true)
	action.OpenWorld(false)
})
```

Clients discover `shipping.quote` with object schemas using `weight_grams` and
`price_cents`. Go types provide inferred schemas; explicit validation adds domain
constraints. Tool names are static protocol identifiers, not per-user routes.
Read-only and other annotations are descriptive hints, not permission checks.

The [Tools guide](https://arkoren.dev/docs/guides/mcp/tools.md) completes this example with weight validation,
output checks, registration, and focused tests. Its fragments keep error results
visible while omitting routine propagation; complete functions and programs
handle their errors.

## Managed Serving

A definition becomes remotely accessible only when you explicitly register it
with `mcpserver.Builder.Tools` and mount the endpoint. These **managed tools**
receive typed input parsing, validation of encoded output, safe error mapping,
panic recovery, and optional payload-free call telemetry.

The endpoint uses stateless Streamable HTTP, not a client-launched stdio command.
It does not own an HTTP listener or retry handlers. Output validation happens
after execution, so a failed result does not undo business effects.

Follow [First Server](https://arkoren.dev/docs/guides/mcp/first-server.md) for a complete loopback server and a real
client discovery/call path. Before serving private operations or remote traffic,
configure [Securing MCP](https://arkoren.dev/docs/guides/mcp/securing-mcp.md). Then use
[Capability Access](https://arkoren.dev/docs/guides/mcp/availability-and-authorization.md) to distinguish offering an
operation from authorizing its selected records.

## Diagnostic Tools

For inspecting an existing application, use
[Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md) rather than writing domain tools.
Diagnostic factories return ordinary tool definitions; importing a package
exposes nothing. Select registrations explicitly, attach application capture
before startup, and follow the diagnostic privacy constraints.

Use [Application Diagnosis](https://arkoren.dev/docs/guides/diagnostics/application-diagnosis.md) for the
inspection workflow and [Diagnostic Tools](https://arkoren.dev/docs/guides/diagnostics/diagnostic-tools.md)
for individual operation names and result boundaries.

The [Arkoren CLI](https://arkoren.dev/docs/guides/cli/overview.md) is an optional MCP client, not a second
catalog or protocol. MCP resources, prompts, native tools, and other transports
require the [native SDK boundary](https://arkoren.dev/docs/guides/mcp/tools.md#native-extensions); they do not gain
managed protections automatically.

For declarations and adapter settings, see the [core MCP API](https://pkg.go.dev/arkoren.dev/core/mcp)
and [MCP server API](https://pkg.go.dev/arkoren.dev/driver/mcp/mcpserver).

---

Source: https://arkoren.dev/docs/guides/mcp/first-server.md
[HTML documentation](https://arkoren.dev/docs/guides/mcp/first-server)

# First Server

Build a small server that returns a fixed service label through MCP. This
walkthrough takes you from an empty Go project to client discovery and a tool
call over HTTP. It needs no database, model API key, or Arkoren CLI.

Use Go 1.27 or later. For the client checks, install Node.js 22.19 or later and
npm to run the [MCP Inspector](https://modelcontextprotocol.io/docs/tools/inspector).
The Inspector is a client, not a server dependency.

## Project Setup

Create a directory and initialize its module:

```sh
mkdir mcp-service
cd mcp-service
go mod init example.com/mcp-service
```

Install the tool definitions and HTTP adapter:

```sh
go get arkoren.dev/core arkoren.dev/driver/mcp
```

The adapter brings in the official Go MCP SDK. The following steps create
`tools.go` and `main.go` in this directory; append each block to its named file.

## Service Operation

Create `tools.go` with its package and imports:

```go
package main

import (
	"context"

	"arkoren.dev/core/mcp"
)
```

Add an object result and a handler that returns only a public sample label:

```go
type ServiceInfo struct {
	Name string `json:"name"`
}

func describeService(ctx context.Context, input mcp.NoInput) (ServiceInfo, error) {
	return ServiceInfo{Name: "parcel-demo"}, nil
}
```

This is static sample data, not a health check or inspection of the running
process. `mcp.NoInput` represents an empty object, `{}`, not JSON `null`.

## Tool Definition

Publish the handler below it in `tools.go`:

```go
var describe = mcp.NewTool("service.describe", describeService, func(action *mcp.ToolBuilder[mcp.NoInput, ServiceInfo]) {
	action.Description("Return the public name of this sample service.")
	action.ReadOnly(true)
	action.OpenWorld(false)
})
```

The definition supplies the name, schemas, and hints clients discover. It opens
no network connection and does not call the handler. `ReadOnly` describes this
operation; it does not enforce access control.

## Endpoint Registration

Create `main.go` with these imports:

```go
package main

import (
	"log"
	"net/http"

	"arkoren.dev/core/http/middleware/csrf"
	"arkoren.dev/core/http/router"
	"arkoren.dev/driver/mcp/mcpserver"
)
```

Add a constructor that explicitly registers the tool:

```go
func newEndpoint() (*mcpserver.Server, error) {
	return mcpserver.New("parcel-demo", "1.0.0", func(listener *mcpserver.Builder) {
		listener.Tools(describe)
	})
}
```

The name and version identify your server. Construction rejects invalid tool
definitions and duplicate names. The returned endpoint is an `http.Handler`,
not a listening server.

## HTTP Mount

This unauthenticated sample is safe only for the fixed public result shown
above. Keep it on loopback. Before exposing private tools or remote access,
configure [TLS, authentication, Host, and Origin policy](https://arkoren.dev/docs/guides/mcp/securing-mcp.md).

Append `main` to `main.go`:

```go
func main() {
	endpoint, err := newEndpoint()
	if err != nil {
		log.Fatal(err)
	}

	routes := router.New()
	routes.Use(csrf.Middleware())
	routes.Mount("/mcp", endpoint)

	log.Print("serving MCP at http://127.0.0.1:8080/mcp")
	log.Fatal(http.ListenAndServe("127.0.0.1:8080", routes))
}
```

The [CSRF middleware](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md) rejects unsafe
cross-origin browser requests. It is not authentication. Your application owns
the listener and mount; the MCP endpoint has no `Start` or `Stop` method.

Run the project:

```sh
go run .
```

The terminal logs the endpoint URL and stays running. If port 8080 is occupied,
choose another port in both the code and the client commands below.

## Client Verification

Leave the server running and use a second terminal to discover its tools:

```sh
npx @modelcontextprotocol/inspector --cli http://127.0.0.1:8080/mcp \
  --transport http --method tools/list
```

The first invocation downloads the Inspector if needed. The response lists
`service.describe` with its description and object schemas. Discovery does not
execute the handler.

Call the listed operation through the same HTTP transport:

```sh
npx @modelcontextprotocol/inspector --cli http://127.0.0.1:8080/mcp \
  --transport http --method tools/call --tool-name service.describe
```

The result's `structuredContent` contains this object, also returned as text
content:

```json
{
  "name": "parcel-demo"
}
```

A successful list and call confirm protocol discovery and execution, not merely
an open HTTP port. To use an AI application's client instead, configure an MCP
server with this URL and Streamable HTTP transport. A stdio-only client cannot
connect. The client must run locally to reach this loopback address; a hosted
client needs a reachable, secured deployment.

## Connection Failures

If the client cannot connect, check that `go run .` is still running and that
the host, port, and `/mcp` path match. Opening `/mcp` in a browser is not an MCP
test: this stateless endpoint uses POST, and GET returns `405 Method Not Allowed`.

For manual protocol requests, send `Content-Type: application/json` and an
`Accept` header containing both `application/json` and `text/event-stream`.
Prefer an MCP client, which also handles protocol negotiation.

## Application Integration

Stop the example with Ctrl+C. This standalone process does not implement
graceful shutdown. When adding MCP to an existing application, retain its
[HTTP lifecycle and timeouts](https://arkoren.dev/docs/guides/http-networking/server.md); register the MCP
handler without adding a second listener or lifecycle owner.

Continue with [Tools](https://arkoren.dev/docs/guides/mcp/tools.md) for validated arguments, output contracts, and
safe failures. For exact declarations and settings, see the
[core MCP API](https://pkg.go.dev/arkoren.dev/core/mcp) and
[MCP server API](https://pkg.go.dev/arkoren.dev/driver/mcp/mcpserver).

---

Source: https://arkoren.dev/docs/guides/mcp/tools.md
[HTML documentation](https://arkoren.dev/docs/guides/mcp/tools)

# Tools

After your [first MCP connection](https://arkoren.dev/docs/guides/mcp/first-server.md), expose an operation your
application owns. Keep its typed handler independent of MCP transport, then
publish a tool definition and register it at your HTTP composition boundary.
The endpoint does not discover service methods automatically.

The examples use `context` and `encoding/json/jsontext` from the standard
library, plus `arkoren.dev/core/mcp`, `arkoren.dev/foundation/validate`, and
`arkoren.dev/driver/mcp/mcpserver`. Focused fragments omit routine error
propagation; complete functions handle errors.

## Application Operation

A domestic shipping quote needs a parcel weight and an object result. Define
those shapes with explicit JSON names:

```go
type QuoteRequest struct {
	WeightGrams int `json:"weight_grams"`
}

type Quote struct {
	PriceCents int `json:"price_cents"`
}
```

For this sample tariff, parcels through 1000 grams cost 500 cents; heavier
parcels cost 900 cents. Keep the calculation in a typed operation:

```go
func quote(ctx context.Context, input QuoteRequest) (Quote, error) {
	price := 500
	if input.WeightGrams > 1000 {
		price = 900
	}
	return Quote{PriceCents: price}, nil
}
```

This function calculates a quote without booking a shipment or contacting a
carrier. For an operation accessing private records, pass an already-authorized
shared service method instead; [Capability Access](https://arkoren.dev/docs/guides/mcp/availability-and-authorization.md)
explains why transport checks do not replace resource policy.

## Input Rules

Limit this tariff to parcels from 1 through 5000 grams. Build a schema for the
typed input before declaring the tool:

```go
var input = validate.Struct(func(object *validate.StructBuilder[QuoteRequest]) {
	object.Field("WeightGrams", func(weight *validate.IntBuilder[int]) {
		weight.Min(1)
		weight.Max(5000)
	})
})
```

`Field` selects the Go field `WeightGrams`; clients send `weight_grams`.
An omitted integer decodes as zero and fails this minimum. This input needs no
normalization. If your schema includes normalizers, core applies them during
input parsing before calling the handler.

Publish the handler with that schema:

```go
var shipping = mcp.NewTool("shipping.quote", quote, func(action *mcp.ToolBuilder[QuoteRequest, Quote]) {
	action.Description("Quote domestic shipping in cents for a parcel of 1–5000 grams.")
	action.Input(input)
	action.ReadOnly(true)
	action.OpenWorld(false)
})
```

`NewTool` does not execute the handler or its availability policy. Check `Err()`
when assembling definitions; server construction also checks it. Names contain
1–128 ASCII letters, digits, underscores, dots, or hyphens. Definitions are
immutable and fit together in a heterogeneous `[]mcp.Tool`.

For an available tool, core strictly decodes one object and parses the input
schema once. Nonobjects, malformed JSON, duplicate members, unknown struct
fields, and rule violations prevent handler execution. Nil arguments mean `{}`.
Cancellation is checked before policy evaluation and again before the handler.

Decoding and rule violations wrap `mcp.ErrInvalidInput`. A dependency error from
an input `Check` instead wraps `mcp.ErrInputSchema` and preserves its original
cause. Use `validate.Fail` for caller-correctable Check violations, not outages.
The [typed parser](https://arkoren.dev/docs/guides/validation/overview.md) remains authoritative: advertised
JSON Schema cannot express every normalization or custom check.

## Output Contract

Both argument and result schemas must have an object root. Wrap a scalar or
list in a result struct. Schemas are inferred when omitted; for this quote,
explicitly require a positive price in the emitted JSON:

```go
var output = validate.Struct(func(object *validate.StructBuilder[Quote]) {
	object.Field("PriceCents", func(price *validate.IntBuilder[int]) {
		price.Min(1)
	})
})
```

Add `action.Output(output)` inside the tool's construction callback. The adapter
checks the encoded result against the published schema. It does not run output
`Parse`, trim values, insert defaults, or execute custom `Check` callbacks. Only
JSON-expressible output rules participate. A successful 1200-gram quote returns
the same object as `structuredContent` and text content:

```json
{
  "price_cents": 900
}
```

Output checks happen after execution. They do not make effects transactional,
and the adapter never retries the handler. Put transactions and business
idempotency in your service. Pass context to dependencies; cancellation is
cooperative and does not roll back completed effects.

Schema bytes are copied during publication and when returned by `Describe()`.
Editing a report or retaining a construction draft cannot mutate the definition.
Handler captures and schema implementations remain shared and must support
concurrent calls.

### Schema Limits

Managed output uses `validate.FromJSONSchemaStrict`, a bounded validator for
draft-07 and 2020-12. It preserves missing members, `null`, and zero as different
states. Defaults and formats are annotations, not transformations or format
assertions. Local references are supported; external references fail without
network access. Unknown assertion keywords, unsupported types, and unsupported
dialects fail registration rather than receiving a permissive fallback;
recognized `x-*` extensions remain annotations.

Numeric comparisons use exact decimal values: `9007199254740993` exceeds a
maximum of `9007199254740992`, and `1e1000` is valid within the resource bounds.
Precision already lost in a Go `float64` cannot be recovered. Integer fields and
`jsontext.Value` retain numeric precision; numbers decoded into `any` follow
JSON v2's `float64` semantics.

Each schema or emitted document is limited to 8 MiB and 256 container levels.
Numeric coefficients allow 4096 digits, explicit exponents range from -4096
through 4096, and coefficient digits plus absolute exponents have a cumulative
64 Mi-digit budget per decoded document. Equality/uniqueness structural keys
have a cumulative 64 MiB encoding budget per validation. These are data/work
bounds, not exact Go heap limits. Additional node, work, reference, and pointer
limits are specified by
[EncodedSchema.Validate](https://pkg.go.dev/arkoren.dev/foundation/validate#EncodedSchema.Validate).

A validator resource refusal or cancellation produces `IsError=true`, message
`tool failed`, and failure code `internal`, not `invalid_output`. This neither
proves invalid business output nor undoes effects. See
[FromJSONSchemaStrict](https://pkg.go.dev/arkoren.dev/foundation/validate#FromJSONSchemaStrict)
for vocabulary and dialect details.

The pinned SDK separately probes numeric literals in both advertised schemas.
For example, a schema containing `"default":1e1000` exceeds that probe's decoder
range and fails construction. The probe does not replace schema bytes or impose
a `float64` range on emitted raw JSON. Client decoders can still round or reject
numbers; see [CLI number precision](https://arkoren.dev/docs/guides/cli/calling-tools.md#number-precision).

## Safe Failures

Annotations describe behavior, not enforcement. `ReadOnly` does not authorize
access, `Idempotent` does not deduplicate calls, and `OpenWorld(false)` does not
restrict networking. Defaults are read-only false, destructive true,
idempotent false, and open-world true.

When a handler has an approved public failure explanation, return a
`mcp.ToolError`. For example, a carrier-backed implementation can decline a
quote without exposing its upstream error:

```go
return Quote{}, mcp.ToolError("shipping quotes are temporarily unavailable")
```

The adapter returns a failed tool result with that exact approved message,
without wrapping context. Plain ToolError text is capped at 4096 UTF-8 bytes,
truncated at a code-point boundary; empty text stays empty and malformed UTF-8
becomes `tool failed`. Core retains the original error for local inspection.
Never mark credentials, private identifiers, or sensitive existence claims safe.

For approved structured diagnostics, `mcp.ToolFailure` places a code, message,
and optional object details under result `_meta["arkoren.dev/failure"]`, not
successful structured content. The plain text cap is not a general size limit
for structured failures or successful output.

Framework classifications override nested safe markers. Invalid input becomes
`invalid tool input`; encoding failures and output schema violations become
`invalid tool output`. Operational input Check errors, validator resource or
cancellation failures, unmarked handler errors, and recovered panics become
`tool failed`. Availability has its own
[failure behavior](https://arkoren.dev/docs/guides/mcp/availability-and-authorization.md#evaluation-failures).

Input violations may expose validation pointers and rendered schema messages,
but not violation parameters, input values, or decoder diagnostics. Keep schema
messages free of secrets. Original errors and panic values are logged locally;
protect log storage and access even though client responses are redacted.

## Endpoint Registration

Assemble the selected tools at application composition time:

```go
endpoint, err := mcpserver.New("shipping", "1.0.0", func(listener *mcpserver.Builder) {
	listener.Tools(shipping)
})
```

Handle `err` before mounting `endpoint` at `/mcp` in your HTTP service.
Construction opens no port and rejects invalid declarations, duplicate names,
and unsupported output schemas before registering any tool with the SDK.
`Tools` appends registrations and copies supplied slices. Importing a package or
calling its `MCPTools(prefix)` factory does not expose anything by itself.

Follow the [HTTP mount](https://arkoren.dev/docs/guides/mcp/first-server.md#http-mount) for the public local sample,
or [authenticated mounting](https://arkoren.dev/docs/guides/mcp/securing-mcp.md#typed-authentication) for private
tools. Authentication, Host/Origin checks, request deadlines, and listener
timeouts remain application responsibilities.

The MCP endpoint has no lifecycle resources or `Start`/`Stop` methods. Retain
your existing HTTP listener's startup, serving runner, and cleanup ownership.
With Arkoren composition, register the listener's `Start`, `Serve`, and `Stop`,
not the endpoint; [HTTP Server](https://arkoren.dev/docs/guides/http-networking/server.md) covers that wiring.
There is no adapter tool-duration setting, and a write timeout alone does not
stop application work.

## Transport Behavior

The adapter uses stateless Streamable HTTP with JSON responses and the SDK's
default 4 MiB request-body limit. It issues no persistent session ID, stores no
events for resumable delivery, and returns 405 for GET/DELETE and 413 for
oversized bodies. It does not serve legacy HTTP+SSE or managed server-initiated
requests.

The module pins Go MCP SDK v1.8.0, supporting protocol versions 2026-07-28,
2025-11-25, 2025-06-18, 2025-03-26, and 2024-11-05. Adapter tests exercise
2026-07-28 and 2025-06-18 clients. Request-cancellation propagation is enabled,
but the SDK applies it only to 2026-07-28 and later: an older client's disconnect
does not necessarily cancel its tool. Outer request context values remain
available to managed handlers.

The SDK's localhost DNS-rebinding protection is enabled. It is not a public
Host allowlist or an Origin policy. The adapter offers no custom body-limit,
OAuth, or execution-timeout setting; configure these at the appropriate
[outer HTTP boundary](https://arkoren.dev/docs/guides/mcp/securing-mcp.md).

## Focused Tests

After verifying an actual MCP connection, test operation behavior directly.
Inside a Go test with `t *testing.T`, call the published `shipping` definition:

```go
result, err := shipping.Call(context.Background(), jsontext.Value(`{"weight_grams":1200}`))
if err != nil {
	t.Fatal(err)
}
if string(result) != `{"price_cents":900}` {
	t.Fatalf("result = %s", result)
}
```

This checks typed decoding, input parsing, and the handler. Add tests for invalid
weights, denied capabilities, cancellation, and resource policy. Direct core
calls preserve error identities and propagate panics; they do not exercise
adapter recovery, redaction, or advertised-output validation. Use an SDK client
against the HTTP handler for those boundaries, as in the
[adapter HTTP tests](https://github.com/arkoren/arkoren/blob/main/driver/mcp/mcpserver/server_test.go).

## Call Telemetry

With an existing telemetry service named `signals`, add
`listener.Telemetry(signals)` inside endpoint construction. Omitted or nil
telemetry follows `telemetry.Default()` per call. Managed tools emit
`mcp call_tool` spans, call counts, and durations in seconds labeled by registered
tool name and success/error outcome. They capture no arguments, output, raw
errors, or model-inference attributes.

Protected error logs remain separate; `listener.Logger(logger)` selects an
existing `*slog.Logger`. Configure [telemetry export](https://arkoren.dev/docs/guides/observability/telemetry.md)
and [diagnostic capture](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md) separately.

## CLI Presentation

Optional CLI metadata makes a tool easier to call without renaming its protocol
operation or granting access. Add these settings to the `shipping` declaration:

```go
action.CLI("shipping", "quote")
action.Positional("weight_grams")
action.Example("Quote a 1200-gram parcel", `{"weight_grams":1200}`)
```

The adapter advertises this under tool `_meta["arkoren.dev/cli"]` in
`tools/list`, the only catalog. Examples are checked for typed JSON compatibility
during construction, not executed as calls or parsed through contextual rules.
The [Arkoren CLI installation](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren) and
[calling guide](https://arkoren.dev/docs/guides/cli/calling-tools.md) explain client use. Reports are JSON;
follow output is NDJSON, as detailed in
[Output Formats](https://arkoren.dev/docs/guides/cli/reference.md#output-formats). See
[Failure Diagnostics](https://arkoren.dev/docs/guides/cli/reference.md#failure-diagnostics) for failure records.

The adapter serializes schemas, CLI metadata, approved failure details, and raw
JSON output without routing their number tokens through `float64`. This does
not guarantee precision after a client decodes those records generically.

## Native Extensions

`endpoint.Server()` exposes the native SDK server for resources, prompts,
native tools, or another SDK transport such as stdio. Arkoren has no typed
resources/prompts facade or managed stdio adapter.

Native tools remain visible in discovery but bypass managed availability,
typed validation, safe-error mapping, recovery, telemetry, and CLI metadata.
Do not replace managed registrations or their list middleware through this
accessor: doing so can bypass guards or leave metadata stale.

For exact declarations and transport settings, see the
[core MCP API](https://pkg.go.dev/arkoren.dev/core/mcp) and
[MCP server API](https://pkg.go.dev/arkoren.dev/driver/mcp/mcpserver).

---

Source: https://arkoren.dev/docs/guides/mcp/securing-mcp.md
[HTML documentation](https://arkoren.dev/docs/guides/mcp/securing-mcp)

# Securing MCP

A private MCP tool needs an authenticated caller before protocol dispatch.
`mcpserver` verifies no credentials and issues no tokens: keep admission in your
HTTP service. Tool availability decides which operations to offer after admission;
shared application services still authorize the records those operations use.

The [first server](https://arkoren.dev/docs/guides/mcp/first-server.md) intentionally exposes only a fixed public
label on loopback. Do not replace that result with private data without adding
the boundaries below.

## Transport Boundary

Keep private tools on a private network or loopback listener, and use TLS whenever
credentials cross a network. Before remote deployment, enforce allowed external
Host values and validate Origin using your gateway or outer HTTP middleware.
Bearer authentication does not replace Origin validation.

The pinned SDK enables localhost DNS-rebinding protection: requests arriving
through a loopback local address with a non-localhost Host receive 403. This is
not a public-host allowlist. The adapter does not enable the SDK's optional
cross-origin protection, so your outer Origin policy remains necessary.

With `router` imported from `arkoren.dev/core/http/router`, adapt your existing
`boundary func(http.Handler) http.Handler` before authentication and mounting:

```go
routes := router.New()
routes.Use(router.HTTPMiddleware(boundary))
```

Here `boundary` is application-owned middleware enforcing the deployment's
Host and Origin policy, not an Arkoren API. Behind a proxy, establish which
component validates the external values and trust forwarded headers only from
that proxy. A private proxy address alone does not make client headers trusted.

## Typed Authentication

Use your existing [bearer verifier](https://arkoren.dev/docs/guides/authentication/api-tokens.md) to resolve
the application's user, not merely decode claims. Assume `User` is your identity
type and `verifyAccessToken` has signature
`func(context.Context, string) (User, error)`.

Import `arkoren.dev/core/http/auth` and add authentication to the router above
before mounting the checked `endpoint` from [tool registration](https://arkoren.dev/docs/guides/mcp/tools.md#endpoint-registration):

```go
routes.Use(auth.Middleware(auth.Bearer(verifyAccessToken)))
routes.Use(auth.Require[User]())
routes.Mount("/mcp", endpoint)
```

`auth.Middleware` stores a verified `User`, but permits anonymous requests when
credentials are absent. `auth.Require[User]()` rejects those requests. Invalid
credentials fail authentication. Configure middleware before `Mount` and serve
the router through your existing HTTP lifecycle.

The mount and adapter preserve request context for policies and handlers.
`auth.FromContext[User]` must use the exact concrete type stored at admission:
`User` and `*User` occupy different slots. Mismatched lookups fail closed.

If existing `net/http` middleware already verifies callers, retain it. Reject
anonymous requests and inject the verified identity with `auth.NewContext`
before calling the endpoint rather than adding another MCP-specific bearer
parser.

## Endpoint Admission

Authentication need not grant everyone access to MCP. If your application has
an endpoint-wide rule, import `arkoren.dev/core/http/auth/gate` and insert it
after authentication but before the mount:

```go
routes.Use(gate.Require(canUseMCP))
```

`canUseMCP` is your existing `func(context.Context, User) (bool, error)` rule.
This admits a caller to the whole endpoint; it does not authorize individual
tools or records. [Capability Access](https://arkoren.dev/docs/guides/mcp/availability-and-authorization.md) handles
those separate decisions.

## Browser Access

For a trusted browser application on another origin, configure
[CORS](https://arkoren.dev/docs/guides/browser-security/cors.md) before authentication and mounting. With
`cors` from `arkoren.dev/core/http/middleware/cors` and standard `net/http`:

```go
routes.Use(cors.Middleware(func(policy *cors.Builder) {
	policy.AllowedOrigins("https://app.example.com")
	policy.AllowedMethods(http.MethodPost)
	policy.AllowedHeaders("Accept", "Content-Type", "Authorization", "MCP-Protocol-Version", "Mcp-Method", "Mcp-Name")
}))
```

This is inserted before the authentication block, not after the completed
mount. CORS answers preflights and lets the trusted browser read responses;
it does not reject requests from disallowed origins. Keep the outer Origin
policy. The `Mcp-Method` and `Mcp-Name` headers support newer protocol clients.

A bearer-only endpoint does not rely on browser-attached credentials, so it
does not need cookie CSRF protection. If the endpoint also accepts session
cookies, add [CSRF protection](https://arkoren.dev/docs/guides/browser-security/csrf-protection.md) before
authentication. Import `arkoren.dev/core/http/middleware/csrf` and trust only
the browser origin that needs access:

```go
routes.Use(csrf.Middleware(func(protection *csrf.Builder) {
	protection.TrustedOrigins("https://app.example.com")
}))
```

Cookies can accompany forged browser requests, so CORS alone does not prevent
their effects. If you intentionally permit credentialed cross-origin browser
requests, also configure `policy.AllowCredentials(true)` in the CORS callback
and an appropriate cookie policy; never combine credentials with a wildcard
origin.

## Execution Bounds

Retain listener header/read/write/idle timeouts and set operation deadlines in
outer middleware or application services. The adapter has no tool-duration
option. Pass request context into dependencies: a write timeout or client
disconnect is not a universal guarantee that work stops, and cancellation
does not undo completed effects.

The [transport contract](https://arkoren.dev/docs/guides/mcp/tools.md#transport-behavior) specifies the 4 MiB body
limit and protocol-dependent disconnect cancellation. Configure any stricter
body limit at your gateway or HTTP boundary. Managed error redaction protects
tool responses, not arbitrary admission responses or application logs.

## Remote OAuth

The bearer example assumes a client already has a token. A remote MCP OAuth
deployment also needs protected resource metadata, bearer challenges, and an
authorization server issuing tokens for this resource. A login flow or a
token verifier alone is not that complete integration.

`oauthauth.Client.Verify` from `arkoren.dev/driver/oauth/oauthauth` verifies
OIDC bearer access tokens with configured audiences and returns
`oauthauth.Identity`. Map it to your application `User` if your policies expect
that type; the [OIDC bearer guide](https://arkoren.dev/docs/guides/authentication/oidc-bearer.md) owns verifier
configuration and lifecycle. The driver is neither a token issuer nor a complete
MCP authorization server.

For native integration, the pinned SDK provides `RequireBearerToken` and
`ProtectedResourceMetadataHandler` in
`github.com/modelcontextprotocol/go-sdk/auth`. These are outer HTTP components,
not `mcpserver.Builder` settings. Configure a real token verifier, accurate public
resource metadata, and authorization-server URLs. Preserve required
`WWW-Authenticate` challenges rather than replacing them with ordinary JSON
authentication errors.

SDK `TokenInfo` is separate from Arkoren's typed user context. Before managed
dispatch, read it with `TokenInfoFromContext`, resolve your application user,
and inject that user with `core/http/auth.NewContext`. Require the exact identity
type your tool policies use. The SDK requires nonzero token expiration by
default. Enforce endpoint scopes with `RequireBearerTokenOptions.Scopes`;
advertising supported scopes in metadata does not enforce them.

The SDK's bearer middleware returns verifier error text in HTTP failure
responses. Return only approved public errors from that verifier and retain
private diagnostics in protected local logs. Managed ToolError redaction does
not apply to this earlier boundary. Resource metadata is public configuration,
not a place for credentials; the SDK metadata handler does not validate its
fields.

Neither endpoint scopes nor metadata authorize a particular shipment or invoice.
Keep resource checks in shared services. The [CLI connection guide](https://arkoren.dev/docs/guides/cli/connecting.md)
shows how to supply an existing token; the CLI does not log in or refresh it.

For endpoint settings, see the [MCP server API](https://pkg.go.dev/arkoren.dev/driver/mcp/mcpserver);
for tool policy declarations, see the [core MCP API](https://pkg.go.dev/arkoren.dev/core/mcp).

---

Source: https://arkoren.dev/docs/guides/mcp/availability-and-authorization.md
[HTML documentation](https://arkoren.dev/docs/guides/mcp/availability-and-authorization)

# Capability Access

A plan may unlock shipment tracking without granting access to every shipment.
**Availability** decides whether a caller is offered an operation.
**Resource authorization** decides whether that caller may access the selected
record. Use a tool availability policy for the first decision and a shared
application service for the second.

Start with [typed authentication](https://arkoren.dev/docs/guides/mcp/securing-mcp.md#typed-authentication) already
storing your application user in context. Admission to `/mcp` is not permission
to use every tool or read every record. Reuse your existing
[Gate rules](https://arkoren.dev/docs/guides/authorization/overview.md), not a separate MCP permission model.

The examples import standard `context`, `arkoren.dev/core/mcp`,
`arkoren.dev/core/http/auth`, and `arkoren.dev/core/http/auth/gate`.

## Capability Rules

Assume these are your application's identity and tracking request/result types:

```go
type User struct {
	ID    string
	Paid  bool
	Admin bool
}

type ReadShipment struct {
	ID string `json:"id"`
}

type Shipment struct {
	ID    string `json:"id"`
	State string `json:"state"`
}
```

An application-wide rule can permit tracking for paid accounts and administrators:

```go
func canTrack(ctx context.Context, user User) (bool, error) {
	return user.Paid || user.Admin, nil
}
```

Adapt that shared rule to the context-only callback MCP needs:

```go
func trackingAvailable(ctx context.Context) (bool, error) {
	user, ok := auth.FromContext[User](ctx)
	if !ok {
		return false, nil
	}
	return canTrack(ctx, user)
}
```

Missing identity denies availability here. The lookup must use the same
concrete `User` type as admission; `User` and `*User` are different slots.
If a real plan lookup fails, return its error rather than turning an outage
into an ordinary denial. Policies must be observational and safe for concurrent
use: listing tools must not change business data. Pass context into backend
lookups.

## Resource Checks

The policy above receives no arguments, so it cannot authorize `input.ID`.
Place that check in the shared operation before loading the shipment. This
sample service accepts your existing loader and resource-rule factory:

```go
type ShipmentService struct {
	Load    func(context.Context, ReadShipment) (Shipment, error)
	CanRead func(string) func(context.Context, User) (bool, error)
}
```

Construct it with non-nil application collaborators. `CanRead(id)` must enforce
your tenant, ownership, or other resource rules; an ID alone is not evidence of
ownership. Its shared method authorizes before loading:

```go
func (service *ShipmentService) Read(ctx context.Context, input ReadShipment) (Shipment, error) {
	if err := gate.Authorize(ctx, service.CanRead(input.ID)); err != nil {
		return Shipment{}, err
	}
	return service.Load(ctx, input)
}
```

Gate rejects missing identity, denies a false rule, and propagates a rule error
before `Load` runs. Use this same method from HTTP and MCP; a check only in an
MCP wrapper leaves other callers able to bypass it. If plan entitlement is a
business requirement on every transport, enforce the shared capability rule
in that service path too. Discovery filtering is not a substitute.

With a configured `shipments *ShipmentService`, publish the operation:

```go
read := mcp.NewTool("shipments.read", shipments.Read, func(action *mcp.ToolBuilder[ReadShipment, Shipment]) {
	action.Description("Read the state of a shipment the caller may access.")
	action.ReadOnly(true)
	action.Available(trackingAvailable)
})
```

Availability controls discovery and execution; `shipments.Read` still owns the
selected-record decision. Add ID validation using the
[input-schema pattern](https://arkoren.dev/docs/guides/mcp/tools.md#input-rules). `ReadOnly` remains a hint, not a guard.

Omitting `Available` permits every admitted caller. Explicit `Available(nil)`
instead records a sticky declaration error, even if a later callback provides
a valid policy.

## Guarded Copies

Package factories such as `MCPTools(prefix)` can return definitions with their
own policies. Select only the tools your endpoint needs, then add application
availability without mutating the owner's definition. With `tool` as one
selected `mcp.Tool`:

```go
guarded := tool.WithAvailability(trackingAvailable)
```

The copy requires the original policy and then the added policy. Denial
short-circuits; neither policy runs during construction. The original tool
stays unchanged, and nil policy is invalid. Captured state remains shared.

This is still a capability gate, not a resource policy. For reusable libraries
and [arks](https://arkoren.dev/docs/guides/core-concepts/arks.md), keep application user types, plan names,
and permission rules application-owned. Pass a shared authorized operation or
a domain authorization callback into your own integration rather than teaching
the library your authentication model.

## Request Evaluation

The adapter evaluates availability for every managed tool during `tools/list`.
It evaluates availability again on every call, before argument decoding and
validation. Guessing a hidden name cannot bypass this check, and a prior list
does not grant a durable permission. Register one static catalog; do not cache
per-user servers to implement discovery filtering.

Discovery returns the complete filtered catalog without pagination or a next
cursor. A nonempty cursor is rejected. Native SDK tools remain visible without
managed availability; see [Native Extensions](https://arkoren.dev/docs/guides/mcp/tools.md#native-extensions).

`Tool.Available(ctx)` can check policy directly without decoding or invoking
the handler. It validates the declaration and cancellation first, checks
cancellation again after successful policy evaluation, and caches nothing.
Core also checks cancellation after input parsing, before handler execution.

## Evaluation Failures

A `false, nil` policy result is an ordinary denial. Direct `Tool.Call` returns
`mcp.ErrUnavailable`; the endpoint returns a failed tool result with message
`tool unavailable`. A policy error preserves its cause under
`mcp.ErrAvailability` in core, but becomes `tool failed` on the wire, even if it
wraps `mcp.ToolError` or `mcp.ToolFailure`.

A policy error or panic during listing fails the entire request with
`tool listing failed`, not a partial catalog. Core propagates policy and handler
panics; the adapter recovers them. Original errors and panic values stay in
protected local logs. [Safe Failures](https://arkoren.dev/docs/guides/mcp/tools.md#safe-failures) explains the other
redaction boundaries.

Test that denied callers never reach decoding or the handler, and that a
permitted capability still cannot read another tenant's resource. Direct core
tests cover policy/service behavior; SDK HTTP tests cover discovery, identity
propagation, and safe responses.

## Shared LLM Policy

If your application also runs an LLM agent, reuse the typed handler, input
schema, and context policy in a separate `llm.NewTool` declaration. These are
shared application inputs, not converters between tool types. MCP lists
availability per request; LLM discovery is fixed once per run and execution
has its own checks. See [Offering Capabilities](https://arkoren.dev/docs/guides/llm/defining-tools.md#offering-capabilities)
for that boundary.

For exact declaration and policy contracts, see the
[core MCP API](https://pkg.go.dev/arkoren.dev/core/mcp) and
[MCP server API](https://pkg.go.dev/arkoren.dev/driver/mcp/mcpserver).

---

Source: https://arkoren.dev/docs/guides/agent-skills/working-with-agents.md
[HTML documentation](https://arkoren.dev/docs/guides/agent-skills/working-with-agents)

# Using Skills

When a coding agent works on your application, give it Arkoren's integration
guides rather than relying on its training data. The `arkoren` **skill** is a
folder containing an entry file, `SKILL.md`, and reference pages that the agent
reads as needed. It includes the same authored guides as the website, packaged
as a fixed snapshot in the Arkoren CLI.

You can install the skill in an existing application without generating a new
project or changing its Go dependencies. For adopting Arkoren packages while
keeping your current layout, follow
[Existing Applications](https://arkoren.dev/docs/guides/first-steps/installation.md#existing-applications).
If your agent reads from the web instead, use the [online indexes](#reading-online).

The skill supplies documentation, not backend tooling. Installing or loading it
does not connect to your service, supply credentials, or authorize file edits,
dependency changes, migrations, or remote operations. Keep those actions within
your agent's configured permissions and your explicit task. Runtime access has
separate setup through the [CLI connection](https://arkoren.dev/docs/guides/cli/connecting.md) or an
[MCP client](https://arkoren.dev/docs/guides/mcp/first-server.md).

## Installing Skills

Install the [Arkoren CLI](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren), then choose the
project-local directory your agent reads. Run the following commands from your
application's root, not from a parent workspace or your home directory.

For [Codex](https://developers.openai.com/codex/skills/), the CLI's default
`.agents/skills` matches its repository skill directory. Preview the destination
and planned changes before writing:

```sh
arkoren skills install --dry-run
```

The JSON report names the destination ending in `.agents/skills/arkoren`, with
planned write, removal, and conflict counts. The preview creates no files or
directories. Add `--verbose` when you need individual paths. If the destination
is the one you intended and there are no conflicts, install the bundle:

```sh
arkoren skills install
```

The installation contains `SKILL.md`, the reference pages, and
`.arkoren-skill.json`, which records managed file hashes. A successful report
includes `completed: true`. This operation works offline: it does not download
documentation, read endpoint configuration, install Go modules, or edit
`AGENTS.md`.

For [Claude Code](https://code.claude.com/docs/en/skills), use its project skill
directory instead. Preview and then apply this alternative installation:

```sh
arkoren skills install --dir .claude/skills --dry-run
```

After reviewing the preview, run the same command without `--dry-run`. The skill
lands at `.claude/skills/arkoren/SKILL.md`. Choose one location for the client you
use; these commands are alternatives, not required duplicate installations.

`--dir` always names the **parent** skills directory; Arkoren appends `arkoren`.
Relative paths start at the current working directory. The CLI does not search
parent directories for a project or an installed skill. Use the same `--dir`
with later status and update commands. For other clients, check their documented
local skill discovery rules rather than assuming either location is recognized.

Commit the chosen skill directory, including its manifest, if your team should
share the snapshot. Keep application-specific instructions in your project's
own instruction files, outside the managed skill. If you already used
[`arkoren new`](https://arkoren.dev/docs/guides/first-steps/creating-a-project.md), check its generated skill
before installing another copy; new projects include agent files by default.

## Loading Guidance

Installation and discovery are separate. Codex can select a skill when a task
matches its description; use `/skills` or mention `$arkoren` to select it
explicitly. In Claude Code, invoke `/arkoren`, or ask for a task that matches the
skill's description. Client policy can restrict skill loading. Confirm your
agent can read the installed `SKILL.md` and one linked guide before relying on
automatic selection; if it cannot find the skill, check its location and restart
the client if needed.

For example, if you want advice on adding a cache to an existing service, give
the agent a bounded reading task before asking for changes:

```text
Load the arkoren skill and read the cache overview and driver guide.
Our service already has its own layout. Compare a process-local cache with
Redis for this service. Explain the trade-offs without editing files.
```

This asks the agent to use relevant guidance without turning the request into
permission to install Redis, change dependencies, or rewrite your application.
Name the provider when you have already chosen one so the agent reads its
specific setup and limitations.

The entry file contains authored usage rules and a generated guide index. The
index points to guides; it is not an API inventory or the source of the rules.
Ask the agent to read the entry file and only the reference pages needed for
the task, rather than loading the whole guide tree into its context.

For an exact API, run `go doc` from your application's module directory. For
example, if that module already depends on core:

```sh
go doc arkoren.dev/core/service/cache.New
```

Go selects the declaration from your application's module graph and active
workspace, not from the installed skill's revision. Check `go.mod` and the
workspace when the selected source is uncertain. This lookup can work offline
once the dependency source is available locally; installing the skill does not
install that source. For online lookup, select the matching package and version
on [pkg.go.dev](https://pkg.go.dev/arkoren.dev/core/service/cache).

## Updating Skills

Check the installed snapshot before changing it:

```sh
arkoren skills status --verbose
```

The report shows bundled and installed revisions and lists modified managed
files, including files that are missing or unreadable. A missing installation
has state `not installed`; a directory without an ownership manifest can be
reported as `unmanaged`. Status does not write files or fail merely because
managed files changed.

A revision is a SHA-256 content identifier, not a Go module version or a
compatibility guarantee. Before updating, compare the documentation's
assumptions with the dependencies selected by your application. Pin the CLI
version if your team needs a reproducible tool and documentation bundle.

To obtain a newer snapshot, upgrade the CLI using your chosen
[installation method](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren). Then preview the
bundle it now carries:

```sh
arkoren skills update --dry-run
```

Review the planned writes, removals, and conflicts. When the plan is acceptable,
apply it:

```sh
arkoren skills update
```

Neither command downloads documentation or looks up the latest release.
`update` means “match this CLI's bundle”; running an older CLI can therefore
replace your documentation with an older snapshot. For a custom installation,
include its parent directory, such as `--dir .claude/skills`, in each command.

## Preserving Changes

`install` accepts a missing or empty destination, or an unchanged managed
installation matching the bundle. Repeating a matching install does not rewrite
files. If another managed revision is present, use `update`. If another
installer owns the directory, keep using that installer for maintenance;
`arkoren skills update` requires Arkoren's ownership manifest. To switch
installers, preserve local work and move the old directory aside before
installing.

During an update, Arkoren replaces unchanged managed files, adds new files, and
removes unchanged obsolete managed files. Unrelated files remain. Local edits
or deletions, unsafe symlinks, and collisions at new paths block the update
before writes. There is no force-overwrite option. A conflicting dry run prints
its report and exits 2; a conflicting non-preview operation reports an error on
stderr and exits 2. Check exit status as well as the report. The
[command reference](https://arkoren.dev/docs/guides/cli/reference.md) describes output and option details.

If local changes block an update, preserve them elsewhere and restore managed
files to their installed contents before retrying. Keep lasting project rules
in your own instruction files instead of modifying managed reference pages.
Neither installation nor updates rewrite `AGENTS.md`.

Run only one installer at a time and avoid editing managed files during updates.
Individual replacements are atomic, but the full operation is not a
transaction: an interruption or I/O failure can leave partial changes. The
manifest is written last. After a failure, run `status`, inspect the affected
paths, and restore a known-good installation from version control. Alternatively,
preserve local work, move the partial directory aside, and install again.

## Removing Skills

There is no `arkoren skills uninstall` or `remove` command. To uninstall a
CLI-managed copy, first run `status --verbose` with the same `--dir` used at
installation and review the reported directory. Preserve any local edits or
unrelated files inside it, then remove only that installation's `arkoren`
directory, including its manifest. Do not remove the parent `.agents/skills`
or `.claude/skills` directory, which can contain other skills.

Remove any pointers to that copy from your own project instructions. If the
directory is tracked, review the deletions before committing them. Refresh or
restart your agent and confirm it no longer discovers that copy. A second local
or personal copy can still be discoverable, and removing files does not erase
guidance already loaded into an ongoing conversation; start a fresh session
when you need to verify behavior without it.

## Reading Online

An agent with web access can read the current website without installing a
skill. Start with an `llms.txt` index, which links individual Markdown pages with
descriptions, then fetch only the pages your task needs:

| Index | Contents |
| --- | --- |
| `https://arkoren.dev/llms.txt` | All guides and the module API catalog. |
| `https://arkoren.dev/docs/guides/llms.txt` | All guides, grouped by section. |
| `https://arkoren.dev/docs/guides/agent-skills/llms.txt` | This group's guides; replace `agent-skills` with another group to narrow the topic. |
| `https://arkoren.dev/docs/api/llms.txt` | Module-level pkg.go.dev links, not copied API declarations. |

Each guide also has raw Markdown at its page URL with `.md` appended. For
example, this guide is available at
[Using Skills in Markdown](https://arkoren.dev/docs/guides/agent-skills/working-with-agents.md).
HTML guide pages advertise that URL with
`<link rel="alternate" type="text/markdown">`; the shared layout points to the
root index with `rel="describedby"`. Markdown exports link back to the HTML
documentation so an agent can cite the page you read.

The website can be newer than an installed snapshot. Its module catalog does
not verify that each module is published or indexed: if pkg.go.dev returns not
found, use locally available dependency source with `go doc`. Online discovery
does not replace checking your application's selected version.

Use `https://arkoren.dev/llms-full.txt` or
`https://arkoren.dev/docs/guides/llms-full.txt` only for deliberate bulk indexing
or large-context reading. Both contain guides only, not API declarations. Each
archive reports its page count, documentation size, and an approximate token
count based on characters divided by four, not a model-specific tokenizer.
Loading the whole archive can exhaust context and increase processing cost.
With JavaScript enabled, ordinary website archive clicks offer a warning and a
route back to the index; direct text requests have no confirmation step.

These indexes make documentation available to agents that read them; they do
not guarantee discovery or recommendation by an assistant or search engine.
Whether you use local files or web pages, keep reading scoped to the task and
approve consequential actions separately.

---

Source: https://arkoren.dev/docs/guides/observability/telemetry.md
[HTML documentation](https://arkoren.dev/docs/guides/observability/telemetry)

# Telemetry

Use telemetry to connect a slow request to the operations it called and to
measure recurring behavior across requests. A **span** describes one operation;
related spans form a **trace**. **Metrics** aggregate measurements over time.
Context-aware logs can carry the same trace identity.

`telemetry.Telemetry` is the shared handle for these signals. Its driver decides
where they go; choose and configure one on [Drivers](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md).
Telemetry can be sampled, aggregated, or dropped. It is not an audit ledger or
proof that a business action completed; use [auditing with events](https://arkoren.dev/docs/guides/events/auditing.md)
when individual actions need a durable record.

## Shared Handle

With your selected `driver`, give the handle to the application. Import
`application` from `arkoren.dev/core/application` and `telemetry` from
`arkoren.dev/core/service/telemetry`:

```go
app := application.New(func(app *application.Builder) {
	app.Telemetry(telemetry.New(driver))
})
```

Construction performs no I/O. After composition succeeds, the application starts
telemetry before user startup callbacks and stops it after cleanup and the final
`app stopped` log. It supplies the service name, version, and environment as the
telemetry resource. Without configured telemetry or diagnostic capture,
`app.Telemetry()` remains a nonnil, unstarted no-op handle.

Pass that handle explicitly when constructing an instrumented service. With an
existing database driver `backend` and `database` imported from
`arkoren.dev/core/service/database`:

```go
db := database.New(backend, func(connection *database.Builder) {
	connection.Telemetry(app.Telemetry())
})
```

The [component catalog](https://arkoren.dev/docs/guides/observability/instrumented-components.md) identifies each builder and
its observations. An omitted or nil handle follows `telemetry.Default()` at
operation time. To publish the application's logger and telemetry together:

```go
app.SetDefaults()
```

This changes process-wide state; prefer explicit handles when applications or
tests share a process. `telemetry.SetDefault(nil)` restores the unstarted no-op
default. Publishing a handle does not start it.

Focused fragments keep error results visible and omit routine propagation.
Complete application code must handle those errors before using results.

## Recording Spans

Suppose your report renderer accepts a caller's `ctx` and a validated format
restricted to `pdf` or `csv`. Name the tracer's **instrumentation scope** after
your application's package, and keep the operation name fixed:

```go
tracer := app.Telemetry().Tracer("example.com/reports")
ctx, span := tracer.Start(ctx, "report render",
	telemetry.Attrs(telemetry.String("format", format)),
)
defer span.End()
```

Pass the returned context to the renderer's dependencies. Their spans can then
be children of this span. The default kind is `Internal`; use `Client` for a
remote call, `Producer` for sending a message, and `Server` or `Consumer` for
receiving work. Without a started trace driver or a trace-capable observer,
`Start` returns the original context and a non-recording span without new IDs.

Record a failure with a fixed description rather than raw diagnostics. Here
`render` is your existing `func(context.Context, string) error`:

```go
err := render(ctx, format)
if err != nil {
	span.SetStatus(telemetry.Error, "render failed")
	span.SetAttributes(telemetry.String("error.type", "render_failed"))
}
```

`RecordError(err)` also records `err.Error()` as an exception event. Use it only
when that text is approved for capture and export. Built-in components classify
failures without recording their text; see [Telemetry Privacy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md).
`Unset` is the initial status, not a failure. `OK` is final; setting `Unset`, an
empty status, or an unknown status does not clear an existing verdict.

Call `End` on every span, normally with `defer`. Its first call closes admission
to updates and waits for already-admitted recorder calls; subsequent calls and
late updates do nothing. Recorder callbacks must return and must not
synchronously end their own span, including from error formatting.

Attributes use `String`, `Int`, `Int64`, `Float64`, `Bool`, `Strings`, or
`Duration` (float seconds). Empty keys are dropped. Attribute containers are
copied at publication and backend boundaries, but copying does not sanitize
their contents. Check `span.IsRecording()` before expensive attribute work.

## Recording Metrics

Create instruments once, for example in your service constructor. For the same
report renderer, count completions by the bounded format value:

```go
meter := app.Telemetry().Meter("example.com/reports")
renders := meter.Counter("reports.renders",
	telemetry.Unit("{render}"), telemetry.Keys("format"),
)
```

At the operation boundary, record one observation with its context:

```go
renders.Add(ctx, 1, telemetry.String("format", format))
```

Use a histogram for distributions, an up-down counter for in-flight work, and a
gauge for a current value. To record rendering duration, import standard-library
`time`; `startedAt` is captured before rendering:

```go
duration := meter.Histogram("reports.render.duration",
	telemetry.Unit("s"), telemetry.Keys("format"),
)
duration.Record(ctx, time.Since(startedAt).Seconds(), telemetry.String("format", format))
```

Keep the histogram declaration in the constructor too. Instruments can be
declared before startup and bind on first use afterward. Matching scope/name
pairs share a backend instrument; a later declaration with different kind or
keys is disabled with a warning through `Setup.Logger`. Driver rejection also
disables that instrument, not the observed application operation.

`Keys` filters recordings to the declared schema. Missing values become empty
strings; undeclared keys disappear and non-string values become text. This does
not bound the number or size of label values: never use request IDs, arbitrary
paths, or user input. Negative counter additions and NaN histogram values are
ignored. Bucket hints are sorted, deduplicated, and stripped of non-finite bounds.
Published key and bucket containers are detached from option drafts, and backends
receive independent copies; retaining or normalizing a draft cannot change the
facade's label schema.

For reusable package instrumentation that follows changing handles,
`telemetry.NewInstruments` lazily builds and caches instruments for the most
recent handle; `For(handle)` rebuilds when the handle changes. Do not keep one
global instrument bound to an unrelated application's handle. `Enabled()` is a
startup-state check, not proof of a configured signal: a started `New(nil)` also
returns true so observers can participate.

## Trace Propagation

A process boundary needs serialized trace context, not a transported Go context.
HTTP, queues, broadcast, and workflows provide their own integration. For an
application-specific transport, initialize a carrier and inject the sending
context:

```go
metadata := telemetry.MapCarrier{}
app.Telemetry().Inject(ctx, metadata)
```

Send this carrier with the message. Only after authenticating or otherwise
trusting its source should the receiver extract it into its local operation
context:

```go
ctx = app.Telemetry().Extract(ctx, metadata)
```

`HeaderCarrier` adapts `http.Header`; `MapCarrier` adapts string metadata. The
driver owns the wire format. Missing or malformed context leaves the context
unchanged. Raw extraction does not authenticate the sender or scrub baggage.
[Request Tracing](https://arkoren.dev/docs/guides/observability/request-tracing.md#incoming-trust) explains the separate HTTP
defaults: reject incoming headers, but allow outgoing injection to internal
destinations.

## Correlated Logs

The application wraps its logger. Use context-aware methods while the context
contains a valid span:

```go
app.Logger().InfoContext(ctx, "report rendered", "format", format)
```

The normal handler receives `trace_id` and `span_id`; plain `Info` has no
context and adds neither. A log-capable driver can also export the complete
record. Correlation is not redaction: review log messages and attributes before
enabling [log export](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md#logs).

`telemetry.SpanFromContext(ctx)` never returns nil. Its `Context()` exposes
`TraceID` and `SpanID`, whose `String()` methods return hex. A valid identity
does not guarantee that a backend retained the trace.

## Passive Observation

Use `Observe` when another in-process component needs facade observations
without replacing the configured backend. With an existing started
`observer` implementing `telemetry.Observer`:

```go
unsubscribe := app.Telemetry().Observe(observer)
defer unsubscribe()
```

Observation neither starts nor stops the observer, and the facade must also be
started. For application-owned bounded history and protected access, use
[Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md#attach-capture), not an ad hoc
public debug endpoint.

Callbacks are synchronous on producing goroutines and can run concurrently.
Keep them fast, nonblocking, panic-free, and nonrecursive for the same signal.
Finished spans report the backend-selected context and parent; attributes,
events, status, and timestamps reflect facade calls, not the whole native SDK
stream. Native OpenTelemetry parent selection takes precedence when valid.

Span snapshots are shared read-only. Copy containers before editing, clone log
records before retaining them, and copy measurement label values before the
callback returns. Only spans started while subscribed are captured; they can
finish after detachment if notification is admitted while telemetry remains
enabled. `Stop` suppresses new notifications but neither drains nor cancels
already-admitted callbacks. Coordinate your own observer completion barrier.

An observer implementing `TraceDriver` or `MetricDriver` can supply a missing
signal; the first eligible observer wins and a configured driver is never
replaced. That observer records through its driver role, without duplicate
passive notification for the same signal. Detaching restores the next eligible
observer or the no-op; existing spans retain their original recorder.

## Lifecycle Rules

Before `Start` and after `Stop`, the facade records nothing. A second `Start`
wraps `ErrStarted`; a start after stop or failed startup wraps `ErrStopped`.
Construct a new handle to retry a failed startup. `Stop` is safe before startup
and on repeated calls. Lifecycle calls serialize across backend work; their
callbacks must return and must not synchronously reenter the same lifecycle.
Cancellation cannot forcibly interrupt backend work or waiting for an earlier
lifecycle call.
Startup receives its context without retaining it; stop forwards the supplied
context, including cancellation, after disabling new recordings and notifications.

The application manages this lifecycle for a configured handle. It gives
telemetry the remaining shutdown budget, with at least ten seconds. Drain
application work and end spans before telemetry stops. Background exporters can
lose the final batch if the process exits without stopping them.

Outside an application, start the handle yourself. Here `handle` is your
configured `*telemetry.Telemetry`:

```go
err := handle.Start(ctx, telemetry.Setup{
	Resource: telemetry.Resource{Service: "reports-cli"},
})
```

After successful startup and completed work, use a fresh bounded shutdown context
instead of reusing a canceled operation context. Import `context`, `errors`, and
`time`; `err` is the operation error you want to preserve:

```go
shutdown, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
err = errors.Join(err, handle.Stop(shutdown))
```

Preserve shutdown errors. OpenTelemetry flushes with this budget but completes
owned native teardown on an uncanceled context, so `Stop` can outlast the
deadline; see [export lifecycle](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md#export-lifecycle).

For assertions on reported work, use the [telemetry fake](https://arkoren.dev/docs/guides/testing/fakes.md).
For all declarations, see the
[`telemetry` API reference](https://pkg.go.dev/arkoren.dev/core/service/telemetry).

---

Source: https://arkoren.dev/docs/guides/observability/telemetry-drivers.md
[HTML documentation](https://arkoren.dev/docs/guides/observability/telemetry-drivers)

# Drivers

Choose **Memory** for bounded process-local history and correlation without a
backend, **OpenTelemetry** for OTLP export, or **Prometheus** for scraped
metrics. You can select a different driver per signal. Start with the signals
you need, and review [Telemetry Privacy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md) before exporting
logs or exposing a metrics endpoint.

## Compatibility Matrix

This compares Arkoren's adapters, not the vendors' entire products. Unsupported
signals are no-ops unless a capable observer supplies them. Recording also
depends on startup and the selected signal configuration.

| Driver | Traces | Metrics | Logs | OTLP Export | Scrape Handler |
| --- | --- | --- | --- | --- | --- |
| [Memory](#memory) | Yes | Yes | Yes — bounded local capture | No | No |
| [OpenTelemetry](#opentelemetry) | Yes | Yes | Yes | Yes — owned pipelines; supplied providers choose export | No |
| [Prometheus](#prometheus) | No | Yes | No | No | Yes — requires a usable gatherer |

Startup does not prove remote delivery. Memory is not durable storage;
Prometheus does not create trace IDs. Its exemplars require sampled context
from another trace source.

For all following alternatives, import `application` from
`arkoren.dev/core/application` and `telemetry` from
`arkoren.dev/core/service/telemetry`. Give your chosen `driver` to the application:

```go
app := application.New(func(app *application.Builder) {
	app.Telemetry(telemetry.New(driver))
})
```

Construction does no lifecycle I/O. The application starts telemetry before
user startup callbacks and stops it last. Name, version, and environment become
the resource identity. [Lifecycle Rules](https://arkoren.dev/docs/guides/observability/telemetry.md#lifecycle-rules) covers
manual startup, terminal facade failures, and shutdown budgets.

## Memory

The bundled `arkoren.dev/core/service/telemetry/memorytelemetry` package needs
no additional module or backend. It generates W3C trace IDs, propagates
`traceparent` and `tracestate`, retains finished spans and logs, and aggregates
metrics locally. Your normal log handler can still send logs elsewhere.

### Local Capture

Keep the store handle so your local inspector can read it, then use `driver`
in the application wiring above:

```go
import "arkoren.dev/core/service/telemetry/memorytelemetry"

driver := memorytelemetry.New()
```

After startup, run a request through [Request Tracing](https://arkoren.dev/docs/guides/observability/request-tracing.md).
A `traceresponse` header and a retained span confirm identity and capture.
With standard-library `fmt`, inspect completed work:

```go
for _, span := range driver.Spans() {
	fmt.Println(span.Name, span.Context.TraceID.String())
}
```

This is a process-local snapshot, not a query across replicas. Protect any
operator interface that reads it. [Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md)
owns application capture and tool registration; this constructor opens no
listener. For assertions rather than operator inspection, use
[telemetry fakes](https://arkoren.dev/docs/guides/testing/fakes.md).

For a standalone store's tool definitions, `driver.MCPTools("memory")` returns
prefixed trace, log, and metric tools without registering or starting anything.
Check their `Err()` results and select/register them on a protected endpoint as
in [Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md#protect-registration).
The store still needs its owner to start and stop it; do not start it separately
if application telemetry already owns that lifecycle. See the
[tool lookup](https://arkoren.dev/docs/guides/diagnostics/diagnostic-tools.md#owner-prefixes) for names.

### Retention Limits

If the default history is too large, construct an alternative store:

```go
driver := memorytelemetry.New(func(store *memorytelemetry.Builder) {
	store.MaxSpans(300)
	store.MaxLogs(300)
	store.MaxSeries(100)
})
```

Each default is 1000. Span and log rings evict their oldest records at capacity.
`MaxSeries` bounds label combinations **per instrument**; a new combination
past capacity is dropped without replacing existing series. Zero or negative
limits disable the corresponding retention. Setting all three to zero preserves
ID generation and propagation without retaining history; nonpositive span capacity
also disables span attribute recording in this driver.

These are count limits, not a total byte or instrument-definition budget. Keep
instrument names, keys, and value sizes fixed and trusted.

### Snapshot Ownership

`Spans()`, `Logs()`, and `Metrics()` return detached eager collections. Spans and
logs are oldest first; metrics follow instrument creation order. Later writes,
ring eviction, and `Reset()` do not alter an existing snapshot. Reset clears
points and history but retains metric registrations and does not restart log
sequence numbers. `LogsAfter(seq)` selects retained entries with greater
sequence numbers; advance from the last observed sequence, not a filtered count.
Empty log snapshots remain nil; empty span and metric snapshots are allocated
empty collections. Snapshot copies detach owned attributes, events, keys,
points, and label values; the collection wrapper itself does not acquire ownership.

Malformed W3C context, uppercase hex, and zero IDs are rejected. Valid parent
identity is reported by the recorder; generated IDs are correlation values,
not credentials. Resource containers are copied at startup and inspection.
Stop synchronizes with storage commits but cannot interrupt caller-supplied
error formatting or `LogValuer` work. Formatting that finishes after stop cannot
append a new log. Process exit discards all history.

For all settings and snapshot types, see the
[`memorytelemetry` API reference](https://pkg.go.dev/arkoren.dev/core/service/telemetry/memorytelemetry).

## OpenTelemetry

Choose this adapter when your collector accepts OTLP traces, metrics, or logs.
It uses the OpenTelemetry Go SDK and enables Go runtime metrics by default.
Install from your application's module directory:

```sh
go get arkoren.dev/driver/otel
```

Import the adapter and construct the driver for the shared application wiring:

```go
import "arkoren.dev/driver/otel/oteltelemetry"

driver := oteltelemetry.New()
```

Standard `OTEL_*` variables configure driver-owned pipelines. HTTP/protobuf is
the default protocol (port 4318); `grpc` uses port 4317. Signal-specific endpoint
and protocol variables take precedence over generic environment settings;
explicit nonempty builder settings take precedence over the environment.

### Collector Settings

Use HTTPS/TLS for a remote collector. For an approved endpoint and credentials
loaded by your application as `endpoint` and `headers`:

```go
driver := oteltelemetry.New(func(instrumentation *oteltelemetry.Builder) {
	instrumentation.Endpoint(endpoint)
	instrumentation.Headers(headers)
})
```

`Endpoint` accepts `host[:port]` or an HTTP(S) URL. Explicit endpoints for owned
exporters are validated at startup: unsupported schemes, malformed addresses,
URL credentials, nonempty queries, and fragments are rejected without echoing
the address. Put authentication in headers. Headers are copied; empty settings
retain native fallback behavior.

An `http` URL or `Insecure(true)` disables TLS. `Insecure(false)` does not force
TLS against an HTTP URL or native environment policy. For a trusted local gRPC
receiver only, use `Protocol(oteltelemetry.ProtocolGRPC)` and `Insecure(true)`.

To retain a fraction of new traces, import `sdktrace` from
`go.opentelemetry.io/otel/sdk/trace` and use a separate construction variant:

```go
driver := oteltelemetry.New(func(instrumentation *oteltelemetry.Builder) {
	instrumentation.Sampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)))
})
```

This samples about ten percent of new traces while following a parent's sampling
decision. It reduces volume, not the sensitivity of retained fields. Incoming
parent trust remains a separate [HTTP policy](https://arkoren.dev/docs/guides/observability/request-tracing.md#incoming-trust).

For owned pipelines, `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, and
`OTEL_LOGS_EXPORTER` accept `otlp` (default) or `none`. Other exporters and
`http/json` are unsupported and fail startup. `OTEL_SDK_DISABLED=true` disables
this adapter's recording and export, including supplied providers, but leaves
the configured propagator active. None of these settings removes propagation
headers or disables independent native SDK pipelines.

### Native Providers

If you already own a native tracer provider `provider`, share it explicitly:

```go
driver := oteltelemetry.New(func(instrumentation *oteltelemetry.Builder) {
	instrumentation.TracerProvider(provider)
})
```

`MeterProvider`, `LoggerProvider`, and `Propagator` accept their corresponding
native interfaces too. Keep those handles for native calls: the adapter exposes
no provider getters and does not replace global providers. Supplied providers
retain their resource, sampling, and exporter configuration; connection settings
and `Sampler` do not reconfigure them. Per-signal exporter environment variables
do not disable them either.

The adapter calls a borrowed provider's `ForceFlush(context.Context) error` if
available, but never shuts that provider down. Stop the adapter before your own
final provider shutdown. Runtime callbacks installed on a borrowed meter remain
the adapter's responsibility and are unregistered on stop or failed startup.

Nil interface settings restore fallback construction. Typed-nil provider or
propagator pointers are invalid at startup; a typed-nil sampler is invalid when
the adapter builds the tracer provider. An unused sampler is ignored with a
supplied tracer provider.

`RuntimeMetrics(false)` disables runtime instrumentation.
`GlobalErrorHandler(false)` leaves the process-wide SDK error policy alone.
Otherwise startup installs an SDK error handler using the application's
unwrapped logger and does not restore it on stop or failure. Native exporter
errors can contain sensitive diagnostics; this is not a redaction boundary.

### Export Lifecycle

Startup builds the resource and owned providers; partial or canceled acquisition
releases acquired resources. A failed **facade** startup is terminal even if the
underlying driver could retry. The application supplies `service.name`,
`service.version`, and `deployment.environment.name`. Empty identity fields
fall back to environment configuration; without a service name the fallback is
`unknown_service:<executable>`. Review extra resource attributes as exported data.

Default trace/log batching and periodic metric export run outside the ordinary
request path. Startup success does not prove a later export reached a collector.
Drain work, close response bodies, and end spans before shutdown. For each owned
pipeline, stop preserves `ForceFlush` errors using the supplied context, then
completes SDK shutdown with an uncanceled context in reverse acquisition order.
Shutdown can perform final exports and outlast the flush deadline; native
timeouts still apply. No teardown goroutine is abandoned. Borrowed flush hooks
and serialized lifecycle calls also require cooperation.

A valid native OpenTelemetry parent wins over a conflicting facade parent,
including remote or non-recording native parents. Otherwise a valid facade
parent is adopted. Recorder and observer parent snapshots preserve the selected
IDs, sampling, remote bit, and tracestate; roots report zero. The returned context
still carries the native child. Native-only spans are not automatically part of
facade observation.

Metric values use string labels with empty values omitted. Log export maps slog
severity, flattens groups into dotted keys, and attaches the context's span;
all emitted record content remains subject to your [log policy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md#logs).

### Verify Exports

With your collector configured, start the application, send a request through
[Request Tracing](https://arkoren.dev/docs/guides/observability/request-tracing.md#incoming-requests), and find the span under
your service name in the backend. Check asynchronous exporter errors if it is
missing; a response trace ID alone does not prove export.

For a trusted local OTLP/HTTP receiver that accepts traces only, run your
already-instrumented application with metrics and logs disabled:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
  OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
  OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none go run .
```

The receiver must already be running and configured to retain or forward traces.
Verify the request there, then stop your application normally to flush its last
batch. See the
[`oteltelemetry` API reference](https://pkg.go.dev/arkoren.dev/driver/otel/oteltelemetry)
for every provider and connection option.

## Prometheus

Choose this adapter for metrics pulled from an HTTP endpoint. It creates an
isolated registry by default and supplies no traces or exported logs. Install:

```sh
go get arkoren.dev/driver/prometheus
```

### Scrape Setup

Import `prometheustelemetry` from
`arkoren.dev/driver/prometheus/prometheustelemetry`. Retain the handle to mount
its scrape handler, and give it to the shared application wiring:

```go
driver := prometheustelemetry.New()
```

Before mounting, establish a restricted [operator listener](https://arkoren.dev/docs/guides/application/health-and-readiness.md)
or authenticated gateway. The handler has no built-in access control and can
expose label values, resource metadata, and native collector errors. With that
listener's router `r` and `router` from `arkoren.dev/core/http/router`:

```go
r.Get("/metrics", router.HTTP(driver.Handler()))
```

The handler serves Prometheus text or negotiated OpenMetrics. It can serve
caller-owned collectors before startup and after stop; a successful scrape is
not proof that Arkoren recording has started.

### Metric Budgets

To lower retained state, construct an alternative driver with explicit budgets:

```go
driver := prometheustelemetry.New(func(metrics *prometheustelemetry.Builder) {
	metrics.MaxSeries(200)
	metrics.MaxInstruments(100)
})
```

Both defaults are 1000. `MaxSeries` bounds actual label combinations per
instrument, including an unlabeled combination. At capacity, new combinations
are dropped before children, buckets, or exemplars allocate; admitted values
continue without eviction or resets. There is no overflow series or per-drop
notification. `MaxInstruments` rejects new definitions beyond its limit while
retaining existing instruments. Zero restores the finite default; negative
disables the corresponding admission. Neither means unlimited.

These limits exclude runtime collectors, `target_info`, and your native
collectors. They bound counts, not bytes or the facade's definition cache.
Keep names, keys, bucket lists, and label sizes trusted and bounded; scrape-side
limits cannot protect already-retained process memory.

### Native Registries

To serve existing native collectors and Arkoren from one registry, import
`prometheus` from `github.com/prometheus/client_golang/prometheus` and retain your
own registry handle:

```go
registry := prometheus.NewRegistry()
driver := prometheustelemetry.New(func(metrics *prometheustelemetry.Builder) {
	metrics.Registry(registry)
	metrics.Namespace("reports")
})
```

`Registry` sets registration and gathering together. Alternatively supply
`Registerer` and `Gatherer`, including `prometheus.DefaultRegisterer` and
`prometheus.DefaultGatherer` for the shared process registry. Omitted or nil
interfaces infer compatible counterparts; a registerer without a usable
gatherer yields HTTP 501. Typed-nil pointers, including `Registry(nil)`, instead
fail startup and yield a fixed HTTP 503 even before startup.

The namespace prefixes managed instruments, not runtime metrics or `target_info`.
`RuntimeMetrics` and `TargetInfo` default to true; explicit false disables each.
Keep the native registry for your own collector registration and removal: there
is no registry getter, and your collectors remain yours after driver stop.
Prevent generated sample-name conflicts yourself. Managed validation cannot
contain arbitrary native collector panics or inspect every emitted sample name.

### Metric Names

Dots and other invalid characters become underscores; units add suffixes and
counters gain `_total`. For example, `http.server.request.duration` with unit
`s` becomes `http_server_request_duration_seconds`, with `_bucket`, `_sum`, and
`_count` series. Up-down counters and gauges both become Prometheus gauges.
The instrumentation scope is omitted, and label keys undergo the same spelling
translation. Avoid names that collide after translation.

Translated name/key collisions, descriptor conflicts, overlapping managed
histogram sample names, and invalid buckets reject an instrument. The facade
warns and disables that instrument; other instruments continue. Retained keys
and buckets are copied. Invalid UTF-8 label values and negative counter
increments are dropped.

Startup registers Go/process collectors and `target_info`, whose labels hold
the resource rather than being copied onto every metric. Invalid resource
labels fail startup; validation does not remove sensitive content. Failed
startup rolls back newly acquired collectors. Stop removes only collectors the
driver registered, leaving equivalent preexisting and caller-owned collectors.
It waits for native `Unregister` calls even with an expired context.

Counter and histogram exemplars carry IDs when recording inside a sampled span
from a trace driver or capable observer. Reading them requires OpenMetrics
negotiation and exemplar storage on the Prometheus server; they are not traces.

### Verify Scraping

After startup, send a request through [Request Tracing](https://arkoren.dev/docs/guides/observability/request-tracing.md).
Read the protected `/metrics` endpoint and look for
`http_server_request_duration_seconds_count` (with your namespace prefix, if
configured). This confirms an observed request, not successful remote scraping.

To connect a Prometheus server, save an application-specific `prometheus.yml`
using a private address reachable by that server:

```yaml
scrape_configs:
  - job_name: reports
    static_configs:
      - targets: ["reports.internal:8081"]
```

Configure authentication/TLS there if the listener requires it. Confirm the
`reports` target is up and query its request-count series. Containerized
scrapers need a reachable private host address; host loopback is not universally
reachable from a container. Do not expose diagnostic tools to solve scrape
connectivity. Normal application shutdown unregisters managed collectors;
retire temporary scrape configuration separately.

See the
[`prometheustelemetry` API reference](https://pkg.go.dev/arkoren.dev/driver/prometheus/prometheustelemetry)
for all registry and budget options.

## Combining Drivers

Use `telemetry.Compose` when each signal needs its own destination. With the
OpenTelemetry and Prometheus imports above, send traces and logs over OTLP and
serve metrics locally:

```go
exporter := oteltelemetry.New()
scraper := prometheustelemetry.New()
driver := telemetry.Compose(
	telemetry.Traces(exporter),
	telemetry.Logs(exporter),
	telemetry.Metrics(scraper),
)
```

Give `driver` to the application and mount `scraper.Handler()` on its protected
route. Composition routes facade signals; it does not reconfigure another
driver's native pipelines. Disable unwanted owned OTLP metrics with
`OTEL_METRICS_EXPORTER=none`. To keep logs on your existing handler only, omit
the log selection and disable the unwanted owned OTLP log pipeline too.

The last selection for a signal wins. A shared pointer driver starts once;
distinct selected drivers start in first-selection order and stop in reverse.
Unselected drivers never start. On failure, already-started drivers stop and
errors are joined; the failing driver owns its partial acquisition. Rollback
passes the startup context, including cancellation, and still attempts every
acquired stop. Lifecycle calls serialize, so native callbacks must return and
must not synchronously reenter the same composite.

Use pointer handles to share identity across signals: noncomparable value
drivers in separate selections are distinct. Missing signals remain no-ops
unless an observer supplies them. Selecting only Prometheus gives no trace IDs;
select `telemetry.Traces(memorytelemetry.New())` as well if you want local
correlation without trace export. `DriverAs[T]()` can find components behind a
composite or an unwrapping decorator.

Custom implementations follow the [driver guide](https://arkoren.dev/docs/guides/drivers/writing-a-driver.md)
and [conformance suites](https://arkoren.dev/docs/guides/testing/conformance-suites.md). Every span recorder
and decorator must report the immutable actual selected parent through
`Parent() telemetry.SpanContext`, preserving IDs, sampled flag, remote bit, and
tracestate, or zero for a root. Do not infer it from the child or a conflicting
facade identity. For contracts and composition options, see the
[`telemetry` API reference](https://pkg.go.dev/arkoren.dev/core/service/telemetry).

---

Source: https://arkoren.dev/docs/guides/observability/instrumented-components.md
[HTML documentation](https://arkoren.dev/docs/guides/observability/instrumented-components)

# Instrumented Components

Use this catalog when you need the exact instrument behind a slow operation or
an alert. It describes observations made through Arkoren facades, not every
native SDK call. Configure a [telemetry handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle)
first; following a trace across processes also requires propagation and telemetry
in each process.

## Shared Rules

Core service scopes are their package paths, such as
`arkoren.dev/core/service/cache`. Unless an entry says otherwise, spans are
`Client` spans and duration histograms use seconds. Each listed error label is
`error.type`; classifications vary by component between known sentinels, fixed
fallbacks, and Go type names. Automatic spans and metrics omit raw error text
and do not call `RecordError`.

Set the component's `Builder.Telemetry(handle)` to select an explicit handle.
Nil or omission follows the process default at operation time; messaging,
workflow, and LLM use their role-specific builders below. Native driver calls
bypass facade instrumentation. An inactive handle skips observation work;
an enabled handle does not imply every signal has a backend.

The exclusions here are not a redaction filter for logs, custom attributes, or
diagnostic tools. Review [Telemetry Privacy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md) separately.
Counts describe observed calls or attempts, not an exactly-once business feed;
[events and auditing](https://arkoren.dev/docs/guides/events/auditing.md) have different responsibilities.

## Database

`database.Builder.Telemetry` selects the handle. Statements through `Exec`,
`Select`, `Find`, and `Cursor` record spans named by SQL verb and known table,
such as `SELECT reports`; without a known table they use the verb, and without
a known verb they use the database system. Cursor timing covers traversal,
not a separate span per row. `Lock` and `TryLock` observe acquisition as `LOCK`
and `TRY_LOCK`.

| Field | Meaning |
| --- | --- |
| `db.system.name` | Dialect-derived `postgresql`, `mysql`, `sqlite`, or `other_sql`; overridden by `Builder.System` |
| `db.namespace` | `Builder.Namespace`, omitted when empty |
| `db.operation.name` | Recognized leading SQL verb, or `LOCK` / `TRY_LOCK` |
| `db.collection.name` | Table known from a table-based query builder |
| `db.query.text` | Span only; compiled SQL when explicitly enabled |

`db.client.operation.duration` uses `db.system.name`, `db.namespace`,
`db.operation.name`, `db.collection.name`, and `error.type`. Recognized leading
verbs are `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `CREATE`, `ALTER`, `DROP`, and
`TRUNCATE`; this classification does not parse comments or CTEs. Missing rows
are not telemetry failures.

`QueryText` defaults to false. Enabling it exports compiled SQL verbatim,
including literals and comments; bound arguments stay separate but embedded
values are not redacted. Query text never becomes a metric label. See
[database monitoring](https://arkoren.dev/docs/guides/database/query-monitoring.md) before opting in.

## Cache

`cache.Builder.Telemetry` selects spans named `cache {operation}` and these
metrics:

| Instrument | Label Keys |
| --- | --- |
| `arkoren.cache.operations` (counter) | `arkoren.cache.operation`, `arkoren.cache.store`, `arkoren.cache.hit`, `error.type` |
| `arkoren.cache.operation.duration` | `arkoren.cache.operation`, `arkoren.cache.store`, `error.type` |

Operations are `get`, `put`, `compare_and_put`, `delete`, `add`, `has`, `pull`,
`increment`, `decrement`, `increment_with_ttl`, `ttl`, `lock`, `try_lock`, and
`remember`. Store is `Builder.Name`, or `default`. Lock spans cover acquisition,
including the blocking wait, not lease extension or release.

Known reads add a boolean `arkoren.cache.hit` span attribute and a `true` or
`false` counter label. A missing key is a miss, not an error; finding bytes
that fail decoding is still a hit. Unknown lookup outcomes and non-read
operations have no span hit attribute and an empty counter hit label.
`compare_and_put` guard mismatch is a successful comparison, not a miss or
failure. Keys, guards, and values are omitted. For a correctly filtered hit
ratio and nested `remember` observations, see [cache monitoring](https://arkoren.dev/docs/guides/cache/monitoring.md).

## Feature Flags

`feature.Builder.Telemetry` observes each evaluation with an internal
`feature evaluate` span.

| Instrument | Label Keys |
| --- | --- |
| `arkoren.feature.evaluations` (counter) | `feature_flag.key`, `feature_flag.provider.name`, `feature_flag.result.reason`, `error.type` |
| `arkoren.feature.evaluation.duration` | `feature_flag.key`, `feature_flag.provider.name`, `error.type` |

The provider name is `Builder.Name` or `default`. Reason comes from the driver,
such as `static` or `targeting_match`, and becomes `error` on failure. A supplied
`feature_flag.result.variant` is span-only. Subject keys, subject attributes,
and flag values are omitted; flag keys and variants still require your privacy
and cardinality review. See [feature evaluation](https://arkoren.dev/docs/guides/feature-flags/overview.md).

## Search

`search.Builder.Telemetry` observes `sync`, `check`, `put`, `delete`, `drop`, and
`search` as `search {operation}` spans. `arkoren.search.operations` is the
counter; `arkoren.search.operation.duration` is the histogram. Both use
`arkoren.search.operation`, `arkoren.search.name`, `arkoren.search.index`, and
`error.type`.

Name is `Builder.Name` or `default`; index is empty for engine-wide `sync`
and `check`. Successful search spans add `arkoren.search.hits` for the returned
page only. Queries, documents, and raw diagnostics are omitted. Index names can
still reveal tenants; keep them bounded. See [Search](https://arkoren.dev/docs/guides/search/overview.md).

## Storage

`storage.Builder.Telemetry` records `storage {operation}` spans for `open`,
`put`, `stat`, `delete`, `list`, and `sign`. The counter
`arkoren.storage.operations` and histogram `arkoren.storage.operation.duration`
use `arkoren.storage.operation`, `arkoren.storage.store`, and `error.type`.
Store is `Builder.Name` or `default`; successful writes add the span-only
`arkoren.storage.object.size` in bytes. Histogram bounds span 1 ms to 1 minute.

`open` ends when the driver returns the body, not after the caller reads or
closes it. `put` measures reader submission; upload helpers reuse that
observation. `list` measures one page, and `sign` measures URL creation rather
than the later request using it. Missing objects are not failures. Object keys,
prefixes, cursors, and signed bearer URLs are omitted. See
[storage observations](https://arkoren.dev/docs/guides/file-storage/events-and-telemetry.md).

## Mail

`mail.Builder.Telemetry` records `mail send` for `Mailer.Send` with a nonnil
context. `arkoren.mail.operation` is `send`; span-only counts are
`arkoren.mail.recipients` (to, cc, and bcc combined) and
`arkoren.mail.attachments`. The `arkoren.mail.operations` counter and
`arkoren.mail.operation.duration` histogram use operation and `error.type`.
Histogram bounds span 10 ms to 1 minute.

Validation failures are observed before submission. Error precedence is
`context.Canceled`, `context.DeadlineExceeded`, `mail.ErrInvalidMessage`, then
the innermost wrapped Go error type. Addresses, subjects, bodies, provider
diagnostics, and receipts are omitted.

These metrics count attempts, not confirmed delivery. A driver panic closes
the mailer's span and records attempt count/duration without ending the parent
span, retaining panic data, or classifying the panic itself; the panic still
propagates and emits no `Sent` or `Failed` notification. Nil context produces
neither telemetry nor events. Notification failure classes are bounded and
separate from telemetry's native-type fallback; see
[mail observations](https://arkoren.dev/docs/guides/mail/events-and-telemetry.md).

## Push Notifications

`push.Builder.Telemetry` observes calls after portable preflight validation as
`push send`. The counter `arkoren.push.operations` and histogram
`arkoren.push.operation.duration` use `arkoren.push.operation` and `error.type`.
Operation is `send`.

`push.ErrOutcomeUnknown` wins over joined or wrapped cancellation, deadline, and
provider failures in telemetry. Otherwise cancellation precedes deadline, known
push sentinels, and the innermost wrapped Go type. This changes observation
classification only: returned errors preserve their causes and notifications
retain their separate bounded classes.

No recipient, endpoint, token, encryption material, content, application data,
or receipt ID enters these fields. Acceptance does not prove device delivery or
display. See [Push Notifications](https://arkoren.dev/docs/guides/push/overview.md).

## Payments

`payment.Builder.Telemetry` observes provider operations after portable
validation as `payment {operation}`. The counter `arkoren.payment.operations`
and histogram `arkoren.payment.operation.duration` use
`arkoren.payment.operation` and `error.type`.

Keys, IDs, amounts, approval URLs, customer details, and payloads are omitted.
API acceptance is not settlement evidence. For the distinction between these
observations and business reconciliation, see [Payments](https://arkoren.dev/docs/guides/payments/overview.md).

## Secrets

`secret.Builder.Telemetry` observes provider `get`, `put`, `update`, and
`delete` as `secret {operation}`. The `arkoren.secret.operations` counter and
`arkoren.secret.operation.duration` histogram use `arkoren.secret.operation`
and `error.type`. Failure classes are only `context.Canceled`,
`context.DeadlineExceeded`, `secret.ErrNotFound`, or `other`, in that order.
Unlike a cache miss, a missing secret is a failure observation.

`Raw` observes one provider read; `String` and typed `Get` reuse it. A later
JSON decode failure is not a provider failure. Direct `Driver()` or native
calls bypass these observations. Names, versions, values, sizes, and native
diagnostics are omitted. See [Managing Secrets](https://arkoren.dev/docs/guides/secrets/managing.md).

## Hashing And Encryption

`hash.Builder.Telemetry` and `crypto.Builder.Telemetry` record metrics only:
their contextless APIs create no synthetic root spans.

| Component | Counter | Duration Histogram | Label Keys |
| --- | --- | --- | --- |
| Hash | `arkoren.hash.operations` | `arkoren.hash.operation.duration` | `arkoren.hash.operation`, `arkoren.hash.outcome`, `error.type` |
| Crypto | `arkoren.crypto.operations` | `arkoren.crypto.operation.duration` | `arkoren.crypto.operation`, `arkoren.crypto.outcome`, `error.type` |

Hash operations are `hash`, `check`, and `needs_rehash`; outcomes are `ok`,
`match`, `mismatch`, `current`, `rehash_needed`, or `error`. Mismatch is not an
error. Failure classes are `hash.ErrInvalidHash` or `error`. Crypto operations
are `encrypt` and `decrypt`; outcomes are `ok`, `previous_key`, or `error`.
Its classes are `crypto.ErrEncrypterClosed`, `crypto.ErrInvalidCiphertext`, or
`error`.

Metrics cover normally completed facade calls, including JSON work; panics do
not create completed observations. Decryption reports one final outcome across
all keys, not a failure per attempted key. No input, output, key, or native
error text/type is retained. Rehash-needed and previous-key outcomes do not
prove a persisted upgrade; applications own persistence as described in
[Hashing](https://arkoren.dev/docs/guides/hashing/overview.md) and [Encryption](https://arkoren.dev/docs/guides/encryption/overview.md).

## Queues And Broadcast

Set telemetry on every participating `queue.DispatcherBuilder`,
`queue.ConsumerBuilder`, `broadcast.PublisherBuilder`, and
`broadcast.ConsumerBuilder`. Producers record `send {destination}`;
consumers record `process {destination}`. With propagation enabled on both
sides and preserved by the transport, consumer work can continue the producer
trace. Queue retries are separate consumer attempts.

| Field | Meaning |
| --- | --- |
| `messaging.system` | Driver-reported system, or `_OTHER`; `System` overrides it |
| `messaging.operation.name` | `send` or `process` |
| `messaging.destination.name` | Queue or broadcast name |
| `messaging.operation.type` | Span-only `send` or `process` |
| `messaging.message.id` | Span-only message ID when available |
| `arkoren.queue.job` | Span-only queue job name |
| `arkoren.queue.attempt` | Span-only queue delivery attempt, starting at 1 |

Counters are `messaging.client.sent.messages` and
`messaging.client.consumed.messages`; histograms are
`messaging.client.operation.duration` for producers and
`messaging.process.duration` for consumers. All four use `messaging.system`,
`messaging.operation.name`, `messaging.destination.name`, and `error.type`.
Send counts include attempted sends, not only confirmed acceptance.

Payloads and metadata values are not span attributes or labels, but propagation
does write trace context into metadata readable by broker operators. HTTP's
incoming trust policy does not authenticate broker messages. See
[queue telemetry](https://arkoren.dev/docs/guides/queues/metadata-and-telemetry.md) and
[broadcast metadata](https://arkoren.dev/docs/guides/broadcasting/metadata-and-telemetry.md).

## Scheduling

`schedule.Builder.Telemetry` records an internal `schedule run` span for a
recurring entry or deferred task. Its returned execution context lets the
task's dependencies become child spans.

The counter `arkoren.schedule.run.count` and histogram
`arkoren.schedule.run.duration` use `arkoren.schedule.entry`,
`arkoren.schedule.kind`, `arkoren.schedule.outcome`, and `error.type`.
Histogram bounds span 1 ms to 1 hour. Kind is `entry` or `task`; outcome is
`ok`, `skipped`, or `failed`. `schedule.ErrSkipped` is not a failed span;
recovered panics use `schedule.panic` as their error class. Task payloads and
error text are omitted, but developer-selected names are recorded. See
[scheduled failures](https://arkoren.dev/docs/guides/scheduled-tasks/error-handling.md).

## HTTP

Use the handle's `Middleware()` and `Transport()` rather than service-builder
options. Their scope is `arkoren.dev/core/service/telemetry`.

| Boundary | Span | Instruments |
| --- | --- | --- |
| Server | `Server`, named `<METHOD> <route>` | `http.server.request.duration`, `http.server.active_requests` |
| Client | `Client`, named `<METHOD>` | `http.client.request.duration` |

Unknown methods use `HTTP` in span names and `_OTHER` in method attributes;
unmatched server requests omit the route from the name. Server duration uses
method, route, status, scheme, and `error.type`; active requests use method and
scheme. Client duration uses method, destination address/port, status, and
`error.type`. Full keys and outcome rules are in
[Request Tracing](https://arkoren.dev/docs/guides/observability/request-tracing.md#recorded-fields).

Server completion includes router error rendering, not proof of delivery to
the client. Client completion includes consuming or closing the response body.
The [retrying HTTP client](https://arkoren.dev/docs/guides/http-networking/http-client.md) already installs
telemetry per attempt and marks resends with `http.request.resend_count`;
wrapping it again duplicates observations.

## MCP

`mcpserver.Builder.Telemetry` selects managed-call observations in scope
`arkoren.dev/driver/mcp`. An internal `mcp call_tool` span, the counter
`arkoren.mcp.calls`, and histogram `arkoren.mcp.call.duration` use only
`arkoren.mcp.tool` (registered name) and `arkoren.mcp.outcome` (`success` or
`error`).

Arguments, results, raw errors, unknown tool names, and model-inference fields
are omitted. Native SDK registrations bypass this instrumentation. This does
not make the tool result itself private; diagnostic tools can return raw logs
and attributes. See [managed MCP observation](https://arkoren.dev/docs/guides/mcp/tools.md#call-telemetry)
and [protected diagnostic registration](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md#protect-registration).

## Language Models

`llm.ModelBuilder.Telemetry` and `llm.EmbedderBuilder.Telemetry` select these
observations in `arkoren.dev/core/service/llm`:

| Operation | Span |
| --- | --- |
| `Model.Generate`, `Model.Stream` | Client `chat {model}` |
| `Embedder.Embed` | Client `embeddings {model}` |
| `Model.Run`, `Model.RunStream` | Internal `invoke_agent {agent}`, containing model turns and internal `execute_tool {tool}` spans |

`gen_ai.client.operation.duration` uses `gen_ai.operation.name`,
`gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.response.model`, and
`error.type`. `gen_ai.client.token.usage` adds `gen_ai.token.type` (`input` or
`output`) when usage is reported. Other span-only fields include response ID,
finish reasons, agent/tool names, token details, agent step count, and embedding
input count. Model, agent, and tool names must remain bounded and nonsensitive.

Prompts, replies, tool arguments/results, and error text are excluded. Stream
traversal and cooperative iterator cleanup affect completion; usage is
provider-reported rather than an invoice. See [Observing Calls](https://arkoren.dev/docs/guides/llm/overview.md#observing-calls) for
the consumption and notification boundaries.

## Application Lifecycle

The application creates no lifecycle spans. It supplies the resource, owns
telemetry startup/shutdown, and wraps its logger for correlation. Built-in log
messages include `app started`, `app stopping`, and `app stopped`; there is no
`app starting` log. `application.Starting` runs after telemetry starts and before
user startup callbacks, so listeners can log through active telemetry. See
[application lifecycle](https://arkoren.dev/docs/guides/application/lifecycle-and-ownership.md).

## Workflow Attempts

Configure `workflow.ClientBuilder.Telemetry` in producers and
`workflow.RegistryBuilder.Telemetry` through `workflow.NewRegistry` in workers.
Each process follows its own dynamic default when omitted. Client spans are
`workflow start`, `workflow signal`, `workflow cancel`, `workflow status`, and
`workflow result`; actual registered handler invocations get internal
`workflow step_attempt` spans.

The counter `arkoren.workflow.operation.count` and histogram
`arkoren.workflow.operation.duration` use `arkoren.workflow.operation`,
`arkoren.workflow.outcome`, and `error.type`. Outcomes are `success`, `error`,
or `already_started`; duplicate-start handling preserves the API result while
not classifying it as a telemetry failure. Names, run IDs, payloads, signals,
and metadata are excluded from these automatic fields.

Retries create separate attempts. Replayed orchestration and recorded results
create none; do not add ordinary observation side effects to replayed workflow
functions. `result` measures the caller's wait, not workflow runtime, and
canceling that wait does not cancel the run. Local events and polling are not
an authoritative terminal feed. Export from each process to a shared backend
for cross-replica inspection; local capture cannot aggregate pods.

Serialized `StartOptions.Trace` is separate from application metadata and is
extracted at supported native attempt boundaries without transporting the
caller's Go context. Signals do not replace the start parent. Upgrade Temporal
workers before traced producers: older workers cannot decode traced envelopes;
untraced histories retain their shape. See the available
[workflow drivers](https://arkoren.dev/docs/guides/workflows/drivers.md) and
[Events And Telemetry](https://arkoren.dev/docs/guides/workflows/running-workflows.md#events-and-telemetry).

For custom observations, follow [Recording Spans](https://arkoren.dev/docs/guides/observability/telemetry.md#recording-spans)
and [Recording Metrics](https://arkoren.dev/docs/guides/observability/telemetry.md#recording-metrics). Full declarations live
in the [`telemetry` API](https://pkg.go.dev/arkoren.dev/core/service/telemetry), with
adapter-specific MCP settings in the
[`mcpserver` API](https://pkg.go.dev/arkoren.dev/driver/mcp/mcpserver).

---

Source: https://arkoren.dev/docs/guides/observability/request-logging.md
[HTML documentation](https://arkoren.dev/docs/guides/observability/request-logging)

# Request Logging

When a request fails, `requestlog` records its method, concrete path, observed
status, and returned error. It is failure logging, not an access log: a handler
error produces a record at any status, and an observed 5xx produces a record
even without a returned error. Other successful responses and directly written
4xx responses produce none.

## Failure Records

The path and raw error can contain personal identifiers, secrets, or native
diagnostics. Choose log access, retention, and any application-owned redaction
before enabling this middleware. If your telemetry driver exports logs, those
same fields can leave the process; span field exclusions do not protect them.
See [Logs](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md#logs).

With an existing application `app` and router `r`, import `requestlog` from
`arkoren.dev/core/http/middleware/requestlog` and select the application's logger
before registering routes:

```go
r.Use(requestlog.Middleware(func(log *requestlog.Builder) {
	log.Logger(app.Logger())
}))
```

If a report handler returns an error rendered as HTTP 503, its record contains
these fields (timestamps and other logger attributes are omitted):

```json
{
  "level": "ERROR",
  "msg": "request failed",
  "method": "POST",
  "path": "/reports",
  "status": 503,
  "error": "503 Service Unavailable"
}
```

The actual error string depends on the returned error. Any `*slog.Logger`
works. If `Logger` is nil or omitted, the middleware resolves `slog.Default()`
per request, so a later default change takes effect. Selecting a logger does
not itself enable telemetry or provide trace correlation.

## Middleware Order

For trace IDs and recovered panics, use this alternative registration with the
application's [configured telemetry](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle). Import
`router` from `arkoren.dev/core/http/router`:

```go
r.Use(
	app.Telemetry().Middleware(),
	requestlog.Middleware(func(log *requestlog.Builder) {
		log.Logger(app.Logger())
	}),
	router.Recover(),
)
```

Telemetry runs first so logging receives the request's span context. The
application logger's telemetry wrapper adds `trace_id` and `span_id` when that
identity is valid. An unrelated plain slog handler does not acquire correlation
just because tracing middleware ran.

Recovery runs inside logging so recovered panics become returned errors the
logger can observe. Do not register both setup variants on the same route
stack: choose one, or you can produce duplicate failure records.

## Response Boundaries

Logging runs after the router completes response handling, including rendering
returned errors. The observed status is what passed through the response writer,
not evidence the client received those bytes. The middleware relies on Arkoren
router response hooks; exercise it through the router rather than invoking an
isolated wrapped handler.

A returned validation error rendered as 422 produces an error log, while its
[server span](https://arkoren.dev/docs/guides/observability/request-tracing.md#recorded-fields) is not marked failed: HTTP
server spans classify 5xx responses as errors. These are intentionally different
signals, not contradictory outcomes.

A successful connection hijack is not a failure. If the handler returns an
error after taking ownership of the connection, it is still logged. Status
`0` means no final HTTP status passed through the writer; it is neither an
inferred 500 nor a status sent to the client. Direct connection writes are
outside the HTTP response hooks.

For configuration declarations, see the
[`requestlog` API reference](https://pkg.go.dev/arkoren.dev/core/http/middleware/requestlog).

---

Source: https://arkoren.dev/docs/guides/observability/request-tracing.md
[HTML documentation](https://arkoren.dev/docs/guides/observability/request-tracing)

# Request Tracing

An incoming request can wait on several outgoing calls. Instrument both HTTP
boundaries to see where that time goes: server middleware starts the request's
span, and the client transport creates child spans from the context you pass
on. A started [telemetry handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle) supplies the trace
driver and metric driver independently.

With an unstarted handle, both integrations pass work straight through. With
only a metrics backend, measurements can work without creating trace IDs. Neither
integration records bodies, headers, query strings, full paths, or raw error
text; [privacy exclusions](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md) do not apply to your logs.

## Incoming Requests

With existing application `app` and Arkoren router `r`, register telemetry before
the rest of the stack. Import `requestlog` from
`arkoren.dev/core/http/middleware/requestlog` and `router` from
`arkoren.dev/core/http/router`:

```go
handle := app.Telemetry()
r.Use(
	handle.Middleware(),
	requestlog.Middleware(func(log *requestlog.Builder) {
		log.Logger(app.Logger())
	}),
	router.Recover(),
)
```

The span remains open through downstream middleware, error rendering, and
recovered panics. Request logging uses the span context and observes errors
produced by recovery. Mount this middleware through an Arkoren router: enabled
instrumentation requires its lifecycle hooks and panics if those hooks are
absent, before starting an observation.

If explicit handle wiring is unavailable, package-level
`telemetry.Middleware()` resolves the process default on each request. Import
`telemetry` from `arkoren.dev/core/service/telemetry`; later options use that
package too. Setting the default later takes effect, but does not start it.

## Recorded Fields

A matched request creates a server span such as `GET /reports/{id}`. The
template, not `/reports/19`, keeps names bounded. Unmatched routes use the
method alone. Unknown methods use `HTTP` in names and `_OTHER` in method
attributes, so arbitrary method strings do not create new values.

| Signal | Fields Or Label Keys |
| --- | --- |
| Server span | `http.request.method`, `url.scheme`, `network.protocol.name`, `network.protocol.version`, matched `http.route`, final `http.response.status_code` |
| `http.server.request.duration` (seconds) | `http.request.method`, `http.route`, `http.response.status_code`, `url.scheme`, `error.type` |
| `http.server.active_requests` (up-down counter) | `http.request.method`, `url.scheme` |

Only 5xx responses mark a server span failed, with `error.type` equal to the
status code and a fixed description such as `HTTP 503`. A 4xx leaves status
unset; it does not mean the caller achieved its goal. A returned error can
therefore produce a [failure log](https://arkoren.dev/docs/guides/observability/request-logging.md) without a failed span.

Duration ends after router response handling, including returned-error
rendering. The observed writer status does not prove the caller received the
response. Raw writes after a connection hijack are not ordinary response-body
observations.

## Incoming Trust

The default is `telemetry.TrustNone`, including `TrustRemote(nil)`: no peer's
incoming propagation headers are extracted, even from loopback, private, or
link-local addresses. Without an existing parent in the request context, the
request starts a root trace. `X-Forwarded-For` is never consulted.

This prevents callers from choosing trace identities, influencing sampling, or
introducing baggage through this middleware. It does not undo extraction by
another instrumentation layer. A valid native OpenTelemetry parent already in
the context wins over conflicting facade identity, even when remote or
non-recording; otherwise a valid facade parent is adopted. Apply trust policy
before any upstream extraction. Facade observers report that actual selected
parent, preserving IDs, sampling, remote bit, and tracestate.

> [!WARNING]
> A private reverse-proxy address does not authenticate browser-supplied trace
> headers. Keep `TrustNone` at a public edge unless the proxy strips untrusted
> context and forwards only context from trusted workloads. Narrowing proxy
> addresses alone does not make those headers safe.

If a restricted workload network establishes that trust, opt in narrowly.
Import standard-library `net/netip`; this is an alternative middleware
construction, not an additional tracing layer:

```go
r.Use(handle.Middleware(func(policy *telemetry.MiddlewareBuilder) {
	policy.TrustedNetworks(netip.MustParsePrefix("10.42.0.0/16"))
}))
```

`TrustedNetworks` tests the immediate peer against those prefixes. For a more
specific request decision, use `TrustRemote` with your own predicate.
`TrustAll` is appropriate only when all reachable callers are trusted workloads.

`TrustInternal` is another explicit opt-in: it accepts loopback, RFC 1918
private IPv4, IPv6 `fc00::/7`, and link-local unicast peers, rejecting public or
unparseable addresses. It was the former default. Services migrating from that
behavior must explicitly choose it after checking their deployment boundary;
outgoing propagation defaults have not changed.

## Response Identity

When a span has valid identity, the middleware returns the W3C `traceresponse`
header as `00-<trace-id>-<span-id>-<flags>` and adds `trace_id` to problem
responses. A client can quote that identity to support; it is not an access
credential or a guarantee the backend retained the trace.

If your clients also need a bare-ID header, configure it at construction:

```go
r.Use(handle.Middleware(func(policy *telemetry.MiddlewareBuilder) {
	policy.TraceHeader("X-Trace-ID")
}))
```

No extra bare-ID header exists by default. `ProblemKey("request_id")` renames
the problem member; `ProblemKey("")` omits it. `TraceResponse(false)` disables
the W3C response header. Disable any configured `TraceHeader` separately if
you want no response identity. Invalid/no-op spans add none of these fields.
Request propagation uses `traceparent`, not an arbitrary correlation header.

## Correlated Logs

With the application logger and the request context, your handler can emit a
record tied to the same span:

```go
app.Logger().InfoContext(req.Context(), "report requested", "format", "pdf")
```

The wrapped handler adds `trace_id` and `span_id`. Plain `Info` has no context
and cannot add them; an unrelated slog handler does not gain correlation from
middleware alone. Memory telemetry supplies IDs without external trace export;
a log-capable driver can export the whole record too.

This does not redact application attributes, exceptions, baggage, or native
SDK logs. In particular, request failure logs include concrete paths and raw
errors even though HTTP spans omit them. Review [Logs](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md#logs)
before enabling export.

## Outgoing Requests

Wrap the base `http.RoundTripper` used by your existing client. Import
standard-library `net/http`; nil selects `http.DefaultTransport`:

```go
sender := &http.Client{Transport: handle.Transport(nil)}
```

Create outgoing requests with the incoming context so the client span can be a
child of the server span. With a validated internal service URL `endpoint`:

```go
outgoing, err := http.NewRequestWithContext(req.Context(), http.MethodGet, endpoint, nil)
```

After handling request-construction errors, send and retain the success guard
before accessing the returned body:

```go
response, err := sender.Do(outgoing)
if err != nil {
	return err
}
defer response.Body.Close()
```

Consume the response as your application requires and handle any body read or
close failure. The span ends at EOF or close, so duration includes reading the
body. Always close response bodies to release HTTP resources and complete spans.
End this work before stopping telemetry.

The transport clones requests before injection and does not mutate the caller's
request. Client spans use method-only names and record method, scheme,
`server.address`, `server.port`, and status. The seconds histogram
`http.client.request.duration` uses `http.request.method`, `server.address`,
`server.port`, `http.response.status_code`, and `error.type`.
Transport failures, non-EOF body-read failures, and 5xx responses mark the span
failed. Error labels are the Go type or status code, not diagnostic text.
Responses without a body and protocol upgrades finish without waiting for body
consumption.

The [retrying HTTP client](https://arkoren.dev/docs/guides/http-networking/http-client.md) already uses this
instrumentation: each attempt has its own span and resends add
`http.request.resend_count`. Do not wrap it again with telemetry. Package-level
`telemetry.Transport` instead follows the process default per request.

## Outgoing Policy

The transport records every destination but injects context only where its
propagation predicate allows it. By default `PropagateInternal` accepts
loopback/private/link-local IP literals, single-label names such as `reports`,
and names ending in `.internal`, `.local`, `.svc`, or `.cluster.local`.
Matching is case-insensitive and ignores the port; a public name merely
resolving to a private address is not automatically internal.

Injection can reveal trace IDs and, with the default OpenTelemetry propagator,
baggage. Do not copy inbound headers wholesale: a denied injection does not
remove propagation headers you already placed on the request.

If an approved partner should receive context, construct this alternative
client transport:

```go
sender := &http.Client{Transport: handle.Transport(nil,
	func(policy *telemetry.TransportBuilder) {
		policy.PropagationHosts("api.partner.example", "*.services.example")
	},
)}
```

The `*.` form matches subdomains, not the bare domain. `Propagation` replaces
the predicate with your own request function; `PropagateAll` injects everywhere
and `PropagateNone` nowhere. These choices govern injection, not collection or
removal of existing headers. The retrying client's equivalent option is
`Propagate`. Keep baggage nonsensitive and restrict which receivers may see it.

## Verify Tracing

In an existing router test, use `telemetrytest` from
`arkoren.dev/core/service/telemetry/telemetrytest`, plus standard packages
`net/http` and `net/http/httptest`. The fake starts its handle and registers test
cleanup. With your application's existing `showReport` handler:

```go
fake := telemetrytest.New()
r := router.New()
r.Use(fake.Telemetry(t).Middleware())
r.Get("/reports/{id}", showReport)
```

Drive the router, not its isolated middleware wrapper, and assert the template
name:

```go
response := httptest.NewRecorder()
r.ServeHTTP(response, httptest.NewRequest(http.MethodGet, "/reports/19", nil))
fake.AssertSpan(t, "GET /reports/{id}")
```

Inspect `response.Header()` for `traceresponse`. Set `RemoteAddr` explicitly
when testing trust: `httptest.NewRequest` is not evidence of a loopback peer.
The fake proves local instrumentation, not remote export; [Drivers](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md)
provides backend verification.

For every HTTP option, see the
[`telemetry` API reference](https://pkg.go.dev/arkoren.dev/core/service/telemetry).

---

Source: https://arkoren.dev/docs/guides/observability/telemetry-privacy.md
[HTML documentation](https://arkoren.dev/docs/guides/observability/telemetry-privacy)

# Telemetry Privacy

Before retaining or exporting telemetry, decide who may read each signal and
how long it may remain available. A collector, scrape endpoint, or local
diagnostic tool can expose information outside the application operation that
produced it. Low cardinality does not make a field nonsensitive.

Arkoren limits fields in its automatic span and metric instrumentation. That
is not a universal redaction layer: your logs, attributes, exceptions, resource
metadata, baggage, native SDKs, exporters, and tool results have separate
boundaries.

## Field Boundaries

By default, built-in spans and metrics omit operation content:

- HTTP bodies, headers, query strings, and concrete paths. Server spans use
  route templates; client spans still record destination hosts.
- Database bound arguments and SQL text. Explicit SQL-text opt-in is a
  separate exception described below.
- Cache keys/values; storage object keys, prefixes, cursors, and signed URLs;
  mail addresses, subjects, bodies, and receipts.
- Queue/broadcast payloads and metadata values as observation fields;
  scheduled payloads; workflow inputs, outputs, run IDs, signals, and metadata.
- Push recipients, endpoints, tokens, encryption material, contents, application
  data, and receipts; payment IDs, amounts, customer details, and approval URLs.
- Secret names, versions, values, and sizes; hashing/encryption inputs, outputs,
  passwords, hashes, ciphertext, plaintext, and keys.
- Model prompts/replies and tool arguments/results. Provider response IDs and
  other model metadata can still appear on spans.

Automatic failures use classified `error.type`, not raw diagnostic text. The
classification can be a known sentinel, fixed fallback, or Go type depending
on the component. These exclusions do not sanitize data passed through an
application logger or explicit `span.RecordError(err)`.

Metric `Keys` filters the **schema**, not the values. Only declared keys survive,
but a declared user-ID label still creates high-cardinality personal data. The
[component catalog](https://arkoren.dev/docs/guides/observability/instrumented-components.md) lists exact labels and exceptions.

## Recorded Metadata

Review these retained names and measurements against your own data policy:

| Boundary | Recorded Metadata | Review Concern |
| --- | --- | --- |
| HTTP server | Method, route template, scheme, protocol, status | API structure |
| HTTP client | Method, host/port, scheme, status | Destination topology, including third parties |
| Database | System, namespace, recognized verb, known table | Schema and database identity |
| Cache/storage | Operation and store name; cache hit state; span-only write size | Store naming and file-size patterns |
| Feature flags | Flag key, provider, reason; span-only variant | Tenant or personal data in developer-selected names |
| Search | Operation, engine/index names, span-only hit count | Tenant-specific index names |
| Mail | Operation, span-only recipient/attachment counts | Traffic patterns without addresses |
| Push/payments/secrets | Operation and classified failures | Operational activity without business payloads |
| Hash/crypto | Aggregate operations, outcomes, bounded errors | Activity and upgrade-needed observations, not inputs |
| Queue/broadcast | Destination/system; span-only message ID, job, attempt | Identifiers and destination names |
| Scheduling | Entry/task name, kind, outcome | Developer-selected names |
| Workflows | Operation, outcome, classified failure | Command/attempt activity, not terminal execution proof |
| MCP | Managed registered tool name and outcome | Available application capabilities |
| Language models | Provider/model IDs, response ID, finish reasons, usage, agent/tool names | Provider correlation values and capability names |
| Resource | Service, version, environment, extra/native attributes | Deployment identity and topology |

Keep names stable and free of personal or tenant-specific data. Native resource
attributes and `OTEL_RESOURCE_ATTRIBUTES` are exported data too. Prometheus
`target_info` exposes resource label values on its scrape endpoint; descriptor
or UTF-8 validation is not confidentiality protection.

## Query Text

Leave `database.Builder.QueryText` off unless you have audited complete SQL
statements for export. Enabling it records compiled SQL **verbatim**: separately
bound arguments remain omitted, but embedded literals and comments are not
redacted. Parameter binding alone does not make the whole statement safe.

If your application has approved those statements, import `database` from
`arkoren.dev/core/service/database` and opt in for that configured `backend`:

```go
db := database.New(backend, func(connection *database.Builder) {
	connection.QueryText(true)
})
```

This enables the span-only `db.query.text` field; it never adds SQL text to
metric labels. Omission or `QueryText(false)` keeps it out without disabling
execution measurements. Table names remain visible in spans and
`db.collection.name`. See [database monitoring](https://arkoren.dev/docs/guides/database/query-monitoring.md).

## Context Boundaries

Trace propagation is serialized context crossing a trust boundary. It is not
authentication, authorization, or a consequence of whether export is enabled.

### Incoming Context

HTTP defaults to `TrustNone`, including `TrustRemote(nil)`. The middleware
extracts no incoming headers, even from private or loopback peers; it starts a
root only if no parent already exists in the context. `X-Forwarded-For` is
ignored. Existing valid native OpenTelemetry context can still take precedence
over facade identity, so enforce trust before upstream instrumentation extracts
headers. This middleware does not scrub previously installed context or baggage.

Opt into `TrustedNetworks`, `TrustInternal`, or another predicate only for
trusted workloads. A private proxy can forward an untrusted browser's headers
unchanged: trusting the proxy address does not establish who supplied them.
The former implicit internal-peer trust now requires explicit opt-in. See
[Incoming Trust](https://arkoren.dev/docs/guides/observability/request-tracing.md#incoming-trust).

### Outgoing Context

HTTP injects context by default only into internal IP literals and names:
loopback/private/link-local addresses, single-label names, and `.internal`,
`.local`, `.svc`, or `.cluster.local` suffixes. It still records client spans
and metrics for external destinations. Allowlisting a partner changes
injection, not observation.

Denied injection does not remove headers already present on the caller's
request. Do not forward inbound headers wholesale. Baggage can contain
application values readable by every receiving service; keep it nonsensitive
and review each receiver's access and retention. The default OpenTelemetry
propagator supports baggage; the memory driver propagates trace context and
tracestate. [Outgoing Policy](https://arkoren.dev/docs/guides/observability/request-tracing.md#outgoing-policy) owns configuration.

Queue and broadcast metadata can carry trace context readable by broker users.
Workflow `StartOptions.Trace` carries it separately from application metadata
through supported engine inputs/headers; engine readers can see those values
too. Restrict submission and engine access. HTTP trust settings do not
authenticate these carriers, and raw `Extract` does no peer authentication.
See [workflow observation](https://arkoren.dev/docs/guides/workflows/running-workflows.md#events-and-telemetry).

### Response Context

The server normally returns `traceresponse` and a problem `trace_id`. These are
correlation values, not backend access credentials. If your policy prohibits
them, construct middleware with both outputs disabled. Here `handle` is your
configured telemetry and `r` your existing router; import `telemetry` from
`arkoren.dev/core/service/telemetry`:

```go
r.Use(handle.Middleware(func(policy *telemetry.MiddlewareBuilder) {
	policy.TraceResponse(false)
	policy.ProblemKey("")
}))
```

Also omit or clear `TraceHeader` if you previously selected an extra bare-ID
header. These settings suppress response identity, not recording or outgoing
propagation. Protect the backend and [diagnostic endpoint](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md#protect-registration)
independently.

## Logs

Logs contain what their producer supplies. The telemetry handler adds correlation
IDs to normal context-aware log output; it does not remove message text or
attributes. [Request Logging](https://arkoren.dev/docs/guides/observability/request-logging.md) records concrete paths and
raw errors, and [health reporting](https://arkoren.dev/docs/guides/application/health-and-readiness.md) can
include check errors revealing connection details.

Memory telemetry exports nothing externally but can retain a local copy of
emitted logs. OpenTelemetry can additionally export every accepted record with
its attributes. Review the complete logging pipeline before enabling that
signal; log handlers and native SDK instrumentation have their own behavior.

To leave application logs on your existing pipeline, omit the log selection
from [driver composition](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md#combining-drivers) and disable
any unwanted driver-owned log exporter. Composition routes facade signals;
it does not reconfigure a supplied native provider's exporter.

Native errors are not universally redacted. The default OpenTelemetry error
handler writes SDK errors to the application's unwrapped logger. Prometheus
gathering errors can reach logs or the scrape error response if no metrics can
be served. OpenTelemetry's explicit endpoint-validation errors omit the
supplied address and reject URL credentials, nonempty queries, and fragments;
that protection does not extend to every native diagnostic. Use header-based
collector authentication and protect the logs containing native failures.

## Diagnostic Access

Local capture is another retention destination, even without external export.
It can retain your explicit exception text, arbitrary attributes, and raw log
records. Passive observation reflects facade calls even when the backend span
is unsampled, so backend sampling does not promise equivalent local omission.

Diagnostic MCP call telemetry omits arguments and results, but that does not
redact the tool's returned data. Read-only trace and log tools can disclose raw
application content. Restrict tool selection, authenticate operators, authorize
capabilities, and bound capture before exposing them; follow
[Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md#protect-registration).
Retention count limits and cursor paging are not secrecy or durability controls.

## Custom Attributes

For your own instrumentation, choose values you are willing to retain in every
enabled destination:

- Use fixed span names such as `report render`, not `render report 19`.
- Use bounded kind labels, not user IDs, email addresses, full paths, or
  arbitrary request values. Moving personal data from a name to an attribute
  does not make it private.
- Prefer fixed failure descriptions and classes. `RecordError(err)` captures
  `err.Error()`; use it only for approved error text.
- Review event attributes, resource fields, and baggage just like log fields.
  A copied container protects ownership, not its contents.
- Keep native collectors and exporters under their own policy. Managed
  instrument admission limits do not cover arbitrary native collectors or
  other independently configured pipelines.

If you need a user identifier for an investigation, put it only in an explicitly
approved, access-controlled log destination. Enabling log export or diagnostic
capture changes that destination set too.

## Export Controls

Sampling reduces trace volume, not the sensitivity of sampled fields. For
driver-owned OpenTelemetry pipelines, use `OTEL_TRACES_EXPORTER=none`,
`OTEL_METRICS_EXPORTER=none`, or `OTEL_LOGS_EXPORTER=none` to disable an unwanted
signal. These do not disable caller-supplied providers: configure those providers
yourself. `OTEL_SDK_DISABLED=true` disables recording/export in this adapter,
including supplied providers, but leaves configured propagation active.

Audit other native pipelines independently. Disabling this adapter is neither
a promise that no telemetry leaves the process nor a way to strip trace headers.
Use [Drivers](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md#opentelemetry) for native ownership and
shutdown details, and the HTTP boundary policies for headers.

Prometheus exposes labels, resource metadata, and native collectors through the
handler you mount. Keep that endpoint on a restricted listener or behind
authentication; do not assume its field validation supplies access control.
Memory capture stays local only until your own code or tools expose it.

For the options discussed here, see the
[`telemetry` API reference](https://pkg.go.dev/arkoren.dev/core/service/telemetry)
and [`database` API reference](https://pkg.go.dev/arkoren.dev/core/service/database).

---

Source: https://arkoren.dev/docs/guides/diagnostics/capture-setup.md
[HTML documentation](https://arkoren.dev/docs/guides/diagnostics/capture-setup)

# Capture Setup

Diagnostic capture keeps recent application evidence in memory so an authorized
operator can inspect a running process. Attach it during application composition,
then publish only the owner-provided tools your operators need. Capture does not
mount an endpoint, install authentication, or turn REST routes into MCP tools.

Start with your existing [application composition](https://arkoren.dev/docs/guides/application/composition.md)
and [MCP server](https://arkoren.dev/docs/guides/mcp/first-server.md). Transport construction and client setup
stay there; exporter configuration stays in [Telemetry](https://arkoren.dev/docs/guides/observability/telemetry.md).

## Attach Capture

Inside the composition callback passed to `app.Start` or `app.Run`, obtain the
application handles you will use and attach capture once. Here `composer` is the
callback's `*application.Composer`, with `application` imported from
`arkoren.dev/core/application`:

```go
logger := composer.App().Logger()
telemetry := composer.App().Telemetry()

tools, err := composer.MCPTools("app")
composer.Require(err)
```

The result is `[]mcp.Tool`, not a handler. `composer.Require` stops composition
on error before those definitions are used. The nonempty prefix `app` names the
tools directly: `app.info`, `app.logs`, and `app.traces`, for example. Their CLI
group is also `app`.

Application span, metric, and log recording becomes active after successful
composition and before `Starting` listeners and startup callbacks. Handles
obtained earlier still reach that recording. Event metadata capture begins when
the listener is attached, so it can also observe events during composition.
A valid span context can supply log correlation IDs before startup without
enabling capture recording.

If you configured application telemetry, capture observes that backend's trace
IDs without replacing it. Otherwise the application enables local recording at
startup. Attaching capture does not itself start the primary telemetry driver,
configure an exporter, or change process-wide logging and telemetry defaults.
The application manages capture with its lifecycle.

Call the factory once per application. Every call attaches another capture;
declaration errors can occur after attachment without rollback. Do not retry
blindly. Calls outside synchronous composition return
`application.ErrInvalidPhase`; an empty prefix is rejected before attachment.

## Bound Storage

New capture retains up to 2000 finished spans, 5000 logs, 1000 distinct label
combinations per metric instrument, and 500 event records. These are count
limits, not a byte budget: application log text and attributes still affect
memory use. Capture is process-local evidence, not a durable audit log.

If you need a smaller log window, replace the preceding attachment with this
alternative; do not call both:

```go
tools, err := composer.MCPTools("app", func(tools *application.MCPToolsBuilder) {
	tools.Limits(application.CaptureLimits{Logs: 1000})
})
composer.Require(err)
```

Zero fields select the defaults. Negative fields disable that signal's
retention; a negative `Events` also omits the events tool. Other diagnostic tools
remain available even with retention disabled. If the application already uses
memory telemetry, capture reuses its capacities rather than resizing it.

Events retain names, payload types, and trace IDs, never payload values. Spans
appear only after they finish. Eviction, reset, and process replacement lose
history; raising a limit later cannot recover it. The [retention contract](https://arkoren.dev/docs/guides/diagnostics/diagnostic-tools.md#capture-limits)
also describes standalone stores. Review [Telemetry Privacy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md)
before adding application-specific messages or attributes.

## Select Owners

Use the router and scheduler that already serve your application. Here `public`
is your `*router.Router` from `arkoren.dev/core/http/router`, and `scheduler` is
your `*schedule.Scheduler` from `arkoren.dev/core/service/schedule`:

```go
tools = append(tools, public.MCPTools("public")...)
tools = append(tools, scheduler.MCPTools("jobs")...)
```

These factories neither start their owners nor register listeners. Distinct
prefixes avoid name collisions between instances. The [owner prefixes](https://arkoren.dev/docs/guides/diagnostics/diagnostic-tools.md#owner-prefixes)
reference gives the exact protocol names and default CLI groups.

For an inspection-only endpoint, select read-only definitions before registering
them. Import `mcp` from `arkoren.dev/core/mcp`; validate every definition so an
invalid declaration cannot disappear during filtering:

```go
var selected []mcp.Tool
for _, tool := range tools {
	composer.Require(tool.Err())
	if tool.Describe().Annotations.ReadOnly {
		selected = append(selected, tool)
	}
}
```

This excludes the scheduler's `run` action. The selection trusts these shipped
owner definitions; annotations themselves are not authorization. Even a
read-only health tool can execute dependency checks when called with `fresh`.

If different callers need different capabilities, apply
`tool.WithAvailability(policy)` to copies before registration. The application
supplies that policy from its authenticated operator identity. Availability
controls discovery and invocation, not access to an individual resource. Keep
resource checks in shared typed application handlers; see [Availability and
Authorization](https://arkoren.dev/docs/guides/mcp/availability-and-authorization.md).

> [!WARNING]
> Manual scheduler runs execute application code and bypass driver-owned overlap
> guards. Leave `run` unregistered unless you explicitly authorize that capability
> and establish overlap and duplicate-effect safety. Never retry it automatically
> after a lost response.

## Protect Registration

Protect the endpoint before making the tools reachable. Diagnostic reads reveal
source paths, handler identities, log text, telemetry attributes, and possibly
approved health error details. Restrict access to authorized operators and
protect any saved output as well as the live endpoint.

In your existing `mcpserver.New` callback, register `selected` with the shared
application handles obtained above. Here `listener` is the
`*mcpserver.Builder` from `arkoren.dev/driver/mcp/mcpserver`:

```go
listener.Tools(selected...)
listener.Logger(logger)
listener.Telemetry(telemetry)
```

Keep the admin listener private, authenticate and admit requests before the
`/mcp` mount, and apply the Host/Origin and TLS controls in
[Securing MCP](https://arkoren.dev/docs/guides/mcp/securing-mcp.md). A loopback listener with an authenticated
tunnel limits reachability, but the tunnel does not replace endpoint
authentication. Tool construction creates no inspection token or access policy.

Keep application `HealthDetails` disabled unless every affected surface is
operator-protected. Enabling it exposes check error text through MCP health,
fail-fast details, and verbose HTTP readiness; it is not a per-endpoint permission.

## Verify Exposure

After starting the application, use an [authenticated CLI connection](https://arkoren.dev/docs/guides/cli/connecting.md)
to inspect the published catalog:

```sh
arkoren tools --json
```

Confirm that only the intended tools appear for that operator. Then check the
process identity and the existing router:

```sh
arkoren app:info
arkoren public:routes
```

The identity should match the intended process, and the routes should match the
live router. If a tool is absent, check registration, selection, and caller
availability before changing names. Continue with [Application Diagnosis](https://arkoren.dev/docs/guides/diagnostics/application-diagnosis.md)
to investigate a request using retained evidence.

For factory settings, see the [application API](https://pkg.go.dev/arkoren.dev/core/application).

---

Source: https://arkoren.dev/docs/guides/diagnostics/application-diagnosis.md
[HTML documentation](https://arkoren.dev/docs/guides/diagnostics/application-diagnosis)

# Application Diagnosis

When a request fails, inspect the running process and its retained evidence
before repeating application work. A route identifies the handler; a trace ID
connects finished spans and captured logs. Health then helps distinguish a
request-specific failure from a dependency problem.

You need [published diagnostics](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md), an
[authenticated CLI connection](https://arkoren.dev/docs/guides/cli/connecting.md), and operator permission to
inspect this application. The examples use application prefix `app`, router
prefix `public`, and scheduler prefix `jobs`. Discover your endpoint's actual
names with `arkoren tools`; not every application publishes every tool.

## Confirm Target

Check process identity before interpreting evidence from an incident:

```sh
arkoren app:info
```

Confirm the application, build, and lifecycle state match the intended process.
Then find the handler for the affected registered path:

```sh
arkoren public:routes --method GET --path /invoices
```

The returned `items` identify the method, registered pattern, handler, and source
location when available. The path filter matches a registered-pattern prefix;
it does not resolve a concrete request URL. An empty result can mean the wrong
router, method, or prefix, not that no listener serves the path.

## Find Evidence

Prefer an existing request's trace ID. With active
[request tracing](https://arkoren.dev/docs/guides/observability/request-tracing.md) and default response
correlation, a valid request span supplies a `traceresponse` header in the form
`00-<trace-id>-<span-id>-<flags>`. A problem response also carries `trace_id`.
Use the 32-hex trace ID, not the entire header.

If the incident has no ID, search recent retained failures:

```sh
arkoren app:traces --status error --since 5m
```

The newest-first summaries let you select a trace without executing another
request. The error filter means at least one retained span ended in error. HTTP
tracing marks server spans as errors for 5xx responses, not 4xx responses; an
unsuccessful client request can therefore be absent from this filtered result.

If latency is the symptom, narrow by registered route and root duration instead:

```sh
arkoren app:traces --route '/invoices/{id}' --min-duration 500ms
```

Use the registered template, not `/invoices/42`. Filters operate on retained
evidence, so a missing result is not proof that the request never ran.

If reproduction is necessary, first confirm permission, a known invoice ID, and
that the application's existing GET route is read-only. For a local development
listener where that route already exists:

```sh
curl -i http://127.0.0.1:8080/invoices/42
```

Inspect its response correlation. Use your application's normal authentication
when required; this example creates neither the route nor credentials. Do not
substitute a mutating request without an explicit execution decision.

## Correlate Records

Set the shell variable `TRACE_ID` to the selected trace ID, then open its retained
span tree:

```sh
arkoren app:trace "$TRACE_ID"
```

The `roots` contain finished spans, their children, and joined logs. Multiple
roots can reflect remote or evicted parents. `orphan_logs` contains logs whose
spans are unfinished or no longer retained. A trace prefix of at least eight hex
digits works only if it identifies one trace; use a longer prefix after
`ambiguous_trace`.

Read the same trace's recent log window separately when you need an oldest-first
view across spans:

```sh
arkoren app:logs --trace-id "$TRACE_ID" --since 5m
```

This is a bounded snapshot, not a complete export. Log prefix filters can match
several trace IDs, so use the full ID when correlating one request. Protect the
output: application messages and attributes can contain private data.

Only finished retained spans appear. `trace_not_found` can mean eviction,
disabled capture, reset, or a request handled by a different process. Check
[capture setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md) and tracing middleware before increasing
limits. Neither a missing trace nor an empty log page establishes that no
application effects occurred. To observe an authorized new reproduction, use
[CLI log following](https://arkoren.dev/docs/guides/cli/waiting-and-following.md); its loss notices matter
when records are overwritten or the store changes.

## Compare Health

Read cached health first so the investigation does not trigger dependency I/O:

```sh
arkoren app:health
```

Inspect the overall `status` and each check's timestamps. Overall health can be
`stale` even when individual checks last passed. Error text is withheld unless
the application enables `HealthDetails`; do not require raw dependency errors
in automation. The CLI prints the report and exits 1 for `failing`; pending or
stale health is not that failing verdict.

If you need a current dependency observation and are authorized to run the
registered checks, request a fresh pass:

```sh
arkoren app:health --fresh
```

This executes every registered check. It neither replaces the background
readiness cache nor publishes health-transition events, so a fresh success alone
does not restore traffic. Deadlines and cancellation are cooperative: the call
waits for every callback to return, including one that ignores its context.

Observe the traffic-readiness decision separately:

```sh
arkoren app:ready
```

A successful read can return `ready: false`. Cached stale health keeps readiness
false; health with no registered checks can be `ok` without proving any external
dependency works. For a shell gate, use the CLI's
[condition waits](https://arkoren.dev/docs/guides/cli/waiting-and-following.md), rather than treating `status`
or an immediate read as a readiness gate. Background intervals and probe policy
belong in [Health and Readiness](https://arkoren.dev/docs/guides/application/health-and-readiness.md).

## Authorize Actions

Diagnosis does not itself authorize execution. If the scheduler tools are
published, inspect their configuration and the action's help first:

```sh
arkoren jobs:entries
arkoren jobs:run --help
```

Entries report schedules and next-run times, not execution history. Before a
manual run, obtain explicit operator authorization and confirm the existing
entry, its effects, overlap safety, and duplicate-effect policy with the
application owner. Driver-owned overlap guards do not protect manual runs.

Only after those checks, an authorized operator can request one run of an
existing entry named `refresh-reports`:

```sh
arkoren jobs:run refresh-reports
```

The tool executes application code immediately and waits for completion. Success
reports the name, start time, and duration, not application payloads. Entry
timeouts and caller cancellation cannot forcibly stop an uncooperative callback.
Do not run concurrently when overlap is unsafe, and never automatically retry
after an error or lost response: effects may already have occurred.

A safe `not_found` means the entry was absent during lookup. An existing entry's
execution error remains an execution failure even if it wraps
`schedule.ErrUnknownName` from a nested scheduling call. It does not prove the
entry was absent or its effects were undone. Verify the application-specific
outcome through an authorized read before considering any further action.

For exact filters, limits, failures, and cursor semantics, use
[Diagnostic Tools](https://arkoren.dev/docs/guides/diagnostics/diagnostic-tools.md).

---

Source: https://arkoren.dev/docs/guides/diagnostics/diagnostic-tools.md
[HTML documentation](https://arkoren.dev/docs/guides/diagnostics/diagnostic-tools)

# Diagnostic Tools

Use this reference to interpret the diagnostic tools your application publishes.
The endpoint's current `tools/list` and input schemas determine what the caller
can use; this inventory does not guarantee that every tool is registered or
available. [Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md) owns publication and protection;
[Application Diagnosis](https://arkoren.dev/docs/guides/diagnostics/application-diagnosis.md) owns the investigation path.

## Owner Prefixes

Factories return definitions over an existing owner's state. They do not mount
HTTP routes, authenticate callers, or start routers, schedulers, or standalone
telemetry stores.

| Factory | Empty Prefix | Explicit Prefix |
| --- | --- | --- |
| `composer.MCPTools` | Rejected before capture; requires a nonempty prefix. | `"app"` produces `app.info`, `app.health`, `app.started`, `app.healthy`, `app.ready`, `app.events`, `app.traces`, `app.trace`, `app.logs`, `app.metrics`; CLI group `app`. |
| `public.MCPTools` on a router | Protocol `routes`; CLI group `router`. | `"public"` produces `public.routes`; CLI group `public`. |
| `scheduler.MCPTools` | Protocol `entries`, `run`; CLI group `schedule`. | `"jobs"` produces `jobs.entries`, `jobs.run`; CLI group `jobs`. |
| `store.MCPTools` on memory telemetry | Protocol `traces`, `trace`, `logs`, `metrics`; CLI group `telemetry`. | `"memory"` produces `memory.traces`, `memory.trace`, `memory.logs`, `memory.metrics`; CLI group `memory`. |

Application capture includes memory telemetry tools under the application prefix
directly, without another `app` or `telemetry` segment. Negative event retention
omits the events tool. Use distinct prefixes for multiple owners.

The CLI form is `GROUP:OPERATION`, such as `app:logs`; exact protocol names are
also callable. Tables below use CLI forms and exact JSON property names.
Properties are optional unless marked required. CLI flags use hyphens where
JSON uses underscores, such as `--min-duration` for `min_duration`; see
[Calling Tools](https://arkoren.dev/docs/guides/cli/calling-tools.md) for input handling. Roles and annotations
describe capabilities but grant no access.

## Application State

These tools use the explicit prefix `app`.

| Command | Inputs | Result |
| --- | --- | --- |
| `app:info` | None. | Identity, build, lifecycle, and fresh Go runtime statistics; identity role. |
| `app:health` | `fresh`: boolean, default false. | Overall `status` and check `items`; health role. Reads the cache unless `fresh=true`. |
| `app:started` | `wait`: boolean; `timeout`: duration. | `started`, lifecycle `state`, optional `started_at`. Successful startup remains recorded after stopping. |
| `app:healthy` | `wait`: boolean; `timeout`: duration; `fail_fast`: boolean. | Cached background `status`, optional `checked_at`, and `checks`; never executes checks. |
| `app:ready` | `wait`: boolean; `timeout`: duration; `fail_fast`: boolean. | `ready` and a safe failure `reason`, using the readiness probe's decision; never executes checks. |
| `app:events [NAME]` | `name`: string glob; `since`: duration; `limit`: integer; `cursor`: string; `wait`: duration. | Oldest-first metadata page; absent when event retention is disabled. See [Cursor Pages](#cursor-pages). |

After successful startup, info includes `uptime`, a Go duration string truncated
to whole seconds, and `uptime_seconds`, the elapsed whole seconds. Both are
omitted before startup completes. Runtime statistics include goroutines, heap
allocation and system memory in bytes and formatted text, GC count, and
`gomaxprocs`.

Health statuses are `ok`, `pending`, `failing`, and `stale`. Cached individual
check records preserve their last verdict, transition time, and check time even
when the overall report becomes stale. Health does not include lifecycle
liveness; use readiness for the traffic decision. With no registered checks,
health can be `ok` without proving dependencies work.

`fresh=true` executes every registered check with the caller context and the
configured readiness timeout. It does not update the background readiness cache
or publish health transitions. The call waits for all callbacks; deadlines
cannot forcibly interrupt application code that ignores cancellation.

Event `name` uses Go `path.Match` syntax, such as `billing.*`, not the event bus's
subscription-pattern syntax. `since` is a nonnegative duration relative to each
call; zero has no age restriction. Event pages default to 100 items and accept
normalized limits of 1–1000. Records contain `time`, `age`, `name`, `payload_type`,
and optional `trace_id`, never payload values.

### Condition Waits

Conditions observe immediately unless `wait=true`. An immediate false condition
is successful structured output, not a failed gate. Omitted or zero `timeout`
selects a 30-second wait budget; a nonzero timeout requires waiting. Durations
are nonnegative Go duration strings.

For example, this is the JSON input for a readiness wait, not a dependency check:

```json
{
  "wait": true,
  "timeout": "30s"
}
```

The tool returns when ready, or fails with the final safe observation. For CLI
gating and transport timeout handling, use [Waiting and Following](https://arkoren.dev/docs/guides/cli/waiting-and-following.md).

Healthy and ready also accept `fail_fast=true`, which requires `wait=true`.
Fail-fast stops on a fresh failed background pass, including an already cached
pass that remains fresh. Pending or stale health continues waiting. Without
fail-fast, checks can recover during the wait budget. Started has no fail-fast
input. An already satisfied condition succeeds even in a terminal lifecycle.

### Condition Failures

Safe tool failures travel in result `_meta["arkoren.dev/failure"]`, not successful
structured content. The CLI treats them as failures.

| Code | Meaning |
| --- | --- |
| `invalid_input` | Nonzero timeout without waiting, or fail-fast without waiting. Schema validation can reject other invalid inputs before execution. |
| `condition_timeout` | The server wait budget expired; details contain the last observation as `result`. |
| `condition_stopped` | Failed, stopping, or stopped lifecycle prevents an unsatisfied condition from completing; details contain `result`. |
| `condition_failed` | Fail-fast observed a fresh failed background health pass; details contain the operation's `result` and safe `checks`. |

Caller cancellation or client deadline expiry can end the request without one
of these typed responses. Health failure text is withheld unless application
`HealthDetails` permits disclosure. That same setting affects verbose HTTP
readiness and fail-fast check details; it is not endpoint authorization.

## Telemetry Records

These examples use application group `app`. A standalone memory store uses its
own prefix or the default `telemetry` group. Durations are nonnegative Go
duration strings; zero removes the age or minimum-duration restriction. Trace
and log limits normalize to 1–1000; omission or zero selects the listed default.

| Command | Inputs | Result And Limits |
| --- | --- | --- |
| `app:traces` | `status`: `error` or `ok`; `route`: string; `name`: string; `min_duration`, `since`: durations; `limit`: integer. | Newest-first summaries in `items`, default 20; `truncated` means matching summaries were omitted. |
| `app:trace ID` | Required `id`: 8–32 hex digits, case-insensitive. | One uniquely matching trace's finished-span tree in `roots`, joined logs, and `orphan_logs`. |
| `app:logs` | `level`: `debug`, `info`, `warn`, or `error`; `trace_id`: 8–32 hex digits; `since`: duration; `limit`: integer; `cursor`: string; `wait`: duration. | Oldest-first cursor page, default 100; level is a minimum severity, trace ID is a case-insensitive prefix filter. |
| `app:metrics [NAME]` | `name`: instrument-name prefix. | Retained aggregates in `items`, including scope, kind, units, label keys, and points; not a query to an external metrics backend. |

For traces, `route` exactly matches the root's `http.route` or span name; `name`
matches a substring of any retained span's name. `min_duration` tests the root's
duration; `since` tests trace start age. A summary reports root duration,
retained span and error counts, status, and optional HTTP status and error type.
Summary status is `error` if any retained span ended in error; otherwise `ok`.

A trace tree can have multiple roots when parents are remote or evicted.
Retained logs whose spans are unfinished or evicted appear as orphans. Log
prefix filtering does not require a unique trace match; trace lookup does.

| Lookup Failure | Meaning And Recovery |
| --- | --- |
| `trace_not_found` | No retained trace matches. Check the full ID and capture history; eviction or missing capture is not proof no work occurred. |
| `ambiguous_trace` | Several retained traces match. Supply a longer prefix or full ID. |

Neither failure discloses candidate IDs. Malformed IDs are rejected by input
validation rather than treated as trace absence.

## Router Tools

With prefix `public`, the router publishes `public:routes [PATH]`. Its string
inputs are `path`, a case-sensitive registered-pattern prefix, and `method`, a
case-insensitive HTTP method. Omit either filter to accept all its values.

For example, this JSON input selects registered GET patterns under `/invoices`:

```json
{
  "method": "GET",
  "path": "/invoices"
}
```

The result contains every match in registration order, with handler identity
and optional source location. `truncated` is always false. This reads the live
registry, including routes added after tool construction; it does not resolve a
concrete URL or execute a handler.

## Scheduler Tools

These commands require prefix `jobs`.

| Command | Inputs | Result |
| --- | --- | --- |
| `jobs:entries` | None. | All entries in registration order: cron spec, description, timeout, overlap setting, and optional next-run time. `truncated` is false; this is configuration, not execution history. |
| `jobs:run NAME` | Required `name`: string identifying an existing entry. | Executes once and waits; success reports name, start time, and duration. |

`run` is non-idempotent application execution, not an inspection read. Publish
and invoke it only with explicit operator authorization, established overlap
safety, and a duplicate-effect policy. Entry timeouts and caller cancellation
apply cooperatively; driver-owned overlap guards do not apply. A callback that
ignores cancellation can keep the call from returning.

Only a failed entry lookup produces safe `not_found`. An existing entry's error
remains an execution failure subject to adapter redaction, including an error
wrapping `schedule.ErrUnknownName` from a nested call. Never automatically retry
after failure or a lost response: neither proves that effects were undone.
Use the [authorized action path](https://arkoren.dev/docs/guides/diagnostics/application-diagnosis.md#authorize-actions) to
decide whether execution is appropriate.

## Cursor Pages

Only application events and memory logs advertise cursor following. Their
`wait` input is a duration string, unlike conditions' boolean `wait`.

Without a cursor, the tool returns the newest matching history up to `limit`,
ordered oldest first. Its cursor anchors at the current end of the store, not
at the first page of a historical export. Older matches omitted by that
snapshot cannot be retrieved by following its cursor.

To skip history, supply `cursor: "tail"`. For example, the following input to
`app.logs` immediately anchors an error-log query:

```json
{
  "level": "error",
  "cursor": "tail"
}
```

The result is an empty `items` array and an opaque cursor. Send that returned
cursor unchanged to the same tool with the same filters to read new records.
You can change page size and wait without changing the query. Keep `since`
unchanged too; its relative age cutoff is evaluated anew for each call.

`wait` ranges from `0s` to `25s` and bounds an idle continuation. Initial
snapshots and tail anchors return immediately regardless of that budget.
Nonmatching arrivals do not extend the fixed deadline. All calls remain
cancelable.

| Field | Interpretation |
| --- | --- |
| `items` | Matching records, oldest first; never null. |
| `cursor` | Next opaque position, even on an empty page; not an authorization token. |
| `truncated` | An initial snapshot omitted older matching history; following cannot page backward. |
| `dropped` | Records overwritten since the supplied position, possibly including nonmatches. |
| `has_more` | Retained matching records remain; drain another page without waiting. Nonmatching records also advance the position. |
| `instance_changed` | Reset or replacement detected; this response is empty with a fresh tail cursor, not a mixture of store generations. |

If `instance_changed` is true and you need the new store's history, make a
separate snapshot request. Do not treat it as uninterrupted continuation.
Malformed, wrong-tool, wrong-filter, and future-position cursors return safe
`invalid_cursor`. Invalid limits or waits can yield `invalid_input` or an input
schema failure. Do not silently restart after failure and label the output
complete.

Store generations do not survive reset or replacement, and rings can evict
records between polls. [CLI following](https://arkoren.dev/docs/guides/cli/waiting-and-following.md) reports
loss and restart notices, but does not reconnect or persist cursor checkpoints.

## Capture Limits

Attach application capture once during composition. Application telemetry
recording starts after successful composition; event metadata capture starts
when its listener is attached. New capture defaults are:

| Signal | Retained Capacity |
| --- | --- |
| Spans | 2000 finished spans. |
| Logs | 5000 records. |
| Metrics | 1000 distinct label combinations per instrument. |
| Events | 500 metadata records. |

Zero `CaptureLimits` fields select defaults; negative fields disable retention.
Existing configured memory telemetry retains its own capacities. Standalone
memory stores follow their builder limits and require their own lifecycle;
calling their tool factory does not start them. Capture does not configure an
exporter or itself start the primary telemetry driver.

These are bounded, process-local observations, not a durable audit trail or
export. Tools do not show unfinished spans and cannot recover unrecorded or
evicted evidence. Event
payload values are excluded, but names, types, and trace IDs still disclose
application structure. Route metadata exposes handler identities and source
paths. Log text, telemetry attributes, and approved health details can contain
private data; protect the endpoint and saved output. Follow
[Telemetry Privacy](https://arkoren.dev/docs/guides/observability/telemetry-privacy.md) when instrumenting
application code.

Custom diagnostics use ordinary [MCP tool definitions](https://arkoren.dev/docs/guides/mcp/tools.md), not a
separate inspector extension system. Factory declarations are in the
[application](https://pkg.go.dev/arkoren.dev/core/application),
[memory telemetry](https://pkg.go.dev/arkoren.dev/core/service/telemetry/memorytelemetry),
[router](https://pkg.go.dev/arkoren.dev/core/http/router), and
[schedule](https://pkg.go.dev/arkoren.dev/core/service/schedule) APIs.

---

Source: https://arkoren.dev/docs/guides/cli/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/cli/overview)

# Overview

Use `arkoren` when you need to inspect an application from your terminal or
prepare local files before deployment. Remote inspection calls the application's
MCP endpoint; project, key, certificate, and bundled-skill commands do not need a
running application.

## Remote Tools

A remote command comes from the endpoint's current tool catalog. Each invocation
negotiates MCP and reads `tools/list`; there is no saved catalog or separate
inspection protocol or Arkoren core-version handshake. The advertised schema
supplies input flags and help.
Optional CLI metadata supplies names such as `public:routes`; exact MCP names
remain callable without that metadata.

Discovery and execution are separate. `arkoren tools` lists available tools, and
remote `--help` describes them without invoking a handler. Calling a discovered
command sends `tools/call`. The server authenticates the caller and authorizes
the operation; a catalog entry is not permission to execute it.

`arkoren status` goes beyond discovery: it invokes uniquely advertised identity
and health roles. Those roles do not guarantee read-only handlers, and the report
is not a readiness gate. Review an endpoint's contracts before calling its tools.

## Local Tasks

Use [project creation](https://arkoren.dev/docs/guides/first-steps/creating-a-project.md) to prepare a starter
application, or [install the bundled skill](https://arkoren.dev/docs/guides/agent-skills/working-with-agents.md)
for an agent working in an existing project. These commands use local assets,
not MCP discovery.

[Key Commands](https://arkoren.dev/docs/guides/cli/keys.md) provision application keys without printing private
material unless you explicitly request an export. [Certificate Commands](https://arkoren.dev/docs/guides/cli/certificates.md)
generate local TLS identities without installing trust. Root trust installation
and removal are separate, explicitly approved operations that change a system
store; they are not part of ordinary file generation.

## Installing Arkoren

With Go 1.27 or later, install the executable:

```sh
go install arkoren.dev/cmd/arkoren@latest
```

Go places `arkoren` in `GOBIN`, or its default binary directory. Add that
directory to `PATH`. Check which CLI is installed without reading endpoint
configuration or connecting to a server:

```sh
arkoren --version
```

The result is JSON containing `version`. To read the local command help, run:

```sh
arkoren --help
```

Help is text. Bare `arkoren` prints the same offline root help. To track the CLI
as a module tool instead of installing it on `PATH`, run
`go get -tool arkoren.dev/cmd/arkoren@latest` in your application module and invoke
it as `go tool arkoren`.

## Connecting Next

If you already have a trusted endpoint and an accepted credential, continue with
[Connecting](https://arkoren.dev/docs/guides/cli/connecting.md), then [Calling Tools](https://arkoren.dev/docs/guides/cli/calling-tools.md). If your
application does not expose diagnostics yet, start with
[Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md). The
[CLI Reference](https://arkoren.dev/docs/guides/cli/reference.md) owns command and flag lookup.

---

Source: https://arkoren.dev/docs/guides/cli/connecting.md
[HTML documentation](https://arkoren.dev/docs/guides/cli/connecting)

# Connecting

Before calling a tool, establish which server will receive your credential.
Start with the [installed CLI](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren), an existing MCP
endpoint, and a token that endpoint accepts. If you own the application but have
not exposed diagnostics, follow [Capture Setup](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md)
first.

## Select Destination

Use the MCP mount URL, such as `https://admin.example.com/mcp`, not a tool URL.
There is no default destination. The CLI accepts HTTP or HTTPS URLs with a host,
but rejects userinfo, query strings, and fragments before requesting anything.
Credentials belong in the token setting, never in the URL. Invalid-URL errors
omit the supplied value, including in verbose mode.

Keep private endpoints on loopback or a private network. Use HTTPS beyond
loopback or a protected tunnel. Server-side authentication, TLS, allowed Host
values, and Origin validation remain the endpoint owner's responsibility;
[Securing MCP](https://arkoren.dev/docs/guides/mcp/securing-mcp.md) explains those boundaries. A terminal
client does not make browser-facing protections unnecessary.

Select the trusted destination explicitly before supplying a credential:

```sh
export ARKOREN_MCP_URL=https://admin.example.com/mcp
```

The CLI normalizes literal trailing slashes but preserves escaped path
separators: `/admin%2Fmcp` does not become `/admin/mcp`. It does not append tool
names or follow HTTP redirects.

## Verify Discovery

Obtain a bearer API key, JWT, or OAuth access token through your application's
existing credential workflow. With that value already in `ACCESS_TOKEN`, supply
it without putting the literal secret in a command argument:

```sh
export ARKOREN_MCP_TOKEN="$ACCESS_TOKEN"
```

The CLI sends `Authorization: Bearer <token>`. It does not log in, refresh tokens,
manage cookies, or automatically configure mutual TLS. Keep tokens out of
version control and shared logs; environment variables and trusted files still
need access protection. `--token` can expose a value in history or process
arguments.

Discover tools without invoking their handlers:

```sh
arkoren tools
```

Check that the JSON report has `inspector.state: "ok"`. A missing or unreachable
endpoint can produce a report with exit 0, so the exit code alone does not prove
discovery succeeded. Successful discovery includes the negotiated `protocol`
and any server-supplied implementation version in `arkoren`, not tool-derived
application identity.

The `tools` array lists commands, descriptions, `kind`, and `follow` support.
These are advertised hints, not authorization or read-only guarantees. An empty
catalog can mean this caller sees no tools. Remote tool help also performs only
discovery; it does not invoke identity, health, or other tools.

## Configuration Sources

URL and token independently select their first nonempty value:

| Setting | First | Second | Third |
| --- | --- | --- | --- |
| Endpoint | `--url` | `ARKOREN_MCP_URL` environment variable | `ARKOREN_MCP_URL` in `./.env` |
| Credential | `--token` | `ARKOREN_MCP_TOKEN` environment variable | `ARKOREN_MCP_TOKEN` in `./.env` |

Empty values do not clear lower-precedence values. `ARKOREN_INSPECT_TOKEN` is
not an alias. Without a URL, tool calls and `status` fail with exit 2; `tools`
instead reports `inspector.state: "not configured"`.

Only those two keys are read from the current directory's `.env`; parent
directories are not searched. The file may be absent. Its syntax supports
`KEY=VALUE`, an optional `export ` prefix, quoted values, and `#` comments.
There is no interpolation: `$HOME` stays literal. Double quotes support `\n`,
`\"`, and `\\` escapes.

`--env-file PATH` replaces `.env`, and the named file must exist. Relative paths
resolve from the working directory, without fallback to `.env`.
`--env-file ''` disables file loading, not environment variables or flags.

## Approve File URLs

A checkout can supply its own `.env`. If that implicitly loaded file's URL would
receive a token selected from your environment or `--token`, the CLI exits 2
before any network request. It does not discard the inherited token or fall
back to a file token.

Prefer pinning a trusted URL with `--url` or `ARKOREN_MCP_URL`. If you instead
want the file to choose the destination, review it before explicitly approving
it for this invocation:

```sh
arkoren --env-file ./trusted-admin.env tools
```

This permits the file's URL to receive an environment or flag token; it does not
change precedence. Explicit `--env-file .env` is approval too. Never add it
blindly to bypass a refusal in an untrusted checkout. If a previous client sent
a token to an unintended destination, rotate that token.

A URL and token both selected from the same file work without extra approval.
An implicit file URL without any token also works. Keep paired configuration in
a trusted, untracked file when using that arrangement.

## Request Status

Use `status` only when you intend application handlers to run. It invokes the
uniquely advertised `identity` role, then the uniquely advertised `health` role,
once each with `{}`. Missing or ambiguous roles are skipped. Review their
contracts first: a role does not guarantee read-only behavior.

```sh
arkoren status
```

The report includes configuration sources without printing the token. Identity
can add application and process details and override the reported server version.
An identity failure stops before health or report output. A health failure is
included in the report without itself failing the command; `status` is not a
readiness gate. Use a [condition wait](https://arkoren.dev/docs/guides/cli/waiting-and-following.md#gate-readiness)
for that purpose.

## Resolve Failures

An actual HTTP 401 exits 4: obtain or replace the accepted credential. An empty
catalog or admission denial requires checking server policy, not guessing tool
names. TLS verification, networking, and protocol failures are separate from
authentication; consult [exit codes](https://arkoren.dev/docs/guides/cli/reference.md#exit-codes).

For private HTTPS, configure the intended
[client trust](https://arkoren.dev/docs/guides/http-networking/certificates.md#choose-client-trust). Do not
disable certificate verification to hide a trust mismatch. CLI help and version
output are offline and cannot verify TLS or endpoint availability.

If an admin listener is loopback-only on another machine, forward its port
instead of publishing it publicly:

```sh
ssh -L 9090:127.0.0.1:9090 user@host
```

While the tunnel is open, explicitly select `http://127.0.0.1:9090/mcp` and
supply the endpoint's existing credential. The tunnel does not replace endpoint
authentication. Once discovery succeeds, continue with [Calling Tools](https://arkoren.dev/docs/guides/cli/calling-tools.md).

---

Source: https://arkoren.dev/docs/guides/cli/calling-tools.md
[HTML documentation](https://arkoren.dev/docs/guides/cli/calling-tools)

# Calling Tools

When you need an application observation or action, choose it from the current
MCP catalog rather than assuming a built-in command exists. Start with a
[verified connection](https://arkoren.dev/docs/guides/cli/connecting.md). The endpoint supplies tools and schemas on
every invocation and authorizes each call; CLI help grants no permissions.

## Choose Tools

List what your caller can discover:

```sh
arkoren tools
```

This initializes MCP and reads `tools/list` without invoking any tool, including
identity or health. The JSON catalog includes `command`, description, `kind`,
and `follow`, plus `native_name` when it differs from the CLI spelling.
`kind: "read"` reflects the owner's read-only annotation, not an enforced
guarantee about its handler.

Optional [CLI metadata](https://arkoren.dev/docs/guides/mcp/tools.md) maps a name such as `public.routes` to
`public:routes`. Without metadata, use the exact MCP name. Exact names remain
callable and take precedence over matching aliases. If several tools advertise
one alias and no exact name matches, the alias fails with exit 2 before a call;
choose an exact name from the catalog instead.

Local command names win unqualified: `arkoren new` creates a project, not a
remote tool call. There are no built-in diagnostic shortcuts, and tool spelling
alone does not declare positional arguments or following support.

## Read Help

Suppose discovery lists the router command `public:routes`. Inspect its inputs
before calling it:

```sh
arkoren public:routes --help
```

Help is text generated from the current schema and metadata. It shows flags,
declared positionals, owner-supplied examples, output schema when present, and
follow support. It performs discovery only. Use `tools --verbose` if you need
the advertised schemas and metadata as JSON instead of a command manual.

## Pass Inputs

To find registered POST routes whose patterns start with `/users`, call the
discovered router tool:

```sh
arkoren public:routes --method POST --path /users
```

This invokes the listing tool, not a POST route handler. Its JSON result contains
matching registrations in `items`; an empty list means no registrations matched
on that router. Connection setup does not invoke identity as an extra step.

Flags map to schema properties. For a nested input or a schema outside generated
flag support, pass one JSON object. The exact MCP name for this example is
`public.routes`:

```sh
arkoren public.routes --input '{"method":"POST","path":"/users"}'
```

This sends the same filters. If you keep that object in `routes.json`, load it
from the working directory instead:

```sh
arkoren public.routes --input @routes.json
```

`--input -` reads the object from stdin. Repeated `--input` selects only the
final source; earlier sources are not read or merged. Loaded bytes stay JSON
data, never another file or stdin selector.

Base JSON is applied first, metadata-declared positionals next, then flags in
command-line order. Later values override earlier ones. Snake_case properties
also accept kebab-case flags, but exact property names win. The
[input reference](https://arkoren.dev/docs/guides/cli/reference.md#input-syntax) covers reserved names, nested paths,
arrays, literal map keys, and tokens that look like flags.

## Interpret Results

A successful non-follow call prints one JSON document, without an MCP envelope.
Structured content is preferred. A single text block containing JSON prints as
JSON; other text prints as a JSON string. Multiple or multimodal blocks retain
their protocol content records.

For an uncolored, one-line result suitable for capture, add presentation flags:

```sh
arkoren public:routes --method POST --path /users --compact --no-color
```

These flags do not change the request or the tool's successful result. Check
the process exit status even when stdout contains a document: a direct
health-role call prints its report before exiting 1 for `status: "failing"`.
An immediate readiness observation can instead succeed with `ready: false`;
use a [condition wait](https://arkoren.dev/docs/guides/cli/waiting-and-following.md#gate-readiness) to gate work.

## Handle Failures

> [!WARNING]
> Calling a mutating tool executes it without a confirmation prompt. A lost
> response, output-validation failure, or broken stdout does not prove rollback.
> Check application state and use domain idempotency before deciding to retry.

Ordinary calls do not retry automatically. An input-required MCP response fails
with exit 1 after one call, without a successful stdout result. The CLI does not
fulfill the request, continue automatically, or replay the handler.

Errors appear on stderr as JSON with an `error` object. Server validation
failures can include JSON pointers and the flags or arguments that supplied
them. Choose `--verbose` on the original invocation if you need complete
captured safe diagnostics; never replay a mutation merely to get more detail.

Approved Arkoren failures use result `_meta["arkoren.dev/failure"]`, not
successful content. Managed tools redact internal handler and availability
failures; their [safe-failure contract](https://arkoren.dev/docs/guides/mcp/tools.md#safe-failures)
does not extend to arbitrary external servers. Treat captured results as
application data with the appropriate access controls.

Stdout failure exits 1 and takes precedence over other verdicts. Already printed
bytes can be partial and are not proof of success. See the
[exit-code reference](https://arkoren.dev/docs/guides/cli/reference.md#exit-codes) before using results in scripts.

## Number Precision

Input JSON retains its number tokens, and numeric flags avoid floating-point
rounding; decimal integer flag spellings are normalized. After a successful
SDK response decode, captured wire JSON also preserves numbers in generic
structured results, unknown domain fields, catalog schemas and examples, and
approved failure details. Formatting can change whitespace and key order; this
is not a byte-for-byte protocol dump.

The SDK still owns decoding and protocol failures. Capture cannot recover a
response the SDK rejects or extend its supported numeric range. Token fidelity
is not output-schema enforcement and cannot repair precision already lost in
application values. Managed servers enforce their own
[numeric validation limits](https://arkoren.dev/docs/guides/mcp/tools.md#schema-limits);
use string-valued fields when your domain or other clients require them.

For a route-to-trace investigation, continue with
[Application Diagnosis](https://arkoren.dev/docs/guides/diagnostics/application-diagnosis.md). For tool
declarations, see the [MCP API](https://pkg.go.dev/arkoren.dev/core/mcp).

---

Source: https://arkoren.dev/docs/guides/cli/waiting-and-following.md
[HTML documentation](https://arkoren.dev/docs/guides/cli/waiting-and-following)

# Waiting and Following

Deployment checks need an observed condition; live investigations need new
records. A condition wait observes state until it is satisfied. Cursor following
repeatedly reads records after an opaque position. Start with a
[connected CLI](https://arkoren.dev/docs/guides/cli/connecting.md) and tools advertising the required inputs; the
two operations use different meanings of `wait`.

## Gate Readiness

If discovery advertises the application group `app`, wait for readiness before
sending test traffic:

```sh
arkoren app:ready --wait --timeout 30s
```

Proceed only after successful completion. An immediate `app:ready` can exit 0
with `ready: false`; it is an observation, not a shell gate. The wait does not
start the application or run dependency checks.

Choose the condition that matches your next action. `app:started --wait` means
startup completed, not that dependencies are healthy. With registered checks,
`app:healthy --wait` requires a recent successful background pass. Without
checks, health can already be `ok`; that says nothing about unregistered
dependencies. Readiness observes whether the application can receive traffic.

## Choose Failure Policy

Healthy and ready waits normally allow failed checks to recover within the
budget. If deployment should stop on a fresh failed background pass, opt into
fail-fast behavior:

```sh
arkoren app:healthy --wait --timeout 30s --fail-fast
```

An already cached failed pass ends this attempt if it is still fresh. Pending
or stale health keeps waiting. `--fail-fast` requires `wait=true` and is not an
input to `started`. Omit it when recovery within the budget is expected.

These are passive observations. If you intend to execute checks now, use the
explicit `app:health --fresh` operation described in
[Application Diagnosis](https://arkoren.dev/docs/guides/diagnostics/application-diagnosis.md).

## Bound Waiting

The CLI grants a condition-wait budget only when the discovered tool is annotated
read-only, declares a boolean `wait` property, and receives JSON `true` in the
final merged input. `timeout` configures the budget only if its advertised type
is string. Omitted or zero duration selects 30s, not an unlimited wait.

One budget counts from before discovery and input processing through connection
retries, execution, and owned session cleanup. The server receives the remaining
duration, not a fresh full timeout. An explicitly non-string JSON `timeout` is
left for server validation, not repaired. Non-string or undeclared timeout
properties otherwise remain ordinary tool data.

The ordinary 10s per-request deadline is replaced by this total budget unless
you explicitly set `--request-timeout`. An explicit setting can impose a shorter
per-request limit; `--request-timeout=0` removes that limit, not the total budget.
Session cleanup has a separate one-second maximum and also honors cancellation
and the remaining budget. Cleanup does not replace a selected result or stdout
failure.

Before discovery, conventional-looking arguments provide only provisional
connection hints. Schema parsing still owns tokens: a string property can
consume a literal `--wait` or `--input` as its value without enabling either
behavior. Only the final parsed input determines execution retry eligibility.

Eligible waiting reads can retry broken transports within the budget. Discovery
backoff starts at 50ms, grows by 1.5, and caps at 500ms; execution retries pause
50ms. HTTP, authentication, certificate-verification, protocol, and tool failures
are terminal. Actions and follow calls are never replayed automatically.

Application condition failures include `condition_timeout`, `condition_stopped`,
and `condition_failed`. Their safe details are defined in
[Diagnostic Tools](https://arkoren.dev/docs/guides/diagnostics/diagnostic-tools.md#condition-failures).
A client deadline can expire before the server's failure response arrives, so
handle ordinary timeout errors too.

## Follow Records

To observe warnings produced while reproducing an issue, start a live tail. This
example assumes discovery advertises `app:logs` with `follow: true`:

```sh
arkoren app:logs --level warn --follow --no-color
```

The first call establishes an empty tail anchor; it does not print existing
history. Subsequent `tools/call` requests send the returned opaque cursor with
an idle wait of at most 25s. Available pages drain immediately. This is long
polling, not a separate diagnostic SSE endpoint.

Follow prints compact newline-delimited JSON (NDJSON), one record per line,
without page envelopes or saved cursor checkpoints. Loss and generation changes
produce notice objects on the same stdout stream. For example, separate polls
might report:

```json
{"dropped":12}
{"instance_changed":true}
```

Consumers must distinguish notices from application records. A notice can also
contain `truncated: true`, and one notice can combine fields. A replaced or reset
store establishes a fresh tail rather than replaying the new generation's
history.

`--request-timeout` does not limit follow requests. Interrupt the process to
stop it. A broken connection ends follow instead of silently reconnecting;
restarting the CLI establishes a new tail and can miss records between runs.

## Retain History

If you need recent context, request a bounded snapshot before reproducing the
problem:

```sh
arkoren app:logs --level warn --limit 100
```

This prints one JSON result, not a live stream. A snapshot followed by a new
`--follow` invocation is not an atomic handoff: records can fall between them.
For controlled continuation, use ordinary calls with `--cursor` and retain the
returned cursor according to the
[cursor contract](https://arkoren.dev/docs/guides/diagnostics/diagnostic-tools.md#cursor-pages).
The CLI's `--follow` mode always chooses a fresh tail rather than resuming your
saved position.

Capture limits, ring eviction, and caller visibility bound the available
evidence. Empty history does not prove an operation never occurred. Protect
captured output like application logs.

## Check Completion

Use a condition wait rather than `status` as a readiness gate. Check the process
exit status even if stdout already contains data. Clean follow cancellation
exits 0; a broken pipe or short write exits 1 and takes precedence over
cancellation or the remote verdict. Output failure does not undo remote effects.

For all command flags and statuses, use [CLI Reference](https://arkoren.dev/docs/guides/cli/reference.md).

---

Source: https://arkoren.dev/docs/guides/cli/reference.md
[HTML documentation](https://arkoren.dev/docs/guides/cli/reference)

# CLI Reference

Local commands have fixed syntax. Remote commands come from the selected MCP
endpoint on every invocation; use `arkoren tools` and the selected command's
`--help` for its current contract. [Connecting](https://arkoren.dev/docs/guides/cli/connecting.md) owns credential
and destination policy; [Calling Tools](https://arkoren.dev/docs/guides/cli/calling-tools.md) demonstrates a call.

## Command Forms

Uppercase operands below are values you supply, not literal words.

| Form | Contract |
| --- | --- |
| `arkoren`, `arkoren -h`, `arkoren --help` | Offline root help as text; no endpoint configuration is read. |
| `arkoren --version` | Installed CLI version as JSON, offline. |
| `arkoren new DIR [flags]` | Create a starter and resolve Go dependencies; `--skip-tidy` creates files offline. Existing files are never overwritten. |
| `arkoren skills install\|update\|status [flags]` | Manage the CLI's bundled documentation offline. |
| `arkoren key generate TYPE --out FILE` | Generate `shared`, `signing`, or `exchange` private material. |
| `arkoren key check TYPE FILE` | Validate stored key material without exporting it. |
| `arkoren key public signing\|exchange FILE [--format pem]` | Export SPKI public-key PEM. |
| `arkoren key public signing FILE --format vapid` | Export a VAPID public key. |
| `arkoren certificate root generate --name NAME --cert FILE --key FILE` | Generate a root CA pair without installing trust. |
| `arkoren certificate generate --ca-cert FILE --ca-key FILE --dns HOST --cert FILE --key FILE` | Generate a server identity; repeat `--dns` or `--ip IP` for explicit SANs. |
| `arkoren certificate check --cert FILE [--key FILE]` | Parse public PEM and optionally check the matching private key. |
| `arkoren certificate root trust FILE [--yes]` | Explicitly install one public root in the supported system store. |
| `arkoren certificate root untrust FILE [--yes]` | Remove trust for that exact root in the selected system scope. |
| `arkoren tools [--json]` | Discover tools as JSON without invoking them; no positional target is accepted. |
| `arkoren status` | Explicitly call uniquely advertised identity and health roles for a report. |
| `arkoren NAME --help` | Discover one remote tool's schema without invoking it. |
| `arkoren GROUP: --help` | Describe an advertised CLI group, unless an exact native name takes precedence. |
| `arkoren NAME [flags] [arguments]` | Invoke an exact advertised MCP name. |
| `arkoren GROUP:OPERATION [flags] [arguments]` | Invoke a metadata-derived alias. |

Local command help is offline, including `tools --help` and `status --help`.
Ask for a leaf's flags without reading configuration or accessing key material:

```sh
arkoren key generate --help
```

Use attached `-h` or `--help`; there is no built-in `help` command. Remote help
needs the endpoint and obeys the same destination-approval rules as execution.
Connection setup, `tools`, and remote help initialize MCP and read `tools/list`
only; they never call identity, health, or other tools.

## Command Names

Unqualified local names win. Exact native MCP names are tried before CLI aliases
and remain callable when an alias exists. A metadata alias uses exactly one
colon with a nonempty group and operation; two-word remote invocation is not
supported. Positionals exist only when metadata declares their order.

If several tools advertise one alias and no exact name matches, dispatch exits
2 before calling anything. Choose an exact name from `tools`; `native_name` is
included when different from `command`. Catalog commands are sorted and chosen
to avoid shadowed aliases. There are no built-in diagnostic aliases.

## Global Flags

Reserved global flags can appear anywhere before `--` and take precedence over
schema-derived flags. One or two leading dashes work; flags do not bundle.

| Flag | Default | Effect |
| --- | --- | --- |
| `--url URL` | No URL | Select the MCP mount; otherwise environment, then file. |
| `--token TOKEN` | No token | Supply an existing bearer credential; otherwise environment, then file. |
| `--env-file PATH` | Implicit `./.env` | Replace the file; an explicit nonempty path must exist and approves its URL for inherited tokens. Empty disables file loading. |
| `--compact` | Off | Print ordinary JSON documents on one line. |
| `--verbose` | Off | Include complete captured safe diagnostics, schemas, local file lists, and status/build detail. |
| `--no-color` | Off | Disable terminal coloring and bold help headings. Nonempty `NO_COLOR` also disables them. |
| `--request-timeout DURATION` | `10s` | Per-request deadline for ordinary requests; `0` removes that deadline. Waiting reads normally use their total budget. Does not limit follow. |
| `-f`, `--follow` | Off | Follow only tools advertising cursor support. |
| `-h`, `--help` | Off | Describe the selected command. |

`wait`, `timeout`, and `fail_fast` are tool inputs, not global flags. A property
named `verbose`, `input`, or another reserved global cannot be set by its own
generated flag. Supply it in the base object instead: for a tool with a boolean
`verbose` input, add `--input '{"verbose":true}'` to the discovered command.

`--verbose` changes presentation, not remote request bodies, successful tool
payloads, follow records, authorization, or raw exports. Use it on the original
invocation; do not replay a mutation to obtain diagnostics.

Session cleanup has its own one-second maximum, including with
`--request-timeout=0`. Caller cancellation or the remaining condition-wait
deadline can end cleanup sooner. This bounds cleanup, not tool execution;
cleanup errors cannot replace a completed verdict or stdout failure.

## Local Options

`new` accepts flags before or after `DIR`:

| Flag | Default or effect |
| --- | --- |
| `--module PATH` | Destination basename if omitted. |
| `--skills-dir PATH` | Project-relative `.agents/skills`. |
| `--no-agent-files` | Omit bundled skills and `AGENTS.md`. |
| `--dry-run` | Report the plan without writing. |
| `--yes` | Approve a nonempty directory, never a generated-file collision. |

See [Creating a Project](https://arkoren.dev/docs/guides/first-steps/creating-a-project.md) for filesystem
safety and dependency setup. A conflicting preview prints one stdout report and
exits 2. Non-preview conflicts go to stderr with `conflict_count` and
`destination`; neither path writes files. Add `--verbose` for per-path lists.

`skills install`, `skills update`, and `skills status` accept `--dir PATH`
(default `.agents/skills`, with `arkoren` appended) and `--dry-run`, before or
after the subcommand. They do not fetch from GitHub, read endpoint settings, or
edit `AGENTS.md`. Install requires an empty destination or matching unchanged
installation; update requires a managed installation. Local changes and
new-path collisions fail with exit 2 before writes. Status reports differences
without failing merely because files changed. See
[Working with Agents](https://arkoren.dev/docs/guides/agent-skills/working-with-agents.md) for recovery and
ownership.

[Key Commands](https://arkoren.dev/docs/guides/cli/keys.md) owns private exports and exclusive file creation.
[Certificate Commands](https://arkoren.dev/docs/guides/cli/certificates.md) owns file-pair safety and supported trust
stores. Certificate generation never installs trust. Trust operations require
default-no terminal confirmation or bare `--yes` noninteractively; `-yes` and
value-bearing approval forms are invalid. The CLI never elevates itself.

## Input Syntax

Each remote call sends one JSON object. Merge order is base `--input`, then
metadata-declared positionals, then flags in command-line order. Later assignments
override earlier values at the same path; repeated scalar-array flags append.
Traversing a value of the wrong object/array shape is an error.

`--input` accepts inline JSON, `@file.json`, or `-` for stdin. Relative files
resolve from the working directory. If repeated, only the final schema-owned
source is read, once, including for waits. Overridden sources are not opened or
merged. Materialized bytes are data: contents beginning with `@` or equal to `-`
fail JSON parsing rather than selecting another source.

The following forms apply only when the advertised schema declares the
corresponding property and type:

| Form | Meaning |
| --- | --- |
| `--address.city Paris` | Nested object property. |
| `--labels.env prod` | Literal map key `env`. |
| `--items.0.sku A` | Array element; indexes must be dense from 0 and no flag index can exceed 10000. |
| `--tag a --tag b` | Append two scalar-array elements. |
| `--tag '["a","b"]'` | Set the whole array from JSON. |
| `--items '[{"sku":"A"}]'` | Set an object-array value from JSON. |
| `--enabled` | Boolean `true`; `--enabled=false` or `--enabled false` sets false. |

String values stay strings; integers parse as decimal without a machine-size
limit, and numbers accept JSON syntax without floating-point conversion.
Objects and arrays parse as JSON. The CLI checks conversion, not the server's
validation rules. Validation errors can map back to their supplying inputs.

`-x value` and `--x=value` are accepted. `--` ends flag parsing; a bare `-` or
negative number can be positional data. Snake_case properties accept kebab-case
aliases, but exact properties win and map keys stay literal. Map keys containing
dots and properties colliding with reserved names require `--input`. For schemas
outside generated-flag support, use the exact tool name with `--input`.

The schema owns flag arity. A string flag followed by `--wait` or `--input`
consumes that token as a literal value. Provisional discovery hints do not
override parsing; execution deadlines and retry eligibility use the final
merged input and advertised types.

Cancellation stops waiting for stdin without closing caller-owned input. An
arbitrary blocked reader can outlive the canceled command; its read goroutine
ends when the owner supplies EOF or an error. The CLI cannot forcibly interrupt
that reader.

## Output Formats

| Output | Format |
| --- | --- |
| Ordinary local or remote result, including `--version` | One JSON document. |
| `--follow` | Compact NDJSON records and loss notices on the same stdout stream. |
| Help and interactive confirmation | Text. |
| `key public` and `key generate --out -` | Raw PEM or base64, not JSON strings. |
| Errors | JSON with an `error` object on stderr. |
| Trust warning before mutation | JSON with a `warning` object on stderr, even with `--yes`. |

Ordinary JSON is indented by default. Terminal output adds color unless
disabled; pipes do not. `--compact` changes indentation, not content. Help uses
bold terminal headings and plain piped headings; remote output schemas follow
the same color policy. Stderr independently checks whether its own stream is a
terminal. Use `--compact --no-color` for one-line JSON diagnostics in captures.

Remote results prefer structured MCP content with sorted keys. A single text
block parses as JSON when valid, otherwise it becomes a JSON string. Multiple
or multimodal blocks retain protocol content records. Generic results preserve
unknown domain fields and exact wire number tokens after successful SDK
decoding, without an MCP envelope. This does not bypass SDK decoding limits,
enforce output schemas, or repair precision lost in application values; see
[Number Precision](https://arkoren.dev/docs/guides/cli/calling-tools.md#number-precision).

Follow emits individual records rather than page envelopes or persisted cursor
checkpoints. Notice fields include `dropped`, `instance_changed`, and
`truncated`; consumers must handle them alongside records. See
[Waiting and Following](https://arkoren.dev/docs/guides/cli/waiting-and-following.md) for gaps and restart behavior.

## Report Fields

`tools` returns `inspector` context and a `tools` array sorted by `command`.
Inspect `inspector.state`, not just the exit status: absent configuration or an
unreachable endpoint can be report data with exit 0. Successful discovery
includes `protocol` and the server implementation version in `arkoren` when
supplied. It never calls identity to obtain `app` or a version override.

Each tool summary contains `command`, optional description, `kind` (`read` or
`action`), and `follow`. `native_name` appears when different. Descriptions can
be abbreviated with `description_truncated`; verbose discovery includes full
descriptions, input/output schemas, positionals, examples, and semantic roles.
It is not a raw MCP catalog or local CLI manual. `tools --json` is a compatibility
spelling of the same focused result.

`status` calls a uniquely advertised identity role before a uniquely advertised
health role, once each with `{}`; missing or ambiguous roles are skipped.
Identity failure stops before health or output. Health failure appears as data
and does not itself fail status. The report retains configuration sources,
negotiated protocol, implementation version, optional process `instance`, and
health. A string identity `arkoren` overrides the implementation version for
this report only. `app` selects name, version, environment, and state when
returned; verbose mode keeps full identity/runtime data. Health lists become
status and check/failing/pending counts; verbose mode keeps full details.
Neither mode makes extra role calls. Arkoren's health tool with `{}` observes
cached checks; external servers define their own role semantics.

Local reports use these fields rather than completion prose:

| Command | Result fields |
| --- | --- |
| `new` | `created`, `destination`, `module`, `file_count`; previews include `dry_run`, `yes`, `conflict_count`. Verbose adds paths and preview conflicts. |
| `skills` | `directory`, `state`, `bundled_revision`, and installed revision/modification count when managed. Install/update add operation, preview state, write/remove/conflict counts; completed writes set `completed`. Verbose adds per-file details. |
| `key generate` to file | `created`, `category`, absolute `path`; no private contents. |
| `key check` | `valid`, `category`, absolute `path`. |
| Certificate generation | `created`, `certificate`, `key`, `trust_changed: false`. |
| `certificate check` | `certificate`, optional `key`, `parsed`, `key_matched`, `expiry_checked: false`, `trust_checked: false`. |
| Root trust/untrust | `action`, `completed`, `sha256`, `target`. |
| `--version` | `version`; verbose adds Go/module and available revision identifiers, not dependency or compiler-setting dumps. |

## Failure Diagnostics

`error.message` is present in emitted error reports. Optional context includes
`type`, actual HTTP `status`, safe `details`, and a `hint`; empty values and
duplicate protocol title/detail messages are omitted. Validation `fields` retain
JSON pointers; `sources` map them to supplied flags or positionals where known.
For example, `/address/city` can map to `-address.city`. Validation fields are
not duplicated in `details`. Embedded discovery and health errors use this same
shape.

Default diagnostics bound long text and show at most three validation fields
with one message each. `truncated`, `omitted_fields`, and `details_omitted` expose
abbreviation. Small safe context stays visible; `--verbose` includes complete
captured safe context, not raw headers or protocol traces. Neither mode prints
the configured bearer token or implicit private-key contents.

Managed Arkoren failures can use `_meta["arkoren.dev/failure"]`; internal handler
and availability details are redacted. Operational input-check errors and
resource/cancellation failures during output validation remain internal failures,
not user validation failures. Managed plain `ToolError` text is capped at 4096
UTF-8 bytes at a code-point boundary; malformed UTF-8 becomes `tool failed`, and
an approved empty message remains empty at the server boundary. These server
guarantees do not apply to arbitrary native or external tools. See
[safe failures](https://arkoren.dev/docs/guides/mcp/tools.md#safe-failures).

An input-required response exits 1 after one call, without successful stdout.
Automatic continuation is disabled. Failure after a mutation does not establish
rollback. A stdout write failure can leave partial bytes, exits 1, and takes
precedence over all other verdicts, including clean follow cancellation.

## Exit Codes

| Code | Meaning |
| --- | --- |
| `0` | Successful command or clean follow cancellation. A false immediate condition can still be a successful read. |
| `1` | Connection, protocol, execution, structured tool failure, deadline, file/trust operation, or stdout failure; also a direct health-role report with `status: "failing"`. |
| `2` | Invalid usage/configuration, unapproved file URL with an inherited token, missing required endpoint, unknown command, or ambiguous alias. |
| `4` | Actual HTTP 401 authentication failure, not a tool failure with a similarly named code. |

The direct health-role verdict applies after its report is printed; pending or
stale health does not trigger that verdict. `status` does not apply the gate.
A broken pipe is failure even if the remote tool succeeded. Always check process
status rather than treating stdout as a success receipt.

## Retry Rules

Ordinary calls do not retry automatically. Only a read-only tool declaring
boolean `wait` and receiving JSON `true` qualifies for condition-wait execution
retries. Only a declared string `timeout` sets or receives its remaining budget;
omitted/zero duration selects 30s from before discovery and input processing.
Other timeout types remain tool data. Provisional wait hints can permit
discovery transport retries, not execution replay or token ownership.

Discovery backoff starts at 50ms, grows by 1.5, and caps at 500ms. Eligible
execution retries pause 50ms within the remaining budget. HTTP, authentication,
certificate-verification, protocol, and tool failures are terminal. Actions and
follow are never replayed. Lost responses or output-validation failures can
follow completed effects; verify state and use domain idempotency instead of
automatically retrying a mutation.

---

Source: https://arkoren.dev/docs/guides/cli/keys.md
[HTML documentation](https://arkoren.dev/docs/guides/cli/keys)

# Key Commands

Provision keys before deployment so application restarts reuse the intended
identity instead of silently replacing it. These commands use the same validated
[application key types](https://arkoren.dev/docs/guides/keys/overview.md) your application loads. They run
locally without MCP discovery, endpoint settings, or `.env` loading.

Use the [installed CLI](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren) in a protected directory.
Never commit private material, and keep backups and parent-directory access
under your control.

## Choose Material

Choose the type your application's protocol requires:

| Type | Generated material | Ordinary purpose |
| --- | --- | --- |
| `shared` | 32 random bytes, padded standard base64 with a newline | Symmetric encryption or HMAC signing. |
| `signing` | ECDSA P-256, unencrypted PKCS #8 private-key PEM | Signing, including a Web Push VAPID identity. |
| `exchange` | X25519, unencrypted PKCS #8 private-key PEM | Key agreement within an authenticated protocol. |

Generate independent keys for unrelated purposes, even when the type matches.
An exchange identity alone supplies neither peer authentication nor an
encryption protocol. Provider-issued credentials have their own provenance:
an Apple Sign In key must come from Apple, not this generator.

## Generate Keys

For an application's encryption key, select a new explicit destination:

```sh
arkoren key generate shared --out app-key.key
```

Success prints JSON with `created: true`, `category: "shared"`, and the absolute
`path`, never the private contents. There is no implicit destination or overwrite
flag. Existing files and symlinks are refused, and missing parent directories
fail.

The file is created exclusively with POSIX mode `0600`, subject to umask.
On Windows, creation atomically installs a protected discretionary access
control list (DACL) granting access only to the current user, without temporarily
inheriting broader access. Native Windows ACL execution remains unverified.
The CLI does not secure parent directories for you.

If the application also sends Web Push notifications, provision a separate
signing identity:

```sh
arkoren key generate signing --out vapid-private.pem
```

This produces a new P-256 private identity under the same file-safety rules.
For a protocol requiring X25519 instead, use a separate exchange identity:

```sh
arkoren key generate exchange --out exchange-private.pem
```

Each invocation generates independent material. Neither uploads the key, edits
environment files, or replaces an application's deployed credential.

> [!WARNING]
> `--out -` explicitly sends newly generated private material to stdout instead
> of a JSON receipt. Use it only with a protected secret-store pipeline; terminal
> capture, shared logs, or a failed downstream consumer can expose or lose it.

## Check Material

Before configuring the application, check the file as its intended type:

```sh
arkoren key check shared app-key.key
```

Success prints `valid: true`, the selected `category`, and the absolute `path`,
without the secret. Shared material must decode to exactly 32 bytes from padded
standard base64. This checks representation and length, not entropy or
deployment permissions.

For the signing identity, select its type explicitly:

```sh
arkoren key check signing vapid-private.pem
```

Signing requires ECDSA P-256 PKCS #8 PEM; checking `exchange-private.pem` with
`arkoren key check exchange exchange-private.pem` requires X25519 PKCS #8 PEM.
A successful check does not prove provider registration, message delivery, or
peer trust. Missing or malformed files fail without printing private contents.

## Export Public Keys

If a peer needs a standard public key, export only that component:

```sh
arkoren key public signing vapid-private.pem
```

The result is raw SPKI `PUBLIC KEY` PEM, not JSON. `pem` is the default format.
An exchange public key uses the same representation:

```sh
arkoren key public exchange exchange-private.pem --format pem
```

Publishing an exchange public key does not authenticate it. Follow your
protocol's [peer-authentication and derivation requirements](https://arkoren.dev/docs/guides/keys/overview.md#using-exchange-keys).
A shared key has no public component, so `public shared` is invalid.

For Web Push browser registration, request VAPID format instead:

```sh
arkoren key public signing vapid-private.pem --format vapid
```

This prints unpadded base64url for the uncompressed P-256 public point, suitable
for `applicationServerKey`; it does not include the private scalar. Only
`signing` supports `vapid`. Format flags can appear before or after the file;
invalid type/format combinations fail before file access.

## Handle Failures

Exclusive creation is not transactional. A write, sync, or close failure can
leave an incomplete new file. Inspect the reported destination before using or
removing it. Keep established identities unchanged; remove only a newly created
incomplete output after confirming it is not deployed, or choose a fresh path.
Do not regenerate keys on startup to hide a missing configuration file.

Relative paths resolve from the command's working directory. Success exits 0;
invalid usage exits 2; file, validation, or output failure exits 1. A broken pipe
or short stdout write is failure even if a file was already created. Inspect
state before retrying rather than treating missing output as proof no key exists.

Use `arkoren key --help` or `arkoren key generate --help` offline for syntax.
Continue with [Application Keys](https://arkoren.dev/docs/guides/keys/overview.md) to load the provisioned
material in your application.

---

Source: https://arkoren.dev/docs/guides/cli/certificates.md
[HTML documentation](https://arkoren.dev/docs/guides/cli/certificates)

# Certificate Commands

Local HTTPS needs both a server identity and client trust in its issuer. These
are separate decisions: generating a private root CA and a server certificate
never installs trust. Use the [installed CLI](https://arkoren.dev/docs/guides/cli/overview.md#installing-arkoren)
before starting the application. Certificate commands and their help run
without MCP, `.env`, or endpoint settings.

## Generate Roots

> [!WARNING]
> Anyone holding a root's private key can impersonate sites to clients that
> trust that root. Generate as your normal user in a protected directory, keep
> private files and backups out of version control, and restrict parent-directory
> access. Do not elevate generation merely because trust installation needs it.

Choose a recognizable local CA name and two unused output paths:

```sh
arkoren certificate root generate --name "Example Local Development" --cert root.pem --key root-key.pem
```

Success creates a self-signed ECDSA P-256 root and an unencrypted PKCS #8 private
key. Its default validity is 5 × 365 days, and it cannot issue intermediate CAs.
Distribute only the public `root.pem`; keep `root-key.pem` private. The JSON
receipt contains `created: true`, both absolute paths in `certificate` and
`key`, and `trust_changed: false`, never the private contents.

Both destinations are required. Parent directories must already exist without
symlinked parents. Existing outputs, including dangling symlinks, are refused;
the two paths must not identify the same destination, including case-only names
in one parent. `-` does not select stdout. The CLI preflights both paths and
then creates them exclusively, never overwriting established material.

Public files use POSIX mode `0644`; private files use `0600`, subject to umask.
Windows private-file creation atomically installs a protected current-user-only
discretionary access control list (DACL), without temporarily inheriting broader
parent access. Native Windows ACL execution remains unverified. Parent-directory
protection is still your responsibility on every platform.

Pair creation is not transactional. A failure can leave an incomplete file or
only part of the pair; inspect the exact reported paths before retrying as
described under [Recover Failures](#recover-failures).

## Issue Server Certificates

With the root pair available, name every DNS name and IP address clients will
use. For a loopback server:

```sh
arkoren certificate generate --ca-cert root.pem --ca-key root-key.pem --dns localhost --ip 127.0.0.1 --ip ::1 --cert server.pem --key server-key.pem
```

This creates a new ECDSA P-256 server-auth identity: the public chain is leaf
then root, and `server-key.pem` contains its private key. Default validity is
90 days, shortened if the issuer expires sooner. The issuer must be a currently
valid self-signed signing root with matching key and the supported no-intermediate
profile.

At least one explicit subject alternative name (SAN) is required. Repeat `--dns`
or `--ip` rather than supplying comma-separated lists. Names must be bare DNS
names or IP addresses, not URLs or host:port values; no localhost or loopback
SAN is added implicitly. A connection using an unlisted name fails hostname
verification even if the client trusts the root.

The same exclusive-pair and partial-write rules apply. Load the resulting pair
with [Serving HTTPS](https://arkoren.dev/docs/guides/http-networking/server.md#serving-https). Missing files
must fail startup, not trigger replacement identities. These commands do not
renew certificates, create intermediates or client identities, validate DNS
ownership, or contact ACME.

## Check Files

To check that the public root bundle parses, without loading a private key:

```sh
arkoren certificate check --cert root.pem
```

The JSON result has `parsed: true`, `key_matched: false`,
`expiry_checked: false`, and `trust_checked: false`. No private material is
printed. To check the server certificate's matching private key as well:

```sh
arkoren certificate check --cert server.pem --key server-key.pem
```

Success now has `key_matched: true` and includes the key path. A malformed bundle,
mismatched key, or missing file fails. Supported unencrypted PEM identities can
also contain existing RSA, ECDSA, or Ed25519 keys.

These checks establish structure and optional key matching, not validity dates,
hostname, TLS purpose, revocation, or client trust. Verify deployment by making
a request from the intended client with the intended URL and trust configuration.
A file check cannot prove the service is running or a browser trusts it.

## Choose Trust

Prefer [client-specific trust](https://arkoren.dev/docs/guides/http-networking/certificates.md#choose-client-trust)
when only one application needs the root. System trust affects other applications
using that store. Install only a root you control and intend to trust broadly;
neither public-file generation nor structural checking grants that trust.

System trust is not universal. Browsers with separate stores, NSS, Java,
containers, remote machines, and independently configured applications can need
their own setup. Long-running clients can cache roots or TLS connections;
restart them after changes and verify the actual client. Do not disable
certificate verification to conceal a mismatch.

## Install Trust

The input must contain exactly one public self-signed CA certificate, with valid
CA constraints and certificate-signing usage. A private key, arbitrary chain,
leaf, or intermediate is not accepted. Installation also requires current
validity.

To request installation, use the public root only:

```sh
arkoren certificate root trust root.pem
```

Before mutation, stderr receives a JSON `warning` naming the action, SHA-256
fingerprint, concrete target, and system-wide effects. Inspect those values
before answering the text prompt. Confirmation defaults to no; only a complete
`y` or `yes` line approves it, ignoring case and surrounding whitespace.

If a reviewed automation procedure deliberately changes this system store,
approve it explicitly:

```sh
arkoren certificate root trust root.pem --yes
```

Noninteractive use requires bare `--yes`. `-yes`, `--yes=true`, and other
value-bearing approval forms are invalid. The warning still appears with
`--yes`. Success prints JSON with `action`, `completed: true`, `sha256`, and
`target`; it does not prove every client uses that store.

The CLI never invokes `sudo`, requests UAC elevation, or relaunches itself.
If permissions are insufficient, rerun only the trust operation with the
privileges required by the selected store. Keep generation and private-key
access under your normal user.

## Supported Stores

Unsupported systems or Linux layouts fail rather than guessing a location.
Native Windows trust execution remains unverified.

| System | Target and prerequisites |
| --- | --- |
| macOS | `/Library/Keychains/System.keychain`, administrator SSL trust, using `/usr/bin/security`; changes require root. |
| Debian/Ubuntu Linux | Existing `/usr/local/share/ca-certificates` plus executable `/usr/sbin/update-ca-certificates`. |
| Fedora/RHEL Linux | Existing `/etc/pki/ca-trust/source/anchors` plus executable `/usr/bin/update-ca-trust extract`. |
| Windows | Crypt32 operations on the LocalMachine `Root` store; administrative permissions are required for changes. |

Operations match exact DER certificate content and fingerprint, not merely
subject names. Matching installation and established absence are repeatable
without replacing unrelated certificates.

On macOS, installation sets this root's administrator policy to SSL trust.
Removal clears its administrator trust settings but keeps the public System
keychain certificate, avoiding deletion of an item other tools might use.
User-domain and vendor trust are unchanged. Unrecognized or localized
missing-record diagnostics fail closed instead of establishing absence.

On Linux, Arkoren owns only its `arkoren-<SHA-256>.crt` or `.pem` anchor in the
selected directory. Existing anchors must contain the exact canonical bytes;
conflicting files and symlinks are refused. Install and removal run the native
refresh tool even on repeat operations, allowing a repeat to repair failed
refresh. Copies installed elsewhere are not managed. Serialize changes with
other administrators' edits; refreshing the store is not transactional.

## Remove Trust

Retain the exact public root file so removal identifies the same certificate,
not every certificate sharing its name:

```sh
arkoren certificate root untrust root.pem
```

Inspect the warning's fingerprint and target before confirming. Default-no
confirmation and noninteractive `--yes` rules are the same as installation.
Removal accepts an expired root but still validates exactly one self-signed CA.
An established absence succeeds; unknown native diagnostics remain errors.

On Linux, removal affects only Arkoren's anchor. On macOS, it leaves the public
keychain item. Removing trust neither deletes your generated files nor revokes
certificates for clients using separate stores. Verify the resulting behavior
with the client that matters.

## Recover Failures

### Partial Files

After generation failure, inspect only the named output paths. Do not delete an
established CA or assume a reported file is complete. Check any pair you intend
to keep. Remove only new incomplete outputs after confirming they are not
deployed, or choose fresh destinations. There is no automatic overwrite or
rollback.

### Partial Trust

A Linux anchor write, sync, or close failure can leave an incomplete file at the
reported path. The CLI neither refreshes the store nor deletes that pathname.
Inspect it and remove it only after confirming it is incomplete and not deployed;
then retry with the same public root. Conflicting existing anchors remain
protected.

An anchor change can succeed before native refresh fails. The error names the
path, requested state, and refresh command; inspect or repair that state, then
repeat the same operation with the same root. A nonzero exit does not establish
rollback.

Native cleanup can fail after effects too. macOS staging errors distinguish no
attempt, potentially partial change, and completed native commands. Windows
handle-cleanup errors preserve completed or failed outcomes; partial removal can
report how many exact roots were removed. Completed effects remain in place,
and these errors do not produce a success receipt. Inspect the reported outcome
before retrying. There is no certificate status command.

All relative paths resolve from the working directory. Invalid usage exits 2;
file, validation, trust, or output failures exit 1; success exits 0. Broken pipes
and short stdout writes fail even after files or trust changed. Continue with
[TLS Certificates](https://arkoren.dev/docs/guides/http-networking/certificates.md) for application loading
and client-specific trust.

---

Source: https://arkoren.dev/docs/guides/testing/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/testing/overview)

# Overview

Keep the application code you want to verify real, and replace only the dependencies outside the test's scope. Arkoren facades accept drivers directly, so your tests can use an in-process backend or a recording fake without changing the code that calls the facade. Go's standard `testing` package is the test runner; no mocking framework is required.

## Choose Boundaries

A test's assertions determine which dependencies it needs:

| Question | Test boundary |
| --- | --- |
| Does application code store and retrieve the right value? | Real facade with an in-process driver. |
| Does an endpoint return the promised status, headers, and body? | Requests through the router, as shown in [HTTP Tests](https://arkoren.dev/docs/guides/testing/http-tests.md). |
| Does an operation request the right external effect? | Real facade with a [recording or scripted fake](https://arkoren.dev/docs/guides/testing/fakes.md). |
| Does production-specific SQL, expiry, or broker behavior work? | [Integration tests](https://arkoren.dev/docs/guides/testing/integration-tests.md) against that backend. |
| Does a custom driver satisfy the tested public contract? | The service's [conformance suite](https://arkoren.dev/docs/guides/testing/conformance-suites.md). |

A memory driver stores and processes data in your test process. A recording fake captures requests instead of performing the external effect; a scripted fake also supplies chosen replies or failures. Neither proves that a production vendor delivers messages, persists data across crashes, or behaves identically under network failures.

## Test Facades

For cached application state, construct the real cache facade over `memorycache`. This complete test uses `testing`, `arkoren.dev/core/service/cache`, and `arkoren.dev/core/service/cache/memorycache` in a `_test.go` file:

```go
func TestCachedLanguage(t *testing.T) {
	store := cache.New(memorycache.New())
	key := cache.NewKey[string]("preferences:language")
	if err := store.Put(t.Context(), key, "en"); err != nil {
		t.Fatal(err)
	}

	language, err := store.Get(t.Context(), key)
	if err != nil {
		t.Fatal(err)
	}
	if language != "en" {
		t.Fatalf("language = %q, want en", language)
	}
}
```

This establishes the fixture and checks typed retrieval. To test your own operation, pass this `store` into that operation, then assert the state it should leave behind. Give each independent test its own driver so previous tests cannot supply missing setup.

The memory cache checks expiry on access without starting background cleanup. Other drivers have their own lifecycle requirements; follow their setup instructions rather than assuming construction starts them. [Cache Drivers](https://arkoren.dev/docs/guides/cache/drivers.md) owns cache setup, and [Cache Keys](https://arkoren.dev/docs/guides/cache/keys.md) explains typed key definitions.

## Run Tests

From your application's module directory, run:

```sh
go test -short -race ./...
```

`-race` checks executed code for data races and requires a supported race-detector toolchain and C compiler. `-short` is a flag your tests inspect, not an automatic infrastructure filter: tests skip only if they call `testing.Short()` and act on it. [Integration Tests](https://arkoren.dev/docs/guides/testing/integration-tests.md#select-infrastructure) shows that guard and the separate backend run.

For expiry and timers, use [Controlling Time](https://arkoren.dev/docs/guides/testing/time.md) instead of adding real delays to otherwise in-process tests.

## Test SQL

Use a real SQL engine for queries and result mapping. An in-process SQLite database can serve tests whose SQL and schema support SQLite; follow [SQLite setup](https://arkoren.dev/docs/guides/database/drivers.md#sqlite), start the driver, register cleanup, and apply the schema your operation needs before calling it.

SQLite is not a substitute for PostgreSQL or MySQL semantics. Test engine-specific types, locking, isolation, and SQL against the production engine. Keep schema changes application-owned, following [Migrations](https://arkoren.dev/docs/guides/migrations/overview.md), rather than expecting driver startup to create tables.

For your next application test, start with [HTTP Tests](https://arkoren.dev/docs/guides/testing/http-tests.md) for a route or [Fakes](https://arkoren.dev/docs/guides/testing/fakes.md) for an operation that requests an external effect.

---

Source: https://arkoren.dev/docs/guides/testing/http-tests.md
[HTML documentation](https://arkoren.dev/docs/guides/testing/http-tests)

# HTTP Tests

Send an HTTP request through your application's router to test routing, middleware, and handler behavior together. `Record` runs the full router pipeline using an in-memory response recorder: it does not bind a port or make a network call. Your handler still receives a genuine `*http.Request`.

## Record Requests

Reuse your application's [route registration](https://arkoren.dev/docs/guides/http-fundamentals/routing.md#register-handlers) when testing a feature. Inject test dependencies before registering its handlers; otherwise a test can accidentally retain a production backend.

To illustrate the boundary, this small route returns the report ID selected by routing. Import `net/http`, `arkoren.dev/core/http/response`, and `arkoren.dev/core/http/router`:

```go
type reportView struct {
	ID string `json:"id"`
}

func reportRoutes() *router.Router {
	routes := router.New()
	routes.Get("/reports/{id}", func(w http.ResponseWriter, r *http.Request) error {
		return response.JSON(w, http.StatusOK, reportView{ID: r.PathValue("id")})
	})
	return routes
}
```

The route intentionally has no persistence or authorization behavior. In an application test, replace `reportRoutes()` with your production route factory and its test dependencies.

Add `testing`, `net/http/httptest`, and `encoding/json/v2` for the test. Check the HTTP status before interpreting the body, and register body cleanup before assertions can stop the test:

```go
func TestReportRoute(t *testing.T) {
	routes := reportRoutes()
	res := routes.Record(httptest.NewRequest(http.MethodGet, "/reports/monthly", nil))
	t.Cleanup(func() {
		if err := res.Body.Close(); err != nil {
			t.Error(err)
		}
	})
	if res.StatusCode != http.StatusOK {
		t.Fatalf("status = %d, want 200", res.StatusCode)
	}

	var report reportView
	if err := json.UnmarshalRead(res.Body, &report); err != nil {
		t.Fatal(err)
	}
	if report.ID != "monthly" {
		t.Fatalf("report ID = %q, want monthly", report.ID)
	}
}
```

This checks that the path pattern selected the ID and the response encoded it. `Record` returns an `*http.Response`; you own its body even though no socket was opened. Apply the same cleanup to every recorded response below.

## Submit JSON

When testing a create endpoint, send the content type and body a client would send. With `strings` imported and your configured router named `routes`, create a request inside your test:

```go
incoming := httptest.NewRequest(http.MethodPost, "/reports", strings.NewReader(`{"name":"Monthly"}`))
incoming.Header.Set("Content-Type", "application/json")
res := routes.Record(incoming)
```

Assert the endpoint's promised status, response headers, and decoded fields, not merely that it did not panic. The router does not choose a successful creation status for you; your handler must return it. Keep [input validation](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#validate-input) in the production request path so malformed and rejected input exercise the same behavior as real clients.

## Assert Errors

For an endpoint that returns Problem Details, check both the HTTP status and the document. After recording a rejected request and registering body cleanup, import `arkoren.dev/foundation/problem` and decode:

```go
var details problem.Details
if err := json.UnmarshalRead(res.Body, &details); err != nil {
	t.Fatal(err)
}
if res.StatusCode != http.StatusUnprocessableEntity || details.Status() != res.StatusCode {
	t.Fatalf("HTTP status = %d, problem status = %d; want 422", res.StatusCode, details.Status())
}
```

This fragment assumes your endpoint promises HTTP 422 for that input. For validation problems, inspect `details.Extension("errors")` and assert the particular field path that should fail. Do not accept any non-200 response as proof that validation worked: an unrelated internal error could satisfy that assertion. [Error Handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md) explains which failures the router renders and which your application must map.

## Cover Middleware

Keep middleware in the router used by a route test. Assert its visible result, such as a required response header, rejected origin, or denied anonymous request. Registering a trivial handler is useful for testing one middleware in isolation; it does not prove that middleware is installed on the production route.

To test a handler without route matching, use `router.Handler(handler).Record(incoming)`. Returned errors still go through the handler lifecycle. Path values are supplied by route matching, so prefer the router when the handler reads them. [Authentication Tests](https://arkoren.dev/docs/guides/testing/auth-tests.md) distinguishes handler identity fixtures from access-enforcement tests.

## Continue Flows

For a cookie-based flow, keep the response cookies and attach them to the next request. With a successful login response named `login`, and after registering cleanup for its body:

```go
incoming := httptest.NewRequest(http.MethodGet, "https://app.example/me", nil)
for _, cookie := range login.Cookies() {
	incoming.AddCookie(cookie)
}
res := routes.Record(incoming)
```

This explicitly carries cookies but does not emulate a browser's cookie policy. Test login, protected access, logout, and subsequent rejection with the real [session middleware](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#enable-sessions). Check session mutation errors and [save failures](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#save-changes); a successful handler alone does not establish session persistence.

If the test needs automatic cookie scope, redirects, or a real HTTP transport, use `httptest.NewServer` with an `http.Client` and cookie jar instead. Prefer `httptest.NewTLSServer` when retaining secure-cookie defaults. Plain HTTP requires test-only `session.Builder.AllowInsecure(true)` configuration; never carry that setting into production. Register server cleanup and close every client response body.

An in-memory recorder collects output rather than observing its arrival over time. Use a networked test for transport cancellation or incremental delivery, following [Streaming Responses](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md).

For recording and handler behavior, see the [`router` API reference](https://pkg.go.dev/arkoren.dev/core/http/router).

---

Source: https://arkoren.dev/docs/guides/testing/auth-tests.md
[HTML documentation](https://arkoren.dev/docs/guides/testing/auth-tests)

# Authentication Tests

A handler that uses the current user and an endpoint that enforces authentication need different tests. Inject an identity when the question is what the handler does for that user. Send credentials through authentication middleware when the question is who can reach the handler. Neither shortcut replaces a test of your actual login mechanism.

The fragments below run inside tests with `t *testing.T`. Import `context`, `net/http`, `net/http/httptest`, `arkoren.dev/core/http/auth`, and `arkoren.dev/core/http/router` as needed. The examples use this test identity:

```go
type testUser struct {
	ID int
}
```

## Inject Identity

For a handler test, attach the exact user type the handler reads with `request.User[testUser]`. With your handler named `showProfile`:

```go
incoming := httptest.NewRequest(http.MethodGet, "/me", nil)
ctx := auth.NewContext(incoming.Context(), testUser{ID: 42})
res := router.Handler(showProfile).Record(incoming.WithContext(ctx))
```

Register response-body cleanup and assert the returned profile, following [HTTP Tests](https://arkoren.dev/docs/guides/testing/http-tests.md#record-requests). This bypasses credential verification intentionally. It proves neither that anonymous requests are rejected nor that one user may access another user's data. Keep [authorization checks](https://arkoren.dev/docs/guides/authorization/overview.md) in the route tests that exercise those decisions.

User types must match exactly: a `testUser` value does not satisfy a lookup for `*testUser`. `auth.Require` treats a mismatched type as absent and rejects it.

## Exercise Enforcement

When testing the router's authentication and authorization path, supply a test-only authenticator instead of production credentials. This verifier accepts one fixed token; never enable it outside test code:

```go
authenticate := auth.Bearer(func(ctx context.Context, token string) (testUser, error) {
	if token != "profile-test-token" {
		return testUser{}, auth.ErrUnauthenticated
	}
	return testUser{ID: 42}, nil
})
```

Pass it into your route factory so tests reuse production middleware ordering and route registration. The essential wiring, with your `showProfile` handler, is:

```go
routes := router.New()
routes.Use(auth.Middleware(authenticate))
routes.With(auth.Require[testUser]()).Get("/me", showProfile)
```

Send a valid token, register body cleanup, and assert the expected profile as well as HTTP 200:

```go
incoming := httptest.NewRequest(http.MethodGet, "/me", nil)
incoming.Header.Set("Authorization", "Bearer profile-test-token")
res := routes.Record(incoming)
```

In separate cases, omit the header and send a rejected token. With the default configuration both return HTTP 401, but through different paths: missing credentials pass `auth.Middleware` anonymously and `auth.Require` rejects the request; rejected credentials stop at authentication middleware. Assert the challenge header when it is part of your endpoint contract, and assert that protected side effects did not happen.

For an authenticated user without permission, assert your application's authorization denial separately. A successful identity lookup is not permission to access a resource.

## Preserve Login Coverage

Keep a test that submits valid credentials to the real login route, carries its session cookie to a protected route, logs out, and verifies subsequent rejection. [Multi-request HTTP tests](https://arkoren.dev/docs/guides/testing/http-tests.md#continue-flows) explains cookie transfer and when to use a test server. Include invalid credentials and session persistence failure cases relevant to your application.

Share the production `auth.NewSessionIdentity(load)` definition between authentication, login, and logout. Check errors from `identity.Login`, `identity.Logout`, and session field mutations before asserting state; see [Session Login](https://arkoren.dev/docs/guides/authentication/session-login.md) and [Save Changes](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#save-changes).

For a focused session-state test, `session.New(expiresAt, values)` creates a low-level session. Initial values must already have the exact Go types expected by your [typed fields](https://arkoren.dev/docs/guides/http-capabilities/sessions.md#store-values); this initializer does not round-trip values through JSON. It does not test cookies, storage, or middleware persistence. A nil session pointer is absent, not usable state.

If production uses a custom session slot, allocate one `session.NewHandle()` and share it across session middleware, the identity, and request lookup. Attach test state with `session.NewContextFor(ctx, handle, current)`. Separate handles select separate context slots; they do not isolate cookies or storage. Zero handles select the default slot.

## Check Authenticators

For your own authenticator, `authtest.Run` checks the public authentication contract. Import `arkoren.dev/core/http/auth/authtest`. Here `apiKeyAuthenticator` is your implementation, and each application helper returns a fresh `*http.Request` carrying the credentials and context for its case:

```go
func TestAPIKeyContract(t *testing.T) {
	authtest.Run(t, apiKeyAuthenticator, anonymousRequest, validRequest,
		authtest.WithInvalid[testUser](rejectedRequest),
		authtest.WithUser(func(t *testing.T, user testUser) {
			if user.ID != 42 {
				t.Fatalf("user ID = %d, want 42", user.ID)
			}
		}),
	)
}
```

The suite checks that absent credentials yield `auth.ErrNoCredentials`, valid credentials resolve and store a user, and rejected credentials yield `auth.ErrUnauthenticated` and stop middleware. `WithInvalid` enables rejection checks; without it those checks are skipped. Add application tests for credential expiry, revocation, and backend failures beyond that shared contract.

See the [`auth` API reference](https://pkg.go.dev/arkoren.dev/core/http/auth) and [`authtest` suite](https://pkg.go.dev/arkoren.dev/core/http/auth/authtest).

---

Source: https://arkoren.dev/docs/guides/testing/fakes.md
[HTML documentation](https://arkoren.dev/docs/guides/testing/fakes)

# Fakes

An application test can check that registration requests a welcome email without delivering one. A **fake** replaces the driver at that external boundary: the real facade still applies its defaults, validation, and encoding, while the fake records the request or returns a scripted result.

Use an in-process driver instead when the test needs stored state or an executing engine. A recorded workflow start, for example, is evidence that your code requested a run, not that the workflow executed successfully.

## Choose Fakes

Fakes live in each service's test package. These entry points select different kinds of evidence:

| Package | What the fake provides |
| --- | --- |
| `mail/mailtest` | Accepted messages and send-failure injection. |
| `push/pushtest` | Accepted messages and completed retention settings. |
| `queue/queuetest` | Dispatched jobs, typed assertions, and an explicit consumer drain. |
| `broadcast/broadcasttest` | Published messages and an explicit listener drain. |
| `schedule/scheduletest` | Registered entries, deferred tasks, and a drain for deferred tasks. |
| `workflow/workflowtest` | Run commands and manually completed or failed results, not engine execution. |
| `payment/paymenttest` | Recorded requests and operation-specific scripted replies. |
| `llm/llmtest` | Recorded model requests and scripted model replies. |
| `telemetry/telemetrytest` | Retained spans, metrics, and logs. |

Paths in this table are below `arkoren.dev/core/service/`; each package exposes `New()`. Use the service-specific guides for [payment outcomes](https://arkoren.dev/docs/guides/payments/testing.md), [model replies](https://arkoren.dev/docs/guides/llm/testing.md), [push tests](https://arkoren.dev/docs/guides/push/testing.md), and [workflow tests](https://arkoren.dev/docs/guides/workflows/testing.md). Those guides cover the extra scripting and assertions their features require.

## Assert Mail

Keep the operation being tested real. This small application function constructs a welcome message using `context` and `arkoren.dev/core/service/mail`:

```go
func sendWelcome(ctx context.Context, mailer *mail.Mailer, address string) error {
	message := mail.NewMessage(mail.Recipients{
		To: []mail.Address{{Email: address}},
	}, mail.Body{Text: "Your account is ready."}, func(letter *mail.MessageBuilder) {
		letter.Subject("Welcome")
	})
	_, err := mailer.Send(ctx, message)
	return err
}
```

`Mailer.Send` checks the completed message, including the sender supplied by facade defaults, before submitting it. The test supplies those defaults and a recording driver. Add `testing` and `arkoren.dev/core/service/mail/mailtest`:

```go
func TestWelcomeMessage(t *testing.T) {
	fake := mailtest.New()
	mailer := mail.New(fake, func(mailer *mail.Builder) {
		mailer.From(mail.Address{Email: "accounts@example.com"})
	})
	if err := sendWelcome(t.Context(), mailer, "reader@example.com"); err != nil {
		t.Fatal(err)
	}

	fake.AssertSentCount(t, 1)
	fake.AssertSent(t, func(message mail.Message) bool {
		recipients := message.To()
		return len(recipients) == 1 && recipients[0].Email == "reader@example.com" &&
			message.Subject() == "Welcome" && message.From().Email == "accounts@example.com"
	})
}
```

`AssertSent` requires at least one message matching every supplied predicate. The separate count assertion catches duplicate sends. `Sent()` returns recorded messages when you need assertions beyond the helpers. `AssertNotSent` checks that no message matches; `AssertNothingSent` checks that there are no accepted messages at all.

This test proves that the application submitted the intended message through the facade. It does not prove SMTP connectivity, provider acceptance, or delivery to an inbox.

## Inject Failures

Use an injected failure to exercise the caller's response, not only the fake's recording. With `errors` imported, this independent test verifies that the operation preserves a send error:

```go
func TestWelcomeFailure(t *testing.T) {
	fake := mailtest.New()
	mailer := mail.New(fake, func(mailer *mail.Builder) {
		mailer.From(mail.Address{Email: "accounts@example.com"})
	})
	failure := errors.New("mail service unavailable")
	fake.FailNext(failure)

	err := sendWelcome(t.Context(), mailer, "reader@example.com")
	if !errors.Is(err, failure) {
		t.Fatalf("error = %v, want injected failure", err)
	}
	fake.AssertNothingSent(t)
}
```

For this mail fake, validation and context checks happen before consuming the injected failure; a failed send is not recorded as accepted. `FailNext` supplies one failure, while `FailWith` persists until cleared with `nil`. `Reset` clears both recorded mail and injected failures. Prefer a fresh fake per independent test so one case cannot consume another's script.

Other fakes have their own validation, recording, and scripting order. Check that package's contract before interpreting an attempted operation as accepted. Telemetry records observations rather than injecting operation failures.

## Separate Execution

A queue dispatch assertion checks a request to enqueue work. To test its handler as well, register that handler on a real consumer and explicitly drain the queue fake. Import `arkoren.dev/core/service/queue` and its `queuetest` subpackage. Inside your test, construct `fake := queuetest.New()` and pass `queue.NewDispatcher(fake)` to the application operation that dispatches your declared `SendReceipt` job. After that operation succeeds, register its `sendReceipt` handler and drain:

```go
consumer := queue.NewConsumer(fake)
if err := consumer.Handle(SendReceipt, sendReceipt); err != nil {
	t.Fatal(err)
}
if err := fake.Drain(t.Context(), consumer); err != nil {
	t.Fatal(err)
}
```

Assert the handler's observable effect afterward. The drain uses the real consumer's decoding, retry, and failure behavior, but not broker persistence or recovery after process crashes. [Producing Jobs](https://arkoren.dev/docs/guides/queues/producing-jobs.md) owns job declarations and typed payloads.

Delayed jobs remain pending until due unless you use `queuetest.WithScheduled()`, which skips the delay rather than advancing time. By default, a drain permits at most 100 deliveries per message and returns `queuetest.ErrDrainLimit` if that bound is exceeded. For actual timer behavior use [Controlling Time](https://arkoren.dev/docs/guides/testing/time.md).

Do not assume every fake has this execution model. The broadcast fake drains listeners; the schedule fake drains deferred tasks. The workflow fake uses `Complete` or `Fail` to supply a result without executing workflow code. Test engine behavior using the [workflow testing tools](https://arkoren.dev/docs/guides/workflows/testing.md) instead.

## Capture Telemetry

When an operation emits telemetry, pass it a test-owned [shared handle](https://arkoren.dev/docs/guides/observability/telemetry.md#shared-handle). With `telemetrytest` imported from `arkoren.dev/core/service/telemetry/telemetrytest`, and your application operation named `refreshCatalog`:

```go
fake := telemetrytest.New()
tel := fake.Telemetry(t)
fake.DumpOnFailure(t)

if err := refreshCatalog(t.Context(), tel); err != nil {
	t.Fatal(err)
}
fake.AssertSpan(t, "catalog refresh")
```

`Telemetry(t)` returns a started handle and registers its shutdown with the test. Start the fake this way or through application lifecycle wiring, not both; it records nothing before startup. End application spans before asserting their completed contents, following [Recording Spans](https://arkoren.dev/docs/guides/observability/telemetry.md#recording-spans).

Add attribute assertions when they distinguish the behavior under test, and use counter, histogram, gauge, or log assertions for those signals. `DumpOnFailure` prints retained diagnostic data only when the test fails. Construction callbacks configure memory recording limits, not conformance-suite options.

For the examples above, see the [`mailtest` API reference](https://pkg.go.dev/arkoren.dev/core/service/mail/mailtest), [`queuetest` API reference](https://pkg.go.dev/arkoren.dev/core/service/queue/queuetest), and [`telemetrytest` API reference](https://pkg.go.dev/arkoren.dev/core/service/telemetry/telemetrytest).

---

Source: https://arkoren.dev/docs/guides/testing/time.md
[HTML documentation](https://arkoren.dev/docs/guides/testing/time)

# Controlling Time

An expiry test should verify a lifetime, not spend that lifetime waiting. Go's `testing/synctest` runs a test inside a **bubble**, a group of goroutines that share a fake clock. Construct time-dependent drivers inside the bubble so their timestamps and background work use that clock.

Time advances when all goroutines in the bubble are durably blocked: they are waiting on work that can only be unblocked from within the bubble, such as its timers or bubble-local channels. A long `time.Sleep` can therefore finish without that much real time passing. Network I/O and mutex waits do not provide that same durable-blocking guarantee.

## Test Expiry

For a temporary cache entry, assert that the value is available first, then advance time and check absence. Import `errors`, `testing`, `testing/synctest`, `time`, `arkoren.dev/core/service/cache`, and `arkoren.dev/core/service/cache/memorycache`:

```go
func TestPreviewExpires(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		store := cache.New(memorycache.New())
		key := cache.NewKey[string]("preview:report", func(entry *cache.KeyBuilder) {
			entry.TTL(time.Minute)
		})
		if err := store.Put(t.Context(), key, "draft"); err != nil {
			t.Fatal(err)
		}
		if value, err := store.Get(t.Context(), key); err != nil || value != "draft" {
			t.Fatalf("value = %q, error = %v; want draft and nil", value, err)
		}

		time.Sleep(time.Minute)

		if _, err := store.Get(t.Context(), key); !errors.Is(err, cache.ErrNotFound) {
			t.Fatalf("error = %v, want cache.ErrNotFound", err)
		}
	})
}
```

The memory cache treats the entry as expired at its expiry instant. This test does not start a janitor: reads check expiry independently of background cleanup. It verifies this process-local driver, not a server's expiry precision. [Cache Keys](https://arkoren.dev/docs/guides/cache/keys.md) owns TTL configuration and key definitions.

If you start driver background work inside a bubble, register its shutdown inside that bubble too. The bubble must not finish with a worker left waiting indefinitely.

## Wait for Work

Advancing time and finishing work are separate events. `synctest.Wait` waits until the other goroutines in the bubble are durably blocked; it does not assert that an application operation succeeded.

This complete test demonstrates the distinction using a timer-driven notification. A buffered channel captures the observable result without blocking the producer:

```go
func TestTimerNotification(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		completed := make(chan string, 1)
		go func() {
			time.Sleep(time.Hour)
			completed <- "ready"
		}()
		synctest.Wait()
		time.Sleep(time.Hour)
		synctest.Wait()

		select {
		case value := <-completed:
			if value != "ready" {
				t.Fatalf("notification = %q, want ready", value)
			}
		default:
			t.Fatal("timer produced no notification")
		}
	})
}
```

The first wait lets the worker reach its timer; the second lets work made runnable by the elapsed hour settle before the assertion. The nonblocking receive fails rather than waiting for an absent effect.

For a long-running consumer, cancel its context, wait for it to return, and check unexpected runner errors before leaving the bubble. Assert successful handler effects separately from runner shutdown. Use the [workflow testing guide](https://arkoren.dev/docs/guides/workflows/testing.md) for executing workflows and the [scheduler guide](https://arkoren.dev/docs/guides/scheduled-tasks/overview.md) for scheduled execution rather than substituting recording fakes for engines.

## Pin Validation

When the test concerns a `Future` or `Past` rule rather than timers, set validation's context clock. Inside a test, import `arkoren.dev/foundation/validate` and construct a scalar rule with an explicit reference instant:

```go
now := time.Date(2026, 6, 1, 12, 0, 0, 0, time.UTC)
ctx := validate.NewClockContext(t.Context(), func() time.Time {
	return now
})
rule := validate.Time(func(moment *validate.TimeBuilder) {
	moment.Future()
})
if err := rule.Validate(ctx, now.Add(time.Minute)); err != nil {
	t.Fatal(err)
}
if err := rule.Validate(ctx, now.Add(-time.Minute)); err == nil {
	t.Fatal("past time passed a Future rule")
}
```

Both assertions use the same instant regardless of the date on the machine. This context controls validation rules, not the process clock or another service's timers. For structured input, pass it to your existing [validation schema](https://arkoren.dev/docs/guides/validation/overview.md).

## Respect Server Time

A synctest bubble cannot advance Redis, a database server, or a remote workflow engine. Use ordinary time in [integration tests](https://arkoren.dev/docs/guides/testing/integration-tests.md) for server-owned expiry and deadlines. Allow for the backend's documented precision and bound how long the test waits; repeatedly sleeping does not establish that a write or delivery succeeded.

Keep most policy and timer tests in process, and retain real-backend cases for behavior that depends on that backend. The [conformance clock options](https://arkoren.dev/docs/guides/testing/conformance-suites.md#declare-capabilities) make the same distinction.

See Go's [`testing/synctest` reference](https://pkg.go.dev/testing/synctest) and the [`validate` API reference](https://pkg.go.dev/arkoren.dev/foundation/validate).

---

Source: https://arkoren.dev/docs/guides/testing/integration-tests.md
[HTML documentation](https://arkoren.dev/docs/guides/testing/integration-tests)

# Integration Tests

Use a real backend when the behavior you need to verify belongs to that backend: PostgreSQL-specific SQL, Redis expiry, or broker recovery after a restart. Memory drivers and recording fakes cannot establish those semantics. Keep ordinary application-policy tests in process, and give each infrastructure test one specific backend claim.

## Select Infrastructure

Go does not classify integration tests automatically. Put a short-mode guard before construction or connection attempts in each test that requires infrastructure:

```go
if testing.Short() {
	t.Skip("integration test: requires PostgreSQL")
}
```

This fragment belongs at the start of a test function; the complete test below includes it. `go test -short ./...` skips guarded tests, while an ordinary run includes them. If a selected backend is unavailable, fail the test rather than silently skipping it: otherwise a broken setup can look like verified behavior.

## Start PostgreSQL

For the example, use PostgreSQL 17 in a disposable local stack. Match the server version and important settings to your deployed backend when adapting it. Install Docker with Compose support and save this `compose.yaml` in your application's module directory:

```yaml
services:
  postgres:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD: local-test-only
    ports:
      - "127.0.0.1:15432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 1s
      timeout: 3s
      retries: 30
```

The test password is suitable only for this loopback-bound local service. Do not use it for a shared or remotely reachable database. The health check waits for the server; the test still verifies connectivity with its actual credentials.

From that directory, choose a new Compose project name and start the stack. Run all command blocks on this page in the same shell so the project selection remains active:

```sh
export COMPOSE_PROJECT_NAME="reports-integration-$$"
docker compose up -d --wait
```

Use this project name only for disposable test resources. It isolates stack teardown, but not the fixed host port: simultaneous runs need separate ports and matching connection strings, or separate hosts.

## Own Connections

Install the driver in your application's module:

```sh
go get arkoren.dev/driver/postgres
```

In a `_test.go` file, import `context`, `testing`, `time`, and `arkoren.dev/driver/postgres/postgresdatabase`. This helper constructs a pool, immediately registers its cleanup, then verifies connectivity with a bounded context:

```go
func connectPostgres(t *testing.T) *postgresdatabase.Driver {
	t.Helper()
	driver, err := postgresdatabase.New(
		"postgres://postgres:local-test-only@127.0.0.1:15432/postgres?sslmode=disable",
	)
	if err != nil {
		t.Fatal(err)
	}
	t.Cleanup(func() {
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		if err := driver.Stop(ctx); err != nil {
			t.Error(err)
		}
	})
	ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second)
	defer cancel()
	if err := driver.Start(ctx); err != nil {
		t.Fatal(err)
	}
	return driver
}
```

Construction does not prove a server connection; `Start` checks it. Cleanup uses its own context because `t.Context()` is canceled before cleanup callbacks run. Finish transactions and close cursors before stopping the pool. [PostgreSQL setup](https://arkoren.dev/docs/guides/database/drivers.md#postgresql) owns pool configuration and native-client ownership.

## Assert Backend Behavior

This test checks PostgreSQL JSON containment through the database facade. Add imports for `arkoren.dev/core/service/database` and `arkoren.dev/foundation/query`:

```go
func TestPostgresJSONContains(t *testing.T) {
	if testing.Short() {
		t.Skip("integration test: requires PostgreSQL")
	}
	db := database.New(connectPostgres(t))
	ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second)
	defer cancel()
	statement := query.Raw("SELECT ?::jsonb @> ?::jsonb", `{"state":"ready","count":2}`, `{"state":"ready"}`)
	contains, err := db.Find[bool](ctx, statement)
	if err != nil {
		t.Fatal(err)
	}
	if !contains {
		t.Fatal("JSON object did not contain the expected state")
	}
}
```

The query needs no schema or persistent fixture, and cannot be justified by a passing SQLite test. For your application's queries, apply its [migrations](https://arkoren.dev/docs/guides/migrations/overview.md) to a disposable database, seed the necessary rows, and call the real store operation. Driver startup does not create application tables.

## Isolate Fixtures

Tests sharing a server need distinct resource names and ownership. Allocate a unique schema, key prefix, queue, or database for each independent fixture, including across simultaneous test runs; `t.Name()` alone does not distinguish separate processes. Register cleanup as soon as acquisition succeeds, and report cleanup errors with `t.Error`.

Delete only resources that test created. Avoid database-wide resets or prefix scans that might match another test's data. A transaction rolled back after a test can isolate some database work, but it cannot undo broker messages or operations performed through a different connection. Use explicit cleanup for those effects.

Use separate fixtures for interruption, timeout, and recovery tests. State which process or connection you stop, bound waits for the expected result, and assert the application outcome after recovery. A successful reconnect alone does not prove that an interrupted write committed or a message survived.

## Run and Clean

With the stack healthy, run the focused test without reusing a cached result:

```sh
go test -count=1 -race -run '^TestPostgresJSONContains$' ./...
```

`-count=1` forces execution against the current server. A busy port or unhealthy service is a setup failure, not evidence about the driver's semantics. If startup or testing fails, capture logs before removing the stack:

```sh
docker compose logs --no-color
```

After either success or failure, remove this disposable project and its database volumes. This deletes test data; run it only with the same test-only project selection and Compose file used above:

```sh
docker compose down --volumes
```

In CI, make teardown an always-run step and preserve the startup or test command's failure status. Capture logs before teardown on failure. Separate fast `-short` tests from infrastructure jobs, and give each job its own disposable project and available ports.

If the backend implements a custom public driver, add [Driver Conformance](https://arkoren.dev/docs/guides/testing/conformance-suites.md) alongside your application-specific integration tests.

---

Source: https://arkoren.dev/docs/guides/testing/conformance-suites.md
[HTML documentation](https://arkoren.dev/docs/guides/testing/conformance-suites)

# Driver Conformance

When your application needs a [custom driver](https://arkoren.dev/docs/guides/drivers/writing-a-driver.md), test the behavior callers rely on, not only the calls your adapter makes to a vendor SDK. A **conformance suite** runs shared contract checks against your driver. It is also useful for a wrapper that adds behavior such as key prefixes: forwarding most methods correctly does not establish that all operations still agree on the same keys.

A passing suite is evidence for the tested public contract, not a certification of every vendor feature, production setting, or failure mode. Keep application-specific and real-backend tests alongside it.

## Run a Suite

Each service's test package exposes its own `Run` function. This complete cache example uses `testing`, `arkoren.dev/core/service/cache`, and the `cachetest` and `memorycache` subpackages:

```go
func TestCacheContract(t *testing.T) {
	cachetest.Run(t, func(t *testing.T) cache.Driver {
		return memorycache.New()
	})
}
```

Replace the returned driver with your implementation or wrapper. The cache suite calls its factory for each subtest, uses unique keys, and runs subtests in parallel. A fresh memory driver supplies isolated empty state. If your factories share a backend, ensure both the implementation and fixture allocation support concurrent tests.

Read the relevant suite's documentation before copying this factory shape. Signatures, readiness requirements, parallelism, and factory lifetimes differ between services.

## Own Test Resources

Register resource cleanup with `t.Cleanup` on the `*testing.T` passed to the factory. Initialize required connections or fixtures before returning. Use a bounded cleanup context independent of the canceled test context, as shown in [Integration Tests](https://arkoren.dev/docs/guides/testing/integration-tests.md#own-connections).

Do not defer a driver's shutdown in the factory: that closes it when the factory returns, before the suite uses it. A defer in a parent test is also too early for parallel subtests; test cleanup runs after its subtests finish.

Workflow factories need particular care. Without `workflowtest.WithSynctest()`, the suite calls its factory once with the parent test and runs parallel cases against that shared driver. With the option, each case gets a fresh driver inside its own bubble and cases run sequentially. Register cleanup on the test supplied by the suite in either mode. [Workflow driver conformance](https://arkoren.dev/docs/guides/workflows/testing.md#driver-conformance) owns the required registry and engine setup; a recording fake does not execute that engine.

## Connect Both Roles

Some contracts need a matching producer and consumer. The queue suite takes a `queuetest.Driver` containing both roles; they must use the same queues. Import `context`, `time`, `arkoren.dev/core/service/queue/queuetest`, and `arkoren.dev/core/service/queue/memoryqueue`:

```go
func TestQueueContract(t *testing.T) {
	queuetest.Run(t, func(t *testing.T) queuetest.Driver {
		driver := memoryqueue.New()
		t.Cleanup(func() {
			ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
			defer cancel()
			if err := driver.Stop(ctx); err != nil {
				t.Error(err)
			}
		})
		return queuetest.Driver{Dispatcher: driver, Consumer: driver}
	}, queuetest.WithSynctest())
}
```

The factory runs once per queue subtest and must supply empty queues. The in-process clock option lets timer cases advance without real waits. For a backend that requires queue provisioning or name mapping, return dedicated empty names in the factory's `Queue` and `Other` fields and clean up only those owned resources.

Do not use the in-process clock option for a driver that talks to an external system. The [queue suite reference](https://pkg.go.dev/arkoren.dev/core/service/queue/queuetest) describes its role and fixture requirements.

## Declare Capabilities

Suite options describe actual supported capabilities or the test environment; they do not redefine an operation's meaning.

For expiry governed by a remote server, the cache suite uses real time when given `cachetest.WithExternalClock()`. With your configured factory named `newDriver`, the invocation is:

```go
cachetest.Run(t, newDriver, cachetest.WithExternalClock())
```

Without that option, timed cache cases construct the driver in a synctest bubble. Queue and workflow suites instead opt into bubbles explicitly with their own `WithSynctest()` options. Follow the particular suite, not a guessed common default; [Controlling Time](https://arkoren.dev/docs/guides/testing/time.md) explains why a process clock cannot control a server clock.

Where a contract permits an unsupported operation, a suite may expose a corresponding option. Use it only when the implementation really has that documented limitation and returns the required unsupported result. Some options assert rejection without submission rather than merely skipping a check. A failing required operation cannot be made conformant by treating it as an optional feature.

## Interpret Coverage

Read subtest names and skips along with the final result. A failure identifies the rule to investigate; fix the driver or its fixtures against the public contract rather than weakening the assertion. A skipped content check is not evidence that the content was correct.

Telemetry illustrates this boundary. `telemetrytest.Run` checks lifecycle and the signal capabilities the driver implements. Content assertions need a `telemetrytest.Reader`, supplied by the driver or through `telemetrytest.WithReader`; without one those checks are skipped. An exporting driver's reader should inspect its test backend. Run the suite on individual drivers rather than a telemetry composite. See the [`telemetrytest` reference](https://pkg.go.dev/arkoren.dev/core/service/telemetry/telemetrytest).

Telemetry event checks require all requested attributes on one event with the requested name. Extra attributes are allowed; attributes split across events cannot combine into a match. With no requested attributes, the event name alone matches.

Other suites need domain fixtures: the push factory returns both a ready driver and a driver-valid recipient, while feature-flag testing installs provider fixtures through a harness. Follow [Push Drivers](https://arkoren.dev/docs/guides/push/drivers.md) and [Feature Flag Testing](https://arkoren.dev/docs/guides/feature-flags/testing.md) for those requirements. Push conformance checks portable direct-driver submission, not device delivery or facade sender defaults.

Run infrastructure-backed suites as [integration tests](https://arkoren.dev/docs/guides/testing/integration-tests.md), with an explicit short-mode guard, isolated resources, and teardown on failure. Deterministic HTTP or SDK fixtures can stay in the short run when they need neither credentials nor a live provider; they still do not establish live-provider behavior.

For authentication's request factories, use [Authentication Tests](https://arkoren.dev/docs/guides/testing/auth-tests.md#check-authenticators). For the cache example's full checks and options, see the [`cachetest` API reference](https://pkg.go.dev/arkoren.dev/core/service/cache/cachetest).

---

Source: https://arkoren.dev/docs/guides/openapi/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/openapi/overview)

# OpenAPI Overview

Clients need a description of your HTTP API without having to discover its behavior
through requests. Add an **operation**, the description of one HTTP method and
path, beside a [route registration](https://arkoren.dev/docs/guides/http-fundamentals/routing.md). The router
exports those descriptions as an OpenAPI document without running your handlers.

An annotation describes behavior; it does not implement it. Your handler still
decodes input, applies validation and authorization, and writes its response.
Exporting the document and [serving documentation](https://arkoren.dev/docs/guides/openapi/serving.md) are separate tasks.

## Describe Output

For a greeting endpoint, use a named response type so the description can follow
the JSON shape your handler writes. These examples use `net/http`,
`arkoren.dev/core/http/response`, `arkoren.dev/core/http/router`, and
`arkoren.dev/foundation/validate`:

```go
type Greeting struct {
	Message string `json:"message"`
}
```

The handler returns that type through the ordinary response helper:

```go
func greet(writer http.ResponseWriter, incoming *http.Request) error {
	return response.JSON(writer, http.StatusOK, Greeting{Message: "Hello"})
}
```

Register it with an operation callback on your existing `routes` router:

```go
routes.Get("/greeting", greet, func(operation *router.OperationBuilder) {
	operation.Summary("Read a greeting")
	operation.JSONResponse(http.StatusOK, validate.Infer[Greeting]())
})
```

The operation describes a `200` response with `application/json` content.
`validate.Infer[Greeting]()` supplies the output type's JSON Schema; it neither
checks response values nor adds runtime validation. Use
[request schemas](https://arkoren.dev/docs/guides/openapi/operations.md#describe-input) when the endpoint also accepts
validated input.

## Export JSON

After registering your API routes, generate the document:

```go
document, err := routes.OpenAPI("Greeting API", "1.0.0")
```

On success, `document` contains deterministic OpenAPI **3.2.1** JSON, including
`GET /greeting` and its declared response. The supplied `1.0.0` is your API's
version, not the OpenAPI specification version. No listener, application startup,
request, or browser UI is required.

Check `err` before saving or distributing the bytes. Generation checks the
supported document structure and declarations; it does not compare them with
handler behavior. The [serving guide](https://arkoren.dev/docs/guides/openapi/serving.md#export-json) explains export
failures and live document caching.

The focused fragments in this group retain error results but omit routine error
propagation. Complete implementations must handle those errors before using
their results.

## Contract Boundaries

Only annotated, nonhidden registrations appear in OpenAPI. Routes without an
operation callback still work but are omitted; an annotation without responses
does not acquire a guessed success response. Grouped routes use their
[final prefixed paths](https://arkoren.dev/docs/guides/http-fundamentals/routing.md#group-routes).

Operation callbacks run synchronously once during registration, in order on the
same draft. Schema bytes and examples are captured when supplied, and published
metadata is detached from retained drafts. Nil callbacks panic. Finish route
registration before exporting concurrently or serving requests.

Annotations do not parse bodies, set headers or statuses, discover middleware
errors, or enforce security declarations. Keep them aligned with
[request validation](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#validate-input),
[response writing](https://arkoren.dev/docs/guides/http-fundamentals/responses.md), and
[error mapping](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md).
[Describing Operations](https://arkoren.dev/docs/guides/openapi/operations.md) covers their declaration limits.

The exporter does not inspect opaque mounted protocols or infer every integration's
contract. Use the [built-in handler guide](https://arkoren.dev/docs/guides/openapi/built-in-handlers.md) to choose shipped
annotations or describe an integration manually. For broker messages rather than
HTTP operations, use [AsyncAPI](https://arkoren.dev/docs/guides/asyncapi/overview.md); it produces a separate
document, not input for Swagger UI.

For declarations and export methods, see the
[`router` API reference](https://pkg.go.dev/arkoren.dev/core/http/router).

---

Source: https://arkoren.dev/docs/guides/openapi/operations.md
[HTML documentation](https://arkoren.dev/docs/guides/openapi/operations)

# Describing Operations

A client needs to know what to send and how to interpret both success and failure.
Operation callbacks record that contract beside your handler registration. Reuse
the handler's input schema and output type where possible, but keep decoding,
validation, and response writing explicit: declarations do not bind or execute
those steps for you.

These examples assume an existing `routes` router and the `Greeting` output type
from the [overview](https://arkoren.dev/docs/guides/openapi/overview.md). They use `net/http`,
`arkoren.dev/core/http/request`, `arkoren.dev/core/http/response`,
`arkoren.dev/core/http/router`, and `arkoren.dev/foundation/validate`.

## Describe Input

To let a caller choose the greeting's name, define a narrow request type and its
[validation schema](https://arkoren.dev/docs/guides/validation/structs-and-reuse.md):

```go
type GreetingInput struct {
	Name string `json:"name"`
}

var greetingInput = validate.Struct(func(object *validate.StructBuilder[GreetingInput]) {
	object.Field("Name", func(name *validate.StringBuilder[string]) {
		name.Required()
	})
})
```

The handler must actually decode and validate that input before producing its
typed response:

```go
func createGreeting(writer http.ResponseWriter, incoming *http.Request) error {
	input, err := request.ParseJSON(incoming, greetingInput)
	if err != nil {
		return err
	}

	return response.JSON(writer, http.StatusOK, Greeting{Message: "Hello, " + input.Name})
}
```

`ParseJSON` returns the parsed, validated input, with schema defaults applied.
An invalid request returns an error instead of reaching the success response.
Now reuse the schema in the operation description:

```go
routes.Post("/greeting", createGreeting, func(operation *router.OperationBuilder) {
	operation.ID("createGreeting")
	operation.JSONBody(greetingInput).BodyRequired(true)
	operation.JSONResponse(http.StatusOK, validate.Infer[Greeting]())
})
```

`JSONBody` describes `application/json`; use `Body(media, schema)` for another
representation. Body presence is optional unless you declare
`BodyRequired(true)`. That flag is separate from a schema's required properties,
and neither declaration rejects requests by itself.

## Describe Failures

The success declaration does not discover the errors returned by `ParseJSON`.
Add its known input problems inside the same registration callback:

```go
operation.Problem(request.ErrInvalidJSON)
operation.Problem(request.ErrValidationFailed)
operation.Problem(request.ErrBodyTooLarge)
```

These describe malformed JSON at `400`, decoding or validation violations at
`422`, and an oversized body at `413`. Operational failures can also escape
parsing; this list is not a claim that every failure has one of those identities.
See [request validation](https://arkoren.dev/docs/guides/http-fundamentals/requests.md#validate-input) for the
runtime boundary.

`Problem` accepts `problem.Details` and describes the router's negotiated RFC 9457
wire response at its status: `application/problem+json`, `application/json`, and
`text/plain`. A zero status becomes `500`. The schema does not expose the Go
value's private fields or error cause. Declare application and middleware problems
explicitly, matching your actual [error mapping](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md).

## Add Parameters

For a route such as `/greetings/{id}`, describe an identifier and optional inputs
independently of the body. Inside its operation callback:

```go
operation.Parameter("id", "path", router.RawSchema(`{"type":"integer","minimum":1}`))
operation.Parameter("language", "query", router.RawSchema(`{"type":"string"}`))
operation.Parameter("X-Request-ID", "header", router.RawSchema(`{"type":"string"}`))
```

The supported locations are `path`, `query`, `header`, and `cookie`. Path parameters
are always required; all other parameters are optional in the document. There is
no setting to require a non-path parameter or choose its serialization style.
Schema constraints do not change these limits or parse the incoming value.

Export checks path parameter names against the final path, including group
prefixes. Undescribed single-segment wildcards receive required string parameters;
an explicit declaration supplies the schema instead. Duplicate names within one
location fail export, with case-insensitive comparison for header parameters.

## Describe Responses

Choose declarations that match what the handler writes. `JSONResponse` describes
a JSON body; `Response` describes a status without adding a body. For a redirect,
declare its status and destination header:

```go
operation.Response(http.StatusSeeOther, "Continue at the returned location")
operation.ResponseHeader(http.StatusSeeOther, "Location", router.RawSchema(`{"type":"string","format":"uri-reference"}`))
```

Statuses must be integers from `100` through `599`; there are no status-range or
`default` response declarations. An empty description uses standard HTTP status
text. `Response` preserves existing bodies and headers at that status; it does not
remove them to make a previously declared response bodyless.

For a binary download, describe the media type rather than inventing a JSON body:

```go
operation.MediaResponse(http.StatusOK, "application/octet-stream", router.RawSchema(`{"type":"string","format":"binary"}`))
operation.ResponseHeader(http.StatusOK, "Content-Disposition", router.RawSchema(`{"type":"string"}`))
```

Different media types at one status describe alternative representations;
repeating a media type replaces its representation. Request and response media
types must parse as a type/subtype. These declarations do not select an encoder,
set headers, negotiate content, or validate output. Check the actual behavior of
[downloads and other integrations](https://arkoren.dev/docs/guides/openapi/built-in-handlers.md#manual-contracts), including
[streaming responses](https://arkoren.dev/docs/guides/http-fundamentals/streaming-responses.md).

Response header names match case-insensitively within one status. A later
`x-request-id` replaces an earlier `X-Request-ID`, including its exported spelling
and schema; other statuses are unaffected. Invalid header names fail export.

## Request Examples

To show clients a valid greeting request, declare the body representation before
adding its named example:

```go
operation.JSONBody(greetingInput).BodyRequired(true)
operation.Example("application/json", "personalGreeting", GreetingInput{Name: "Ada"})
```

The example is for the request body, not a response. Its JSON encoding is captured
immediately, but it is not validated against the schema. Encoding failures become
export errors. Adding an example before its media type records an error even if
you declare that body representation later.

## Declare Security

If your routes enforce bearer authentication, name that scheme when exporting:

```go
document, err := routes.OpenAPI("Greeting API", "1.0.0", func(definition *router.OpenAPIBuilder) {
	definition.SecurityScheme("bearer", router.RawSchema(`{"type":"http","scheme":"bearer"}`))
})
```

Then reference it in each protected route's operation callback:

```go
operation.Security("bearer")
```

This declares a requirement; it does not install authentication or authorization.
Configure the [actual authentication policy](https://arkoren.dev/docs/guides/authentication/overview.md)
separately and declare its relevant error responses. Middleware does not supply
schemes or requirements automatically. Pass the same document configuration to
`OpenAPIHandler` if you also serve the JSON.

The supported schemes are HTTP, API-key, and mutual TLS. HTTP requires a valid
`scheme` token. API-key requires a nonempty `name` and an `in` location of `query`,
`header`, or `cookie`; a header name must be an HTTP token. For example:

```json
{
  "type": "apiKey",
  "name": "X-API-Key",
  "in": "header"
}
```

OAuth flows and OpenID Connect scheme definitions are unsupported. Scheme names
use letters, digits, `.`, `_`, or `-`; unsupported fields are rejected except
`x-` extensions. Standard supported fields must be strings.

Every referenced scheme must be declared. Repeated `Security` calls describe
alternatives (OR), not a combined requirement (AND). Nonempty scopes fail export.
The document API has no server URL setting.

## Schema Limits

Bodies, responses, parameters, and headers accept `router.Schema`, an interface
with `JSONSchema() ([]byte, error)`. Published validation schemas implement it,
but export includes only their representable rules. Cross-field comparisons,
conditions, custom checks, normalizers, and clock-based rules remain runtime
behavior; see [JSON Schema](https://arkoren.dev/docs/guides/validation/json-schema.md).

`validate.Infer[T]()` describes a Go type's JSON output without applying value
rules. Interfaces and opaque custom `MarshalJSON` implementations produce open
schemas unless a supported explicit type provider describes their shape.
Recursive types become open at the repeated level, and pointer nil values are
not modeled. Types without a JSON representation produce inference errors.

If inference cannot describe your wire shape, supply `router.RawSchema`. Its root
must be a JSON object or boolean, not an array, string, or null. The router checks
syntax and root shape, not arbitrary JSON Schema keywords. Schema bytes are
captured at declaration time and remain inline; references are neither rewritten
nor fetched. Successful export is not proof of equivalence with your runtime
validator or encoder.

## Check Declarations

Use `Summary`, `Description`, and `Tags` to identify and organize operations.
Descriptions pass CommonMark through without interpretation or sanitization.
`Deprecated(true)` labels an operation but does not disable it or add a warning
header.

Nonempty operation IDs must be unique across the whole document. If one
registration expands to several methods, omit its ID or register each method
separately with a unique ID. Equivalent templates such as `/users/{id}` and
`/users/{name}` also conflict, even under different methods. `QUERY` has its OpenAPI
3.2 method field; `CONNECT` and extension methods appear in `additionalOperations`.

Leave mounts and catch-all registrations unannotated, or set `Hidden(true)`.
Visible annotated opaque paths fail export. Hiding an operation suppresses its
metadata errors too, but does not prevent requests or remove it from `Routes()`.

Invalid metadata does not stop route execution. The first declaration error is
retained even if a later setter replaces the invalid value. Run
[offline export](https://arkoren.dev/docs/guides/openapi/serving.md#export-json) before deployment to catch those errors,
invalid media types, duplicate IDs, and path conflicts rather than discovering
them on the first documentation request.

For all declaration methods, see the
[`router` API reference](https://pkg.go.dev/arkoren.dev/core/http/router).

---

Source: https://arkoren.dev/docs/guides/openapi/serving.md
[HTML documentation](https://arkoren.dev/docs/guides/openapi/serving)

# Serving Documentation

You can distribute your HTTP contract as a build artifact without exposing a
documentation endpoint. If clients need live access, register a JSON handler;
if people need a browser viewer, add Swagger UI separately. None of these choices
starts a listener or installs authentication.

The examples assume an annotated `routes` router, as introduced in the
[overview](https://arkoren.dev/docs/guides/openapi/overview.md). Finish registration before exporting concurrently or
serving requests.

## Export JSON

For a contract test, client generator, or release artifact, export after registering
the API routes:

```go
document, err := routes.OpenAPI("Greeting API", "1.0.0")
```

On success, `document` holds OpenAPI **3.2.1** JSON bytes. Save them through your
build's artifact-writing step. Identical declarations and document settings
produce identical bytes; no application startup, listener, or request is needed.

Title and API version must be nonempty. Invalid declarations, duplicate operation
IDs, and equivalent path templates return contextual errors instead of a partial
document. Make export success a test or build requirement before publishing.
This checks [declarations](https://arkoren.dev/docs/guides/openapi/operations.md#check-declarations), not whether handlers
implement them, and it does not fill a live JSON handler's cache.

If you declared [security requirements](https://arkoren.dev/docs/guides/openapi/operations.md#declare-security), include
their schemes in the export callback. `OpenAPIBuilder.Description` can also set
the document's Info description; this does not replace operation descriptions.

## Serve JSON

The JSON handler exposes endpoint names and schemas. If that information is
private, register it on a protected router or restricted listener, not an
unprotected public route. The handler adds no access controls.

To publish the contract on an appropriately protected `routes` router:

```go
routes.Get("/openapi.json", routes.OpenAPIHandler("Greeting API", "1.0.0"))
```

Serve that router through your [HTTP server](https://arkoren.dev/docs/guides/http-networking/server.md).
A successful GET returns the document with `Content-Type: application/json`.
The registration has no operation callback, so it is omitted from its own export.

Document configuration callbacks run once when you construct `OpenAPIHandler`.
Generation happens on its first request, capturing all routes registered before
that request. The handler caches either the generated bytes or the generation
error. Later registrations do not invalidate it; construct another handler when
you need a different document. Construction does not compile or freeze the router.

This is an in-process generation cache, not an HTTP cache policy. The JSON handler
sets no `Cache-Control` header; configure that policy in your middleware if needed.
A generation failure returns through the router's ordinary
[error handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md) on every request to that
handler, rather than retrying generation.

## Add Swagger UI

Swagger UI lets people browse the document and make API requests from their
browser. Protect its complete subtree as well as the JSON endpoint when the
contract is private; protecting only the HTML page leaves the JSON exposed.

Import `arkoren.dev/core/http/swagger` and construct a viewer for your JSON URL:

```go
viewer, err := swagger.Handler("/docs", "/openapi.json")
```

Invalid paths or URLs return an error. After checking `err`, register both the
base route and its subtree on the appropriately protected router:

```go
routes.Get("/docs", viewer)
routes.Get("/docs/{path...}", viewer)
```

The viewer serves `/docs` and `/docs/`, with assets beneath `/docs/`. Keep these
paths equal to the base path passed to the factory. Leave both registrations
unannotated: the exporter does not discover operations inside the UI, and an
annotated catch-all would fail export.

Once the server runs, opening `/docs/` makes the browser fetch `/openapi.json`.
The factory neither generates nor fetches that document. Pinned JavaScript and
CSS are bundled locally; the initializer disables the remote validator and
query-string configuration. The browser still needs access to the document and
to any API it calls, so bundled assets do not make a remote API available offline.

If the default tab title, `API documentation`, is too generic, use this alternative
construction:

```go
viewer, err := swagger.Handler("/docs", "/openapi.json", func(page *swagger.Builder) {
	page.Title("Greeting API")
})
```

The title is escaped as plain text; an empty title remains empty. Construction
callbacks run once, synchronously in order, even for invalid configuration. Nil
callbacks panic. Retaining the draft cannot change the completed handler, which
supports concurrent requests.

## Choose URLs

Prefer a root-relative document URL such as `/openapi.json` when the viewer and
document share an origin. HTTP(S) and relative URLs are also accepted. Relative
URLs resolve against the configured base path plus `/`, consistently whether the
page was opened at `/docs` or `/docs/`; request Host and forwarding headers do not
change that resolution.

The base path must be a clean absolute URL path with unreserved ASCII segments.
It cannot have a trailing slash unless it is `/`. Wildcards, escaped separators,
and queries are invalid. The document URL must be nonempty and cannot contain
credentials, fragments, scheme-relative URLs, whitespace, or backslashes.

The viewer serves only its page and named assets; unknown paths return `404`.
It accepts GET and HEAD, with no response body for HEAD. Other methods return
`405` with `Allow: GET, HEAD` if they reach the handler; router method matching can
reject them first. HTML and the initializer use `Cache-Control: no-store`;
pinned JavaScript and CSS use `public, max-age=3600`.

## Protect Documentation

A private contract needs protection at every entry point: the JSON route, viewer
page, and viewer subtree. Use your existing authentication middleware or a
[restricted listener](https://arkoren.dev/docs/guides/application/health-and-readiness.md). Hiding an operation
from OpenAPI does not protect the live endpoint, and publishing its security
requirement does not enforce it.

The browser must authenticate its document fetch as well as API requests made
from the viewer. Swagger UI does not bypass API authentication, authorization,
CORS, or other middleware. If you choose a cross-origin document URL, the browser
policy and authentication mechanism must permit that fetch.

If you configure [Content Security Policy](https://arkoren.dev/docs/guides/browser-security/secure-headers.md),
allow same-origin scripts, inline styles used by Swagger UI, and connections to
the document and API origins. Allow data images if you use the UI's embedded
images. No inline script permission is required.

[Diagnostic route tools](https://arkoren.dev/docs/guides/diagnostics/capture-setup.md) are a separate operator
interface. They inspect registered routes; they do not mutate OpenAPI schemas or
turn REST endpoints into MCP tools. Publishing a contract does not replace an
[MCP server](https://arkoren.dev/docs/guides/mcp/first-server.md).

For document export and settings, see the
[`router` API reference](https://pkg.go.dev/arkoren.dev/core/http/router).
For viewer configuration, see the
[`swagger` API reference](https://pkg.go.dev/arkoren.dev/core/http/swagger).

---

Source: https://arkoren.dev/docs/guides/openapi/built-in-handlers.md
[HTML documentation](https://arkoren.dev/docs/guides/openapi/built-in-handlers)

# Built-in Handlers

Some shipped handlers have matching operation callbacks, so you can describe
their responses without repeating their schemas. Pass the callback when
registering the handler; the exporter does not infer a contract from its type or
package. The callback describes behavior, but neither registers the route nor
changes its access policy.

Use this page to find application annotations and recognize common manual
integration boundaries. It is not a guarantee that every built-in handler or
domain integration has a complete reusable contract.

## Application Annotations

With your existing `app` and `routes`, import `arkoren.dev/core/application`.
Choose the callback that matches the handler you are registering:

| Handler | Callback | Declared Output |
| --- | --- | --- |
| `LiveHandler()` | `LiveOperation` | Plain-text `200` or `503`. |
| `StartupHandler()` | `StartupOperation` | Plain-text `200` or `503`. |
| `ReadyHandler()` | `ReadyOperation` | Plain text or verbose JSON at `200` or `503`. |
| `InfoHandler()` | `InfoOperation` | Application and build information as `200` JSON. |
| `WelcomeHandler()` | `WelcomeOperation` | A welcome page as `200` HTML. |

For probes, pass the callbacks directly to their registrations:

```go
routes.Get(application.LivePath, app.LiveHandler(), application.LiveOperation)
routes.Get(application.StartupPath, app.StartupHandler(), application.StartupOperation)
routes.Get(application.ReadyPath, app.ReadyHandler(), application.ReadyOperation)
```

The path constants are `/livez`, `/startupz`, and `/readyz`. Custom paths and group
prefixes work too: descriptions follow the registered path, not a hard-coded
endpoint. Omit a callback to leave that handler out of OpenAPI without disabling
the route.

All three callbacks describe plain-text `ok\n` at `200` and `unavailable\n` at
`503`, with `Cache-Control: no-store` and `X-Content-Type-Options: nosniff`.
Liveness succeeds while the application is live. Startup records completed
startup, including while stopping and after stop; it is not a readiness check.
Configure the lifecycle and dependency checks through
[health and readiness](https://arkoren.dev/docs/guides/application/health-and-readiness.md).

For the public welcome page, choose your own path:

```go
routes.Get("/", app.WelcomeHandler(), application.WelcomeOperation)
```

`WelcomeOperation` describes HTML and the shared no-store/nosniff headers, without
capturing the configured application name. The page displays that name and
Arkoren resource links, not runtime, configuration, or build diagnostics. Choose
an application name suitable for public display.

## Readiness Details

Readiness reports whether the application is live and its cached health pass is
fresh and passing. Without checks it equals liveness. With checks, it waits for
the first passing pass; a result older than three readiness intervals is stale
and fails readiness. Requests do not run checks themselves.

`ReadyOperation` describes an optional string query parameter named `verbose`.
Its presence selects JSON: `?verbose`, `?verbose=`, and `?verbose=false` all have
the same effect. It is not a boolean opt-out. The callback includes JSON schemas
at both `200` and `503`, alongside the plain-text representations.

The static schema has a required `status` and optional `reason`, `checked_at`,
and `checks`. Each check can report its status and an error string. Export does
not insert current dependency names, check results, or failure messages into the
document.

> [!WARNING]
> The live verbose response reveals dependency names. Failed checks normally
> report `reason withheld`, but `HealthDetails(true)` can also reveal connection
> information in error messages. Restrict the endpoint to operators and the
> platform; omitting its annotation does not restrict access.

## Application Information

If operators need application identity and build details, register the information
handler on an already protected `admin` router:

```go
admin.Get(application.InfoPath, app.InfoHandler(), application.InfoOperation)
```

`InfoPath` is `/infoz`. The callback describes its `200` JSON response and
no-store/nosniff headers. Its schema includes identity, environment, state, log
level, build metadata, and optional startup and uptime fields. Runtime fields
that can be omitted remain optional; current application and binary values are
not embedded in the contract.

The live response can reveal versions and revisions, so expose it only to
operators. Neither the handler nor callback supplies authentication. Use your
existing protected router or a
[restricted listener](https://arkoren.dev/docs/guides/application/health-and-readiness.md), and separately
protect documentation that includes private endpoint names and schemas.

## Extend Annotations

If your protected information route requires bearer authentication, keep the
shipped callback and add your policy description afterward. This fragment also
uses `arkoren.dev/core/http/router`:

```go
admin.Get(application.InfoPath, app.InfoHandler(), application.InfoOperation, func(operation *router.OperationBuilder) {
	operation.Tags("operations")
	operation.Security("operatorBearer")
})
```

Declare `operatorBearer` in the document's
[security configuration](https://arkoren.dev/docs/guides/openapi/operations.md#declare-security). This fragment assumes
the `admin` router already enforces authentication; `Security` only describes it.
Add middleware failure responses explicitly when they apply.

Callbacks run in order on the same draft, so later scalar settings replace
earlier ones. Shipped application callbacks do not set operation IDs; add unique
IDs if your clients need them. They do not infer middleware errors or security
requirements.

## Manual Contracts

Other handlers depend on provider configuration or implement a protocol that a
single operation cannot describe. Annotate only the behavior you actually serve:

- [OAuth login](https://arkoren.dev/docs/guides/authentication/oauth.md) uses redirects and provider callbacks.
  Describe actual statuses and the `Location` header with `Response` and
  `ResponseHeader`, rather than assuming JSON. The document API does not support
  OAuth-flow or OpenID Connect security schemes.
- [Webhook receivers](https://arkoren.dev/docs/guides/webhooks/receiving-webhooks.md) need provider-specific
  bodies and signature headers. Declare them explicitly; annotations neither
  verify signatures nor discover provider-specific failure responses.
- [Downloads](https://arkoren.dev/docs/guides/file-storage/uploads-and-downloads.md) and
  [Prometheus metrics](https://arkoren.dev/docs/guides/observability/telemetry-drivers.md#prometheus) need their
  actual media types and headers. Use `MediaResponse`, not `JSONResponse`, for
  binary or text output. The [response declarations](https://arkoren.dev/docs/guides/openapi/operations.md#describe-responses)
  show the pattern.
- Mounted [MCP endpoints](https://arkoren.dev/docs/guides/mcp/first-server.md) remain a separate protocol.
  Export does not traverse their handlers or infer internal operations. Leave
  mounts and catch-all routes unannotated or hidden; visible annotated opaque
  paths fail export.

For every integration, match the body, headers, statuses, and security policy to
the configured handler and middleware. A static annotation or diagnostic route
listing does not discover or maintain that domain contract for you.

For shipped callbacks, see the
[`application` API reference](https://pkg.go.dev/arkoren.dev/core/application).
For manual declarations, see the
[`router` API reference](https://pkg.go.dev/arkoren.dev/core/http/router).

---

Source: https://arkoren.dev/docs/guides/asyncapi/overview.md
[HTML documentation](https://arkoren.dev/docs/guides/asyncapi/overview)

# AsyncAPI Overview

Another service needs to know which messages your application sends, where they
travel, and what a delivery contains. `arkoren.dev/core/asyncapi` combines your
typed messaging definitions with explicit documentation metadata to export
AsyncAPI **3.1.0** JSON. Generating that document requires no broker connection.

Use it for [broadcast topics](https://arkoren.dev/docs/guides/broadcasting/overview.md#defining-topics) and
[queued jobs](https://arkoren.dev/docs/guides/queues/producing-jobs.md#defining-jobs). It neither publishes
messages nor registers consumers. The synchronous [event bus](https://arkoren.dev/docs/guides/events/overview.md)
is a different mechanism; [OpenAPI](https://arkoren.dev/docs/guides/openapi/overview.md) describes HTTP contracts.

## Contract Model

A **message** describes one complete wire payload. A **channel** names the
destination where messages travel, such as a topic or queue. Several message
types can share a channel. An **operation** selects the messages your application
sends or receives on that channel: `Send` means out of this application, and
`Receive` means into it, never the broker's perspective.

The typed helpers connect these declarations to application-owned definitions.
`JSONTopic` takes a `broadcast.Topic[T]`; `JSONJob` takes a `queue.Job[T]`. Each
helper uses the definition's logical name to describe an Arkoren JSON envelope
and stores the decoded payload schema separately. You still supply the physical
destination and the application's operations. Nothing discovers producers,
samples traffic, or reads driver configuration.

These are descriptions, not runtime validation or delivery policies. A document
does not establish durable delivery, retries, ordering, or authorization. Keep
those responsibilities in your messaging drivers, handlers, and infrastructure.

## Shared Definitions

Suppose your reporting service announces completed reports. Keep the topic in
your application's shared contract package so the runtime publisher and the
document can use the same identity. With `broadcast` imported from
`arkoren.dev/core/service/broadcast`, define its small logical payload:

```go
type ReportReady struct {
	ReportID string `json:"report_id"`
}

var Ready = broadcast.NewTopic[ReportReady]("report.ready")
```

`Ready` identifies the fact; it does not identify a broker endpoint. Use
`JSONTopic` only if your transport JSON-encodes `broadcast.Message`. A different
wire representation needs an explicit [message schema](https://arkoren.dev/docs/guides/asyncapi/messages-and-schemas.md#other-wire-formats).

## Document Generation

Import `arkoren.dev/core/asyncapi` and `arkoren.dev/foundation/validate`. In the
same application package, describe the physical destination and sending role:

```go
var Document = asyncapi.New("Reporting events", "1.0.0", func(definition *asyncapi.Builder) {
	definition.Channel("reports", "production.reports", func(channel *asyncapi.ChannelBuilder) {
		channel.JSONTopic("ready", Ready, validate.Infer[ReportReady]())
	})
	definition.Send("announce-report", "reports", func(action *asyncapi.OperationBuilder) {
		action.Messages("ready")
	})
})
```

`validate.Infer` describes the Go payload's JSON shape. The wire message is a JSON
envelope whose `payload` field contains base64-encoded JSON, not an inline
`ReportReady` object. [Messages and Schemas](https://arkoren.dev/docs/guides/asyncapi/messages-and-schemas.md#wire-and-logical)
explains both layers and the supported schema profile.

This portable document omits broker endpoints. If you add
[server declarations](https://arkoren.dev/docs/guides/asyncapi/bindings-and-security.md#broker-endpoints), every channel
must explicitly select at least one server. Every operation must select at least
one message from its channel, whether or not servers are declared.

Generate the bytes in a test, build command, or other application code:

```go
data, err := Document.JSON()
```

On success, `data` contains deterministic JSON with `asyncapi: "3.1.0"`, a
`reports` channel, and an `announce-report` send operation. `1.0.0` is your
contract's version, not the AsyncAPI specification version. Handle `err` before
using the bytes; [Serving Documentation](https://arkoren.dev/docs/guides/asyncapi/serving.md) covers file exports and HTTP
access. Focused snippets in these guides keep errors visible but omit routine
propagation; complete application code must handle them.

## Stable Names

Choose document IDs independently of runtime names. This example separates four
values with different purposes:

| Value | Example | Purpose |
| --- | --- | --- |
| Channel ID | `reports` | Local reference used by operations. |
| Channel address | `production.reports` | Literal destination used by the deployed transport. |
| Topic name | `report.ready` | Logical discriminator carried in the envelope. |
| Message ID | `ready` | Local selection within the channel. |

Document IDs accept nonempty ASCII letters, digits, dots, underscores, and
hyphens. Duplicate IDs are errors, not replacements. Keep channel/message ID
pairs unambiguous: typed logical schema components use `<channel>.<message>`,
so pairs such as `reports.v1` / `ready` and `reports` / `v1.ready` collide.

The exporter does not apply a driver's destination prefixes or routing rules.
Supply the deployed physical address yourself. An empty address exports as
`null`, meaning unknown; it does not choose a default destination. Address
templates containing braces are unsupported.

## Early Checks

`New` runs supplied construction callbacks synchronously, once in order, and
captures schemas and examples immediately. It retains declaration errors rather
than returning them. `JSON()` checks the completed graph, including forward
references, and caches either the export or its error. Run an export before
deployment so an invalid declaration does not first fail on an HTTP request.

The resulting document is immutable and has no startup or cleanup work. Construct
a new document when the contract changes; changing a retained builder or an
exported byte slice does not update it. The exact
[snapshot and failure behavior](https://arkoren.dev/docs/guides/asyncapi/serving.md#snapshot-behavior) also applies when
you serve the document.

For all declarations and options, see the
[`asyncapi` API reference](https://pkg.go.dev/arkoren.dev/core/asyncapi).

---

Source: https://arkoren.dev/docs/guides/asyncapi/messages-and-schemas.md
[HTML documentation](https://arkoren.dev/docs/guides/asyncapi/messages-and-schemas)

# Messages and Schemas

A subscriber receives transport bytes, not your publisher's Go struct. Describe
that complete **wire payload** in the message schema. For Arkoren JSON envelopes,
the typed helpers also expose a **logical payload** schema: the application value
inside the envelope after decoding.

The examples reuse `Ready`, `ReportReady`, and the `reports` channel from the
[overview](https://arkoren.dev/docs/guides/asyncapi/overview.md#shared-definitions). They use `asyncapi`, `broadcast`, and
`validate` from `arkoren.dev/core/asyncapi`, `arkoren.dev/core/service/broadcast`,
and `arkoren.dev/foundation/validate`.

## Wire and Logical

`JSONTopic("ready", Ready, validate.Infer[ReportReady]())` describes the JSON
encoding of `broadcast.Message`. A minimal report announcement looks like this
on the wire:

```json
{
  "name": "report.ready",
  "payload": "eyJyZXBvcnRfaWQiOiJycHQtMSJ9"
}
```

The `payload` string is base64-encoded JSON. Decode it to obtain the logical
value `{"report_id":"rpt-1"}`. The wire schema requires `name` and `payload`,
fixes `name` to `report.ready`, and also describes optional `id` and string-valued
`metadata` fields. It does not replace the base64 string with the logical object.

The helper puts the logical schema in
`components.schemas["reports.ready"]`, using the channel ID and message ID, not
the topic name or address. On the channel message it adds:

```json
{
  "x-arkoren-logical-payload": "#/components/schemas/reports.ready"
}
```

This is a plain JSON Pointer string in an Arkoren annotation, not a `$ref`
object. External viewers might not follow it. The wire schema's Draft 7
`contentEncoding` and `contentMediaType` annotations do not decode or validate
the base64 contents; read the logical component separately.

`JSONJob` uses the same separation for a queue message. It also describes job
identity and delivery fields, as shown in
[Queues and Operations](https://arkoren.dev/docs/guides/asyncapi/queues-and-operations.md#shared-queue-messages).

## Wire Examples

Examples let consumers recognize deliveries, but they are exported data. Use
fictional values and review schemas for internal field names before distributing
the document. An inferred schema exposes the described fields even if examples
omit them; inference is not a privacy filter.

To attach a report announcement without manually encoding base64, prepare a real
wire envelope. Import `encoding/json/v2` for the logical JSON encoding:

```go
payload, err := json.Marshal(ReportReady{ReportID: "rpt-1"})
```

After successful encoding, construct the envelope with those bytes:

```go
example := broadcast.Message{
	Name:    Ready.Name(),
	Payload: payload,
}
```

Use that value in place of the overview's original `JSONTopic` declaration,
inside the `reports` channel callback:

```go
channel.JSONTopic("ready", Ready, validate.Infer[ReportReady](), func(message *asyncapi.MessageBuilder) {
	message.Summary("A report is ready")
	message.Description("Published after the report artifact is stored.")
	message.WireExample("completed-report", example)
})
```

`WireExample` JSON-encodes and copies the whole envelope at declaration time, so
its byte payload becomes the base64 string shown above. It does not check that
the example satisfies the schema. Unsupported JSON values become deferred export
errors. The description records an application promise; it does not arrange
storage or publication ordering.

## Transport Media Types

Messages default to `application/json`. If the transport advertises another
media type for that same JSON envelope, set the message's content type to match.
For example, Arkoren's MQTT transport advertises `application/octet-stream`:

```go
channel.JSONTopic("ready", Ready, validate.Infer[ReportReady](), func(message *asyncapi.MessageBuilder) {
	message.ContentType("application/octet-stream")
})
```

This is an alternative declaration, not a second registration of `ready`.
`ContentType` changes metadata only; it neither changes the bytes nor configures
transport headers. Do not use `JSONTopic` or `JSONJob` for a transport that carries
a different wire representation.

## Other Wire Formats

An external integration might send a text status directly instead of an Arkoren
envelope. In its channel callback, use `Message` with the actual wire schema:

```go
channel.Message("status", asyncapi.RawSchema(`{"type":"string"}`), func(message *asyncapi.MessageBuilder) {
	message.ContentType("text/plain")
	message.WireExample("completed", "ready")
})
```

This message describes the text itself, with no implicit envelope or separate
logical component. Select `status` in the relevant operation's `Messages` call.
If several raw messages share a channel, you must ensure their wire schemas
distinguish them; the exporter does not prove that raw alternatives are disjoint.

## Schema Providers

A provider implements `JSONSchema() ([]byte, error)`. Published `validate`
schemas and `validate.Infer[T]()` implement that interface. Each declaration
calls its non-nil provider once during document construction and immediately
copies the returned bytes. Sharing a provider across declarations calls it once
per declaration. Errors are retained for `Document.JSON()` or the HTTP handler;
nil and typed-nil providers are errors rather than empty schemas.

All providers must produce the same supported, self-contained Draft 7 profile.
Implementing the interface alone does not make a schema acceptable. See
[Validation's JSON Schema guide](https://arkoren.dev/docs/guides/validation/json-schema.md) for export and
inference limits, including omitted rules and custom JSON encoders. A schema
accepted here does not prove equivalence to runtime validation, normalization,
or domain checks. Providers must not perform network resolution.

For a raw alternative, `RawSchema` accepts object or boolean schemas, including
composition. This message permits a nonempty text result or an integer code:

```go
channel.Message("result", asyncapi.RawSchema(`{
  "oneOf": [
    {"type": "string", "minLength": 1},
    {"type": "integer"}
  ]
}`))
```

The exporter checks keyword shapes in schema positions, not satisfiability or
instance validity. Exported message payloads use an explicit Multi Format Schema
Object with `schemaFormat: "application/schema+json;version=draft-07"`.

## Draft 7 Profile

Use this accepted keyword profile when preparing a provider or raw schema:

| Purpose | Accepted Keywords |
| --- | --- |
| Types and values | `type`, `enum`, `const` |
| Objects | `properties`, `patternProperties`, `additionalProperties`, `propertyNames`, `required`, `minProperties`, `maxProperties` |
| Arrays | `items`, `additionalItems`, `contains`, `uniqueItems`, `minItems`, `maxItems` |
| Composition | `allOf`, `anyOf`, `oneOf`, `not`, `if`, `then`, `else` |
| Numbers | `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`, `multipleOf` |
| Strings | `minLength`, `maxLength`, `pattern`, `format`, `contentEncoding`, `contentMediaType` |
| Annotations | `title`, `description`, `default`, `examples`, `readOnly`, `writeOnly`, `$comment` |
| Dialect and extensions | `$schema`, any `x-` keyword |

`items` accepts a schema or a nonempty Draft 7 tuple. Composition arrays must be
nonempty, counts must be nonnegative integers, and `multipleOf` must be positive.
If supplied, `$schema` must equal `http://json-schema.org/draft-07/schema#`.

References and identifiers (`$ref`, `$id`), definitions, `dependencies`, unknown
non-extension keywords, and newer-dialect features such as `$defs`, `prefixItems`,
and `contentSchema` are rejected in schema positions. No schema fetching,
reference resolution, or conversion from Draft 2020-12 occurs. Changing a dialect
label does not convert a schema.

Contents of `x-` extensions and data inside `default`, `const`, `enum`, and
`examples` remain opaque. Property names are not interpreted as keywords.
Formats and patterns are annotations here: the exporter neither executes them
nor converts regular expressions for another validator.

For message and schema options, see the
[`asyncapi` API reference](https://pkg.go.dev/arkoren.dev/core/asyncapi).

---

Source: https://arkoren.dev/docs/guides/asyncapi/queues-and-operations.md
[HTML documentation](https://arkoren.dev/docs/guides/asyncapi/queues-and-operations)

# Queues and Operations

A reporting worker can build new reports and remove expired ones from the same
queue. Model the destination as one channel, each job type as a message, and the
worker's receiving role as an operation selecting those messages. The document
describes that role; it does not dispatch jobs or register handlers.

Reuse your application's typed [job definitions](https://arkoren.dev/docs/guides/queues/producing-jobs.md#defining-jobs)
instead of maintaining separate documentation names. The examples import
`arkoren.dev/core/asyncapi`, `arkoren.dev/core/service/queue`, and
`arkoren.dev/foundation/validate` in an application contract package.

## Shared Job Definitions

Keep the payload specific to each task. Building a report needs its
source account; removing an artifact needs the report's identity:

```go
type BuildReport struct {
	AccountID string `json:"account_id"`
}

type RemoveReport struct {
	ReportID string `json:"report_id"`
}
```

Define both jobs on the logical queue `reports`, sharing these same definitions
with runtime dispatchers and consumers:

```go
var Build = queue.NewJob[BuildReport]("build-report", func(job *queue.JobBuilder) {
	job.OnQueue("reports")
})

var Remove = queue.NewJob[RemoveReport]("remove-report", func(job *queue.JobBuilder) {
	job.OnQueue("reports")
})
```

The logical queue is part of each job definition. It need not equal the physical
destination after your driver's prefixes or routing configuration are applied.
The exporter does not derive that destination from the job.

## Shared Queue Messages

Suppose both jobs travel through `production.report-work`. Describe that physical
destination once and add both job envelopes to it:

```go
var Document = asyncapi.New("Reporting worker", "1.0.0", func(definition *asyncapi.Builder) {
	definition.Channel("work", "production.report-work", func(channel *asyncapi.ChannelBuilder) {
		channel.JSONJob("build", Build, validate.Infer[BuildReport]())
		channel.JSONJob("remove", Remove, validate.Infer[RemoveReport]())
	})
	definition.Receive("process-reports", "work", func(action *asyncapi.OperationBuilder) {
		action.Messages("build", "remove")
	})
})
```

`JSONJob` describes the JSON encoding of `queue.Message`, not every queue
transport. Its wire schema requires `id`, `name`, `queue`, `payload`, and
`attempts`; optional `metadata` maps strings to strings. The name and logical
queue are constants from the job definition. `attempts` remains an integer,
not a constant taken from retry policy, because it can change between deliveries.

The wire `payload` is a base64 string containing the logical value's JSON.
The separate logical components are `work.build` and `work.remove`, linked by
each message's `x-arkoren-logical-payload` annotation. See
[wire and logical schemas](https://arkoren.dev/docs/guides/asyncapi/messages-and-schemas.md#wire-and-logical) before
interpreting those components in a viewer.

Generate `Document.JSON()` in a test or build step to check the declaration.
No worker or broker needs to run. Job policies and
[consumer registration](https://arkoren.dev/docs/guides/queues/consuming-jobs.md) still govern execution;
these schemas neither enforce those policies nor guarantee delivery.

## Application Roles

An operation describes the application named by the document. The worker above
receives both jobs. A separate frontend that only requests report creation
should declare a sending operation selecting only `build`. In that producer's
document callback, use this instead of the worker's `Receive`:

```go
definition.Send("request-report", "work", func(action *asyncapi.OperationBuilder) {
	action.Messages("build")
	action.Summary("Request report creation")
})
```

Declaring `remove` on the channel does not automatically add it to the producer's
operation. Every operation must explicitly select at least one message from its
own channel. Selections preserve their supplied order; duplicates, including
duplicates across repeated `Messages` calls, are errors rather than deduplication.

Channel and message declarations may appear after their operation: references
are resolved against the completed document at export. A selection cannot reach
into another channel, and operation IDs must be unique across both directions.

## Completion Events

A worker can receive queue jobs and publish broadcast facts. Keep the actual
destinations separate rather than placing every message on the work channel.
Reuse `Ready` and `ReportReady` from the
[overview's shared definitions](https://arkoren.dev/docs/guides/asyncapi/overview.md#shared-definitions), with the same
definitions used by your runtime publisher.

Inside the worker document callback, retain the `work` channel and receiving
operation, then add the completion destination and sending role:

```go
definition.Channel("events", "production.reports", func(channel *asyncapi.ChannelBuilder) {
	channel.JSONTopic("ready", Ready, validate.Infer[ReportReady]())
})
definition.Send("announce-report", "events", func(action *asyncapi.OperationBuilder) {
	action.Messages("ready")
})
```

The document now describes receiving work and sending a completion fact. It
does not connect the two operations, store an artifact, or ensure publication
after successful processing. Implement that sequence and its failure handling
in your application. If you add server declarations, both channels must
[select their servers](https://arkoren.dev/docs/guides/asyncapi/bindings-and-security.md#broker-endpoints) explicitly.

## Distinct Wire Messages

Typed helpers reject repeated discriminator identities within one channel:
duplicate topic names and duplicate job name/queue pairs are export errors,
even when their message IDs differ. Two jobs with the same name but different
logical queues are allowed because their queue constants distinguish them.

A topic and job cannot share a wire name on one channel. A job's extra envelope
fields do not keep it from matching the topic schema, so different queue values
cannot resolve that overlap. Raw `Message` schemas are not checked for overlap
with each other or with typed helpers; review those alternatives yourself.

For typed helpers and operation options, see the
[`asyncapi` API reference](https://pkg.go.dev/arkoren.dev/core/asyncapi).

---

Source: https://arkoren.dev/docs/guides/asyncapi/bindings-and-security.md
[HTML documentation](https://arkoren.dev/docs/guides/asyncapi/bindings-and-security)

# Bindings and Security

Payload schemas do not tell an integrator where to connect or which credentials
the broker requires. Add **servers** for broker endpoints, **security schemes**
for authentication methods, and **bindings** for protocol-specific details.
These declarations describe your deployment; they do not configure drivers,
provision resources, or enforce access.

The snippets use `arkoren.dev/core/asyncapi` inside the construction callbacks
introduced in [Queues and Operations](https://arkoren.dev/docs/guides/asyncapi/queues-and-operations.md). Keep credentials,
private keys, tokens, and sensitive example values out of every declaration:
the resulting metadata is included in the exported document.

## Broker Endpoints

To document the reporting worker's AMQP broker, add a server in the document
callback. Its local ID is separate from the host and protocol:

```go
definition.Server("primary", "broker.example:5672", "amqp", func(listener *asyncapi.ServerBuilder) {
	listener.Description("Reporting broker")
	listener.ProtocolVersion("0.9.1")
	listener.Pathname("/reports")
})
```

The host accepts a host with an optional port, including bracketed IPv6, not a
URL. Schemes, credentials, paths, queries, and fragments are rejected. Use
`Pathname` for the endpoint path. `ProtocolVersion` describes the protocol; it
does not negotiate one.

Once a document declares any servers, every channel must explicitly select at
least one. In the existing `work` channel callback, add:

```go
channel.Servers("primary")
```

If you also declare the completion channel, add its server selection too. With
no declared servers, channels can omit this setting for a portable document.
Server IDs may be declared later, but unresolved or duplicate selections fail
export. Defining a server does not connect to it or inspect its configuration.

## Broker Authentication

To describe username/password authentication, register a scheme in the document
callback and select it on the server. Replace the earlier `primary` declaration
with this one, retaining its optional endpoint metadata if needed:

```go
definition.SecurityScheme("login", asyncapi.RawObject(`{"type":"userPassword"}`))
definition.Server("primary", "broker.example:5672", "amqp", func(listener *asyncapi.ServerBuilder) {
	listener.Security("login")
})
```

The scheme describes a method, not a username or password. Configure actual
authentication and authorization on your broker and driver separately. The
exporter checks supported scheme type names, required fields for common scheme
families, and local references. It does not fully validate protocol rules or
OAuth flow semantics.

You can declare schemes after their selections. Duplicate scheme IDs and
unresolved local references fail export. Raw scheme aliases to local schemes
are checked within the security-scheme collection; external references remain
opaque and are never fetched.

## Requirement Alternatives

If a broker accepts either username/password or a client certificate, register
both schemes and select both on the server. This is an alternative to the
previous authentication declaration:

```go
definition.SecurityScheme("login", asyncapi.RawObject(`{"type":"userPassword"}`))
definition.SecurityScheme("certificate", asyncapi.RawObject(`{"type":"X509"}`))
definition.Server("primary", "broker.example:5671", "amqps", func(listener *asyncapi.ServerBuilder) {
	listener.Security("login", "certificate")
})
```

The server's security array means `login` **OR** `certificate`, not both.
`OperationBuilder.Security` likewise lists alternatives within an operation.
Server and operation requirements apply together (**AND** across the levels);
an operation's selection does not replace server authentication.

For example, after declaring a separate scheme named `publisher`, adding
`action.Security("publisher")` inside a sending operation documents an additional
requirement. It does not authorize that operation or define broker permissions.
Duplicate selections are errors, including across repeated `Security` calls.

Broker security metadata does not protect the HTTP endpoint that serves this
document. Apply separate [document access controls](https://arkoren.dev/docs/guides/asyncapi/serving.md#protected-access)
before exposing that endpoint.

## Protocol Bindings

A binding is a raw protocol-specific object attached to a server, channel,
operation, or message. Choose the level defined by that protocol's binding
specification. To describe the reporting work destination as an AMQP queue,
add this inside its channel callback:

```go
channel.Binding("amqp", asyncapi.RawObject(`{
  "is": "queue",
  "queue": {
    "name": "production.report-work",
    "durable": true
  },
  "bindingVersion": "0.3.0"
}`))
```

This exports the supplied queue metadata. It does not create the queue, make it
durable, or guarantee durable delivery. Configure and verify the deployed broker
independently; a successful export cannot establish that these details match it.
Server, operation, and message builders also expose `Binding` for their levels.

Bindings receive only JSON-object syntax checks, protocol-key ID checks, and
duplicate-key checks at their own level. Nested references and protocol-specific
semantics are opaque. Document graph references and local security aliases are
checked separately; they do not turn binding contents into resolved references.

No broker inventory, producer discovery, traffic sampling, or runtime address
parameter expansion occurs during export. Keep these declarations aligned with
your application's actual transport and infrastructure configuration.

For endpoint, binding, and security options, see the
[`asyncapi` API reference](https://pkg.go.dev/arkoren.dev/core/asyncapi).

---

Source: https://arkoren.dev/docs/guides/asyncapi/serving.md
[HTML documentation](https://arkoren.dev/docs/guides/asyncapi/serving)

# Serving Documentation

You can distribute an AsyncAPI file without exposing an HTTP endpoint. If
consumers need a URL, explicitly mount the document's handler on your existing
router. Both paths use the same immutable document and cached export; neither
starts messaging infrastructure or discovers runtime contracts.

These examples use `Document` from the [overview](https://arkoren.dev/docs/guides/asyncapi/overview.md#document-generation).
Review broker addresses, schema fields, examples, and security requirements
before distributing it. Document metadata is not automatically redacted.

## File Exports

For a generated artifact, obtain validated bytes before opening the destination.
With `os` imported from the standard library, this application-owned helper
leaves the file untouched if document generation fails:

```go
func Export(path string) error {
	data, err := Document.JSON()

	if err != nil {
		return err
	}

	return os.WriteFile(path, data, 0600)
}
```

Call `Export("asyncapi.json")` from your build command and handle its returned
error. On success, the file contains the same deterministic JSON returned by
`Document.JSON()`. No application startup, HTTP listener, or broker is needed.

Choose a generated-artifact path: `os.WriteFile` overwrites an existing file,
and an I/O failure can leave a partial file. The mode `0600` applies when creating
the file; it does not tighten permissions on an existing file. If publication
must replace an artifact atomically, implement an appropriate temporary-file
and replacement procedure in your build tooling.

## Protected Access

> [!WARNING]
> The document can expose internal broker addresses, message shapes, and example
> data. Before mounting its handler, restrict the listener or require your
> application's documentation authentication and authorization. Declared broker
> security does not protect an HTTP documentation route.

With an existing `routes` router and your application's `requireDocsAccess`
middleware, register the endpoint inside a protected group. Import `router` from
`arkoren.dev/core/http/router`:

```go
routes.Group("/docs", func(protected *router.Router) {
	protected.Use(requireDocsAccess)
	protected.Get("/asyncapi.json", Document.Handler())
})
```

This serves `/docs/asyncapi.json` through that middleware. The middleware is
application code, not supplied by AsyncAPI; [group routes](https://arkoren.dev/docs/guides/http-fundamentals/routing.md#group-routes)
describe how to share access controls. If an existing listener or router already
enforces the required access boundary, you can instead register the handler
directly on that restricted router.

The handler returns `application/json`; a successful GET returns 200 with the
document body. HEAD sends headers without a body and, through Arkoren's router,
returns 204 because the handler does not explicitly commit a status.
`Handler()` alone does not register a route, bind a listener, add authentication,
or require lifecycle hooks. It serves the same cached document as `JSON()`;
it does not rebuild the contract on each request.

## Snapshot Behavior

Construction and export happen at different times:

1. `New` runs supplied callbacks synchronously, once in their supplied order,
   including callbacks for invalid declarations.
2. Each message declaration calls its non-nil schema provider once and copies
   the returned bytes immediately. Each `WireExample` encodes and copies its
   value when declared. Raw objects and retained construction drafts are also
   detached from the published document.
3. The first export checks the completed document graph. `JSON()` and every
   handler from that document share one cached result, including an error.
   Concurrent callers do not repeat validation or schema-provider calls.

Later changes to captured inputs or retained builders cannot change the contract.
Do not mutate inputs concurrently with capture, and keep providers free of network
resolution. The package performs no broker I/O; application callbacks and
providers remain your code.

Every successful `JSON()` call returns a detached copy of the cached bytes.
Changing that slice cannot affect later exports or HTTP responses. The document
has no invalidation or reload method: construct a new one when the contract
changes. There is no `Start`, `Stop`, or cleanup to register.

## Declaration Failures

Export during a test or build step, before exposing the route. Empty title or
version, duplicate IDs or selections, unresolved local references, unsupported
schemas, invalid job/topic definitions, and invalid metadata become errors from
`JSON()` or the handler. Forward references are valid if their targets exist in
the completed document. The first captured declaration error is retained even
if a later declaration would otherwise replace its value.

The handler returns generation errors before setting headers or writing a body,
so your [router's error handling](https://arkoren.dev/docs/guides/http-fundamentals/error-handling.md) controls
the response. Response-write errors also propagate, but can occur after headers
or part of the body have already been sent.

Omitting construction callbacks is valid. Supplying a nil callback panics, and
application panics are not recovered. Nil or typed-nil schema providers instead
become deferred export errors without invoking their methods. A nil or zero
`Document` returns an error from `JSON()`; construct documents with `New`.

## External Viewers

The package includes no AsyncAPI viewer. Use a compatible external viewer if
you need a browser interface, and check whether it displays the
[`x-arkoren-logical-payload` annotation](https://arkoren.dev/docs/guides/asyncapi/messages-and-schemas.md#wire-and-logical).
Otherwise, inspect the logical schema component separately from the envelope.

Swagger UI displays OpenAPI HTTP documents, not these messaging contracts.
[OpenAPI serving](https://arkoren.dev/docs/guides/openapi/serving.md) belongs to that separate contract.
Do not upload internal documents to a hosted viewer unless your data-sharing
policy permits exposing their metadata and examples.

For export and handler behavior, see the
[`asyncapi` API reference](https://pkg.go.dev/arkoren.dev/core/asyncapi).