Indexing Documents

A search document is a projection of a record your application owns elsewhere. Keep only the fields search needs, and retain a source from which you can rebuild the index. Arkoren stores the JSON you send; it does not load records, follow database changes, or remove private fields for you.

Document Shape

For a tenant-scoped catalog, define the searchable representation explicitly. Import search from arkoren.dev/core/service/search and standard-library time for the timestamp:

type Product struct {
	ID          string    `json:"id"`
	TenantID    string    `json:"tenant_id"`
	Title       string    `json:"title"`
	Description string    `json:"description"`
	Category    string    `json:"category"`
	Price       float64   `json:"price"`
	InStock     bool      `json:"in_stock"`
	Backorder   bool      `json:"backorder"`
	CreatedAt   time.Time `json:"created_at"`
}

Fields in a declaration refer to top-level JSON keys, not Go field names or database db tags. Use UTC timestamps for text-based time comparisons and keep one scalar kind per filterable/sortable field across documents. Searchable fields contain strings; filterable and sortable fields contain strings, numbers, or booleans.

The whole encoded document is stored and returned, including fields not listed in the index policy. Excluding a field from Searchable does not make it private. Do not index secrets or unnecessary personal data. Tenant fields permit permission filters; they do not enforce them automatically.

Field Policies

Declare which fields each search operation can use, and select a stable document ID from the source record:

var Products = search.NewIndex("products", func(product Product) string {
	return product.ID
}, func(fields *search.IndexBuilder) {
	fields.Searchable("title", "description")
	fields.Filterable("tenant_id", "category", "price", "in_stock", "backorder", "created_at")
	fields.Sortable("price", "created_at")
})

Products is immutable. Builder methods append fields; publication copies field lists and detaches retained drafts. Field getters and Schema() return sorted, deduplicated copies. The key function and its captured state are not cloned, so keep identity selection stable and safe for concurrent calls.

IDs must be unique within the whole index, including across tenants. A tenant filter does not partition document identity; two records with the same ID replace each other. Use source IDs that already satisfy that scope, or encode both source and tenant identity into an allowed stable ID.

Index and field names are 1–64 bytes of lowercase letters, digits, and underscores. A declaration needs a key function and at least one searchable field. Products.Err() wraps search.ErrInvalidName for an invalid declaration. Callbacks run synchronously once in order; supplied nil callbacks panic, while expanded empty callback slices add no edits. Validation uses the completed draft.

Register Products with Builder.Indexes when constructing the engine in Search Drivers, then synchronize it before writes. Duplicate names and invalid registrations fail every operational engine call. Custom search.Indexer implementations must provide both Schema() and Err(); registration checks Err() first. Keep custom definitions stable for the engine’s lifetime and report complete validation failures, not only schema-visible ones.

Putting Documents

Assume engine registers Products, synchronization succeeded, and ctx is your operation context. Index a product already validated and saved in your source of truth:

product := Product{
	ID:       "p-17",
	TenantID: "shop-4",
	Title:    "Red running shoes",
	Category: "shoes",
	Price:    85,
	InStock:  true,
}
err := engine.Put(ctx, Products, product)

Put encodes the value with encoding/json/v2, derives its ID with the key function, and upserts it. After success it is searchable, subject to Postgres transaction visibility. A missing backend index returns search.ErrNotFound; writes do not implicitly synchronize it.

To reflect a changed source record, put the complete projection again:

product.Price = 79
err = engine.Put(ctx, Products, product)

The stored document is replaced entirely, not patched. A partially populated Product replaces prior values with the JSON that value encodes. There is no portable partial-update operation. Source writes and search writes are separate unless you deliberately use the Postgres driver’s shared database transaction. For other backends, schedule indexing after a successful source commit and retain enough information to retry failed updates.

Removing Documents

When an authorized source deletion or visibility change removes products from search, delete their stable IDs:

err := engine.Delete(ctx, Products, "p-17", "p-18")

Missing document IDs succeed, so retries are safe. An absent index still returns search.ErrNotFound. IDs are 1–255 bytes of ASCII letters, digits, hyphens, and underscores; invalid IDs return search.ErrInvalidID before deletion starts. Select deletion targets from trusted application state, not an unauthorized request’s arbitrary IDs.

Delete removes documents, not the index declaration. Drop destroys the entire backend index and its documents; keep that action in a planned deployment or rebuild, never an ordinary document update.

Batch Behavior

For a backfill, submit a bounded source batch such as an existing []Product named products:

err := engine.Put(ctx, Products, products...)

The facade validates every ID and encodes every document before the first driver write. Invalid IDs, encoding failures, or values that do not encode as JSON objects write nothing. Duplicate IDs within one call keep the last document. An empty Put or Delete does no driver work after definition/context checks.

This preflight is not a transaction. Drivers can split large batches; a later failure or cancellation can follow successful writes. Repeating the same upsert batch finishes it safely, but coordinate retries with newer source updates so an old retry does not overwrite a newer projection. Meilisearch can continue a server task after the caller stops waiting. Consult the provider limits for reserved fields, field lengths, and native task behavior.

Choose application batch sizes to bound encoding memory and retry work; there is no portable streaming bulk writer. Cancellation is checked between key and encoding callbacks and before passing the batch to the driver, not inside an individual callback. Once driver work begins, reconcile outcomes rather than assuming cancellation rolled them back.

For every function and option, see the search API reference.