Telemetry

Client examples use CLI v0.13.0 and Go SDK v0.15.0. See CLI setup and Go configuration. Go snippets assume a configured cfg, a ctx from context.Background(), and imports for log, basaltic (github.com/basaltic-sh/sdk-go) and telemetry (github.com/basaltic-sh/sdk-go/telemetry). LOG_GROUP_ID / logGroupID is the returned log-group UUID. See Resource references for accepted reference kinds, lookup scopes, canonical identities and exact list filters. Telemetry stores the three observability signals your workloads emit: log records grouped into log groups, metric samples on a timeseries store, and trace spans you can reassemble into a waterfall. You write through a native JSON API, an OTLP receiver, or Prometheus remote_write — whichever your existing agent already speaks. The service is regional. Data is stored in the region you wrote it to, and there is no cross-region read:

https://telemetry.sa-saopaulo-1.basaltic.sh   native API
https://otlp.sa-saopaulo-1.basaltic.sh        OTLP receiver

Logs

Creating a group, ingesting into it, and the search window that is not optional.

Metrics

What “Prometheus-compatible” does and does not mean here.

Traces

Span ingest, waterfall reads, and the one retention setting per account.

OTLP

Pointing a collector at us, and the authentication constraint you will hit first.

Every telemetry endpoint is scoped to one account. Send the account’s handle in X-Account-Id; without it the request is rejected before it reaches a handler. The handle selects which account you are acting on — it is not a credential, and the IAM check still runs against that account’s resources.

Log responses include log_group as a ResourceRef with id, crn and the current name. The reference remains when the group is deleted, without a name. Ingest and search selectors remain strings. GET /v1/retained-data reports {retained_data: {has_data}} for all logs, spans and metrics in the account and region, including expired data awaiting removal. It requires telemetry:GetRetainedData and does not expose record contents.

Logs

Create the log group first

A log group is the unit of retention, encryption and IAM scope. Records cannot be written to a group that does not exist — an ingest naming an unregistered group has that record rejected, not created for you.

Console
API
CLI
Go

Go to Telemetry → Log Groups and choose Create Log Group. Under Log group details set the Name and an optional Description; under Retention & encryption set Retention, or turn on Never expire, and optionally pick an Encryption key. Tags takes key/value labels.The Retention field will accept any whole number from 1 to 3650, but the service accepts only the twenty-two values listed below. A number that is not one of them is refused when you submit, not while you type it.

POST https://telemetry.sa-saopaulo-1.basaltic.sh/v1/log-groups
{
  "name": "app/prod/api",
  "description": "Production API request logs",
  "retention_days": 30
}

basaltic telemetry log-group create --name app/prod/api --description "Production API request logs" --retention-days 30

_, err := telemetry.New(cfg).CreateLogGroup(ctx, &telemetry.CreateLogGroupRequest{
    Name: "app/prod/api", Description: basaltic.String("Production API request logs"),
    RetentionDays: basaltic.Int(30),
})
if err != nil {
    log.Fatal(err)
}

name is 1–512 characters of A-Za-z0-9_./#- and is immutable. Renaming a group would change the CRN every existing policy references, so a rename is a delete and a recreate.

A leading / is rejected. /app/prod/api is not a valid log group name — the CRN already uses / to separate resource type from id, so a leading slash would render as log-group//app/prod/api. Slashes inside the name are fine and encouraged.

Hierarchical names pay off in IAM. The name lands in the CRN verbatim, so a policy can wildcard a whole subtree:

crn:telemetry:sa-saopaulo-1:my-account:log-group/app/prod/api
crn:telemetry:sa-saopaulo-1:my-account:log-group/app/prod/*

Retention

retention_days accepts one of twenty-two values, or null for never expire:

1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180,
365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653

The closed set is not arbitrariness for its own sake. Retention is part of the storage partition key, which is what lets expiry drop a whole partition instead of rewriting one to strip short-lived rows out from between long-lived ones. With free-form integers the partition count becomes a function of how many distinct numbers customers happen to type. Twenty-two options bound it.

Retention applies to records as they are written. Lowering it reaches only records ingested after the change; what is already stored keeps the retention it was stamped with. To make existing retention unlimited, send retention_days: null. Omitting the field preserves its current value.

Console
API
CLI
Go

Open the group from Telemetry → Log Groups, turn on Never expire under Settings, and choose Save.

PATCH /v1/log-groups/{id}
{ "retention_days": null }

echo '{"retention_days":null}' | basaltic telemetry log-group update "$LOG_GROUP_ID" --from-file -

_, err := telemetry.New(cfg).UpdateLogGroup(ctx, logGroupID, &telemetry.UpdateLogGroupRequest{}, basaltic.WithNullFields("retention_days"))
if err != nil {
    log.Fatal(err)
}

Ingest

POST /v1/logs
{
  "logs": [
    {
      "log_group": "app/prod/api",
      "log_stream": "api-host-07",
      "severity": "INFO",
      "body": "request completed status=200 dur=12ms",
      "attributes": { "route": "/v1/users", "status": "200" },
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "span_id": "00f067aa0ba902b7"
    }
  ]
}

log_group, log_stream and body are required per record. log_group accepts an exact name, UUID or CRN in your account; CRNs must match the serving region and your account handle. Invalid references never fall back to another lookup. The same contract applies to the log_group search filter. List log groups with exact name and crn filters; supplying both intersects them, including pagination. Malformed or empty CRN filters return 400; valid foreign or mismatched CRNs return an empty page. log_stream is free-form and conventionally identifies the producer — a host, a container, a task. timestamp is optional and defaults to ingest time.

A 202 does not mean every record landed. The status reports that the batch was accepted at the wire level. Read accepted, rejected and errors in the body — a record naming a group that does not exist, or failing validation, is dropped individually while the rest of the batch flows:

{ "ingestion": { "accepted": 998, "rejected": 2,
  "errors": [
    {"index": 3, "code": "NOT_FOUND", "message": "Log group does not exist"},
    {"index": 7, "code": "INVALID_INPUT", "message": "body is required"}
  ] } }

Native log and span ingestion returns {ingestion: {accepted, rejected, errors}}. Each error contains the zero-based input index, a stable code and a readable message. OTLP keeps its standard partial-success response. Two limits bound one call: at most 1000 records per batch, and a 4 MiB request body.

Timestamps you supply are bounded

A caller-supplied timestamp is accepted only between 2000-01-01 and 24 hours ahead of the receiving server’s clock. The skew allowance covers a drifting producer clock and a batching exporter.

The ceiling exists because your timestamp decides both which retention partition a record lands in and when expiry drops it. A record stamped in 2200 would sit alone in a partition nothing queries and outlive its retention window by however far ahead it was stamped.

Two attribute keys are reserved: basaltic_account_id and basaltic_org_id are stripped from whatever you send and set from your signed identity. A workload with shell access on one of your instances cannot label its logs as someone else’s.

Search

GET /v1/logs?from=2026-01-15T00:00:00Z&to=2026-01-16T00:00:00Z
    &log_group=app/prod/api
    &min_severity=WARN
    &q=status=500
    &attr.route=/v1/users

from and to are required, and the window must be at most 31 days — an unbounded search would turn into a full-retention scan. Results come back newest-first with an opaque marker cursor.

Parameter	Effect
`q`	Case-insensitive substring against the record body
`attr.<key>=<value>`	Exact match on one attribute; repeat for more
`min_severity`	Records at or above the band: `TRACE` `DEBUG` `INFO` `WARN` `ERROR` `FATAL`
`log_group` / `log_stream`	Narrow to a group, or a stream inside it
`trace_id`	Every record correlated to one trace

trace_id is the useful one when you already have a trace: it pulls the log lines emitted inside those spans, so you can read a request’s logs and its waterfall against each other.

Deleting a group

Deleting removes the group’s administrative record. Log records already written keep their reference to it and keep expiring on their own schedule — but the group name no longer resolves, so you cannot search them by group after the delete.

Console
API
CLI
Go

On Telemetry → Log Groups, each row carries a Delete log group action.

DELETE /v1/log-groups/{id}

basaltic telemetry log-group delete "$LOG_GROUP_ID"

err := telemetry.New(cfg).DeleteLogGroup(ctx, logGroupID)
if err != nil {
    log.Fatal(err)
}

Metrics

What “Prometheus-compatible” means here

Metrics use Prometheus remote_write for ingestion. The read endpoints use Basaltic structured queries and return query_result with metric and series. Each series has labels and points; every point has a Unix-seconds timestamp and a string value, which can also represent NaN and infinities. These endpoints do not accept PromQL or implement the Grafana Prometheus data source API.

Ingest

POST /v1/metrics/write
Content-Type: application/x-protobuf
<snappy-compressed prometheus WriteRequest>

Returns 204 on success, per the Prometheus specification. Every timeseries is stamped with your account and organization on the way in, and any tenant labels the payload already carried are stripped first — a producer cannot claim another account’s series.

Querying

An instant query returns a vector — the aggregate over one lookback window ending at time:

GET /v1/metrics/query?metric=http_requests_total
    &agg=rate
    &match[]=job="api"
    &by[]=route
    &step=5m

A range query returns a matrix over buckets of width step:

GET /v1/metrics/query-range?metric=http_requests_total
    &agg=rate&match[]=job="api"&by[]=route
    &start=2026-01-15T09:00:00Z&end=2026-01-15T10:00:00Z&step=15s

required

The metric name. One metric per query.

avg | sum | min | max | count | last | rate | increase

How samples collapse inside a bucket. Defaults to avg when omitted. rate and increase derive from successive counter samples and are reset-guarded, so a counter restart does not read as a spike.

repeated

Label matchers using PromQL’s four operators — =, !=, =~, !~. For example match[]=job="api" and match[]=route=~/v1/.*.

repeated

Group-by label names. Omit it and you get one series per distinct label set.

duration

Bucket width on query-range (defaults to 60s, minimum 1s); lookback window on query (defaults to 5m).

Every query endpoint also accepts POST with an application/json body. Use arrays named match and by in JSON; GET keeps repeated match[] and by[] parameters.

A range query is capped at 11 000 points. start/end divided by step above that is rejected with query yields too many points; widen step or shorten the window rather than materialising the matrix. Widen step first — it is almost always the wrong knob that got turned.

Discovery

GET /v1/metrics/names?start=…&end=… returns the distinct metric names you emitted in the window. GET /v1/metrics/series?metric=…&start=…&end=… returns the distinct label sets for one metric. Together they are what a dashboard builder needs to offer a picker instead of a blank text field. Discovery returns {metric_names, meta} or {series, meta}. Set limit (up to 100) and follow meta.marker while meta.has_more is true, keeping the metric and time window unchanged. Results are not silently truncated at 10,000 entries.

Metric retention is fixed at 30 days

Unlike logs and traces, metric retention is not per-tenant and not configurable — samples expire 30 days after their timestamp. That is also why the query window is capped at 31 days: a longer window could only ever return a partly-empty range.

Traces

Ingesting spans

POST /v1/spans
{
  "spans": [
    {
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "span_id": "00f067aa0ba902b7",
      "parent_span_id": "a2fb4a1d1a96d312",
      "name": "GET /api/users",
      "kind": "SERVER",
      "service_name": "api-gateway",
      "start_time": "2026-01-15T09:30:00Z",
      "end_time": "2026-01-15T09:30:00.012Z",
      "status_code": "OK"
    }
  ]
}

Same batch shape as logs: up to 1000 spans, 202 with per-record rejected and errors.

A span needs a service name even though the request schema does not mark it required. It is taken from the top-level service_name, or from resource["service.name"] if that is absent. A span carrying neither is rejected — with no service, a trace cannot be attributed to anything on the read side.

trace_id is 32 lower-hex characters and span_id is 16, matching the OpenTelemetry wire format. parent_span_id is empty for a root span. end_time must not precede start_time. Span timestamps are bounded by the same ingest window as logs. kind defaults to INTERNAL and status_code to UNSET.

Reading traces

GET /v1/traces?from=2026-01-15T00:00:00Z&to=2026-01-15T01:00:00Z
    &service=api-gateway&status_code=ERROR&min_duration_ms=100

This returns one summary per distinct trace — root operation, root service, duration, span count, error count, service count — which is the shape a trace list renders from without fetching anything else. GET /v1/traces/{trace_id} returns {trace: ...} with the same summary as the trace list. Read spans from GET /v1/traces/{trace_id}/spans, which returns {spans, meta} ordered by start_time and span_id. Follow meta.marker while meta.has_more is true. The console loads the waterfall in pages with Load more. The 31-day window cap applies here too, and from/to are required.

Trace settings

Trace retention is set once per account, not per log group — one setting decides the fate of everything the account ever traces.

PATCH /v1/trace-settings
{ "retention_days": 90 }

The same twenty-two retention values apply, for the same partitioning reason.

Trace settings are API only. The console’s Telemetry → Traces page reads traces and nothing else — there is no control there for span retention or for the key spans are encrypted under, so a PATCH is the only way to change either.

Spans always expire. Omit retention_days to preserve its value, or send null to restore the default 30 days. Log-group retention uses null for no expiry.

A trace is the highest-volume signal the platform accepts — one row per operation, carrying attributes, events and links. The ceiling, 3653 days, is already past any horizon a trace is read at. Like log groups, a retention change reaches only spans ingested after it. An account that has never written settings reads back the default of 30 days, which is exactly what its spans are being stamped with.

There is no server-side sampling control. Trace settings cover retention and key association, nothing else — the API stores every span you send it. Sample in your SDK or collector, before the data leaves your workload.

Encryption at rest

A log group and an account’s trace settings each take an optional kms_key. When set, record bodies (and span names, status messages, attributes, events and links) are envelope-encrypted under that key. Inputs accept its UUID, CRN or exact name in your account and region. Log-group responses return kms_key as a ResourceRef with id, crn, and name when available. Both log groups and trace settings return kms_key as a ResourceRef. On PATCH, omission preserves the key; kms_key: null clears encryption for future data. In-use keys, including keys protecting retained encrypted data, cannot be deleted. Existing data stays bound to its original key UUID. Indexed fields stay in the clear so search still works without unwrapping anything: for spans that is service_name, trace_id, span_id, timing and status_code.

Associating or disassociating a key affects only data ingested after the change. Already-stored records keep whatever encryption state they were written with. Send kms_key: null to disassociate.

A log group’s key is settable in the console: Encryption key on Create Log Group, or on an existing group’s Settings followed by Save. The span-side association lives on trace settings, which have no console page, so that one is API only.

OTLP

The OTLP receiver is a separate host serving the canonical OpenTelemetry paths, so an SDK resolves them from one variable:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.sa-saopaulo-1.basaltic.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Signal	Path
Logs	`POST /v1/logs`
Traces	`POST /v1/traces`
Metrics	`POST /v1/metrics`

Binary protobuf only. Content-Type must be application/x-protobuf; the JSON protobuf encoding is rejected. Set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, not http/json.

Rejections come back in OTLP’s own partial_success envelope with a rejected_log_records count and a joined error message, rather than as an HTTP error — the same per-record semantics as the native ingest, expressed in the protocol’s shape.

Resolving the log group

OTLP has no log-group concept, so the receiver derives one from resource attributes:

Field	Resolved from, in order
Log group	`basaltic.log_group`, then `service.name`
Log stream	`basaltic.log_stream`, then `service.instance.id`, then `host.name`, then `default`

On both HTTP and gRPC, either selector accepts an exact log-group name, UUID or CRN. The full slash-bearing name is preserved. The resolved group still has to exist in the account. Create it before you point a collector at the receiver, or every record comes back rejected. Severity uses severity_text when the SDK set one; otherwise the numeric band maps per the OpenTelemetry specification — 1-4 TRACE, 5-8 DEBUG, 9-12 INFO, 13-16 WARN, 17-20 ERROR, 21-24 FATAL.

Authentication

OTLP over HTTP and gRPC accepts an OAuth bearer token, with the selected account in X-Account-Id. Use Authorization: Bearer <access_token> for HTTP, or equivalent lowercase gRPC metadata. The account must match the service account or role session that issued the token. Configure your collector to refresh the token before it expires. An expired token embedded permanently in exporter configuration will stop working. The same bearer authentication applies to POST /v1/metrics/write. See authentication for credential exchange and role sessions.

Permissions

Telemetry actions authorize against the CRN of the thing being touched, so a policy can be scoped to one group or one naming convention. See policies for how the evaluation works.

Action	Resource CRN
`telemetry:CreateLogGroup` `telemetry:UpdateLogGroup` `telemetry:DeleteLogGroup` `telemetry:DescribeLogGroups`	`log-group/<name>`
`telemetry:WriteLogs`	`log-group/<name>` — checked once per distinct group in a batch
`telemetry:ReadLogs`	`log-group/<name>` when the search names one, `log-group/*` otherwise
`telemetry:WriteSpans` `telemetry:ReadTraces`	`trace/*`
`telemetry:GetTraceSettings` `telemetry:PutTraceSettings`	`trace-settings/default`
`telemetry:GetRetainedData`	`retained-data/default`
`telemetry:WriteMetrics` `telemetry:ReadMetrics`	account-scoped

A batch that touches several log groups is authorized per group. One denied group fails only its own records — the rest of the batch is written.

Limits

1000 records

Per POST /v1/logs and per POST /v1/spans.

4 MiB

On both the native API and the OTLP receiver.

31 days

Required and capped on GET /v1/logs, GET /v1/traces, and every metric query.

11 000

(end - start) / step on query-range.

2000-01-01 to now + 24h

Applies to log timestamps and to span start_time / end_time.

30 days

Fixed. Log and trace retention are yours to choose.

Troubleshooting

Ingest returns 202 but nothing is searchable

Read rejected and errors in the 202 body. The most common entry is a log group that was never created — ingest is strict, and naming an unknown group rejects that record rather than creating the group.The second most common is a timestamp outside the accepted window. Check the producer’s clock: anything more than 24 hours ahead of ours is dropped per record.

Creating a log group returns 400 on the name

Names cannot start with /. app/prod/api is valid; /app/prod/api is not. Slashes elsewhere in the name are fine.The full rule is 1–512 characters of A-Za-z0-9_./#-, and the first character cannot be /.

retention_days is rejected

Retention is a closed set, not a range: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653. A value like 45 or 3650 is refused even though it falls between valid entries.

Trace retention keeps reverting to 30 days

Explicit null resets trace retention to 30 days. Omitted fields are unchanged. Check the PATCH body if you unexpectedly see the default.

A metric query returns an empty result

Confirm the metric name with GET /v1/metrics/names for the same window — a name that was never emitted returns success with an empty result, not an error. Then check the label set with GET /v1/metrics/series: a matcher against a label the series does not carry filters everything out.Also check the window against retention. Metrics expire after 30 days, so a query near the edge of the 31-day cap can be reading past the data.

A Grafana Prometheus data source will not connect

It cannot. The read endpoints use native JSON envelopes and structured queries — there is no /api/v1/query and no query=<promql> parameter, and they use the platform bearer authentication flow.Ingest is the compatible half: remote_write from Prometheus or an agent works with a valid bearer token and account header.

A collector gets 401 on every export

Check the bearer token’s expiration, account binding, and telemetry permissions. Send Authorization: Bearer <token> and X-Account-Id. Refresh expired tokens through the OAuth endpoint; legacy request signatures are not accepted.

Authentication

The signing procedure every ingest and query request needs.

Policies

Scoping a policy to a log group subtree.

Regions

Which host to call, and why telemetry is regional.

API reference

Every telemetry operation, with request and response schemas.

Get started

Products

Platform

Help

Logs

Metrics

Traces

OTLP

Logs

Create the log group first

Retention

Ingest

Timestamps you supply are bounded

Search

Deleting a group

Metrics

What “Prometheus-compatible” means here

Ingest

Querying

Discovery

Metric retention is fixed at 30 days

Traces

Ingesting spans

Reading traces

Trace settings

Encryption at rest

OTLP

Resolving the log group

Authentication

Permissions

Limits

Troubleshooting

Next

Authentication

Policies

Regions

API reference

Logs

Metrics

Traces

OTLP

​Logs

​Create the log group first

​Retention

​Ingest

​Timestamps you supply are bounded

​Search

​Deleting a group

​Metrics

​What “Prometheus-compatible” means here

​Ingest

​Querying

​Discovery

​Metric retention is fixed at 30 days

​Traces

​Ingesting spans

​Reading traces

​Trace settings

​Encryption at rest

​OTLP

​Resolving the log group

​Authentication

​Permissions

​Limits

​Troubleshooting

​Next

Authentication

Policies

Regions

API reference

Logs

Create the log group first

Retention

Ingest

Timestamps you supply are bounded

Search

Deleting a group

Metrics

What “Prometheus-compatible” means here

Ingest

Querying

Discovery

Metric retention is fixed at 30 days

Traces

Ingesting spans

Reading traces

Trace settings

Encryption at rest

OTLP

Resolving the log group

Authentication

Permissions

Limits

Troubleshooting

Next