> ## Documentation Index
> Fetch the complete documentation index at: https://docs.basaltic.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Telemetry

> Ingest and search logs, metrics and traces, with OTLP and Prometheus remote_write as first-class entry points.

Telemetry stores the three observability signals your workloads emit: log
records grouped into log groups, metric samples on a timeseries store, and
trace spans you can reassemble into a waterfall. You write through a native
JSON API, an OTLP receiver, or Prometheus `remote_write` — whichever your
existing agent already speaks.

The service is **regional**. Data is stored in the region you wrote it to, and
there is no cross-region read:

```
https://telemetry.sa-saopaulo-1.basaltic.sh   native API
https://otlp.sa-saopaulo-1.basaltic.sh        OTLP receiver
```

<CardGroup cols={2}>
  <Card title="Logs" icon="scroll-text" href="#logs">
    Creating a group, ingesting into it, and the search window that is not
    optional.
  </Card>

  <Card title="Metrics" icon="chart-line" href="#metrics">
    What "Prometheus-compatible" does and does not mean here.
  </Card>

  <Card title="Traces" icon="git-branch" href="#traces">
    Span ingest, waterfall reads, and the one retention setting per account.
  </Card>

  <Card title="OTLP" icon="plug" href="#otlp">
    Pointing a collector at us, and the authentication constraint you will
    hit first.
  </Card>
</CardGroup>

<Note>
  Every telemetry endpoint is scoped to one account. Send the account's handle
  in `X-Account-Id`; without it the request is rejected before it reaches a
  handler. The handle selects which account you are acting on — it is not a
  credential, and the IAM check still runs against that account's resources.
</Note>

## Logs

### Create the log group first

A log group is the unit of retention, encryption and IAM scope. Records cannot
be written to a group that does not exist — an ingest naming an unregistered
group has that record rejected, not created for you.

<Tabs>
  <Tab title="Console">
    Go to **Telemetry → Log Groups** and choose **Create Log Group**. Under
    **Log group details** set the **Name** and an optional **Description**;
    under **Retention & encryption** set **Retention**, or turn on **Never
    expire**, and optionally pick an **Encryption key**. **Tags** takes
    key/value labels.

    The **Retention** field will accept any whole number from 1 to 3650, but
    the service accepts only the twenty-two values listed below. A number that
    is not one of them is refused when you submit, not while you type it.
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    POST https://telemetry.sa-saopaulo-1.basaltic.sh/v1/log-groups
    {
      "name": "app/prod/api",
      "description": "Production API request logs",
      "retention_days": 30
    }
    ```
  </Tab>
</Tabs>

`name` is 1–512 characters of `A-Za-z0-9_./#-` and is **immutable**. Renaming a
group would change the CRN every existing policy references, so a rename is a
delete and a recreate.

<Warning>
  A leading `/` is rejected. `/app/prod/api` is not a valid log group name —
  the CRN already uses `/` to separate resource type from id, so a leading
  slash would render as `log-group//app/prod/api`. Slashes **inside** the name
  are fine and encouraged.
</Warning>

Hierarchical names pay off in IAM. The name lands in the CRN verbatim, so a
policy can wildcard a whole subtree:

```
crn:telemetry:sa-saopaulo-1:my-account:log-group/app/prod/api
crn:telemetry:sa-saopaulo-1:my-account:log-group/app/prod/*
```

### Retention

`retention_days` accepts one of twenty-two values, or `null` for never expire:

```
1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180,
365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653
```

<Info>
  The closed set is not arbitrariness for its own sake. Retention is part of
  the storage partition key, which is what lets expiry drop a whole partition
  instead of rewriting one to strip short-lived rows out from between
  long-lived ones. With free-form integers the partition count becomes a
  function of how many distinct numbers customers happen to type. Twenty-two
  options bound it.
</Info>

Retention applies to records as they are written. Lowering it reaches only
records ingested after the change; what is already stored keeps the retention
it was stamped with.

Moving a group to never-expire after it has a bounded value takes an explicit
clear — `retention_days: null` alone is not enough.

<Tabs>
  <Tab title="Console">
    Open the group from **Telemetry → Log Groups**, turn on **Never expire**
    under **Settings**, and choose **Save**.
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    PATCH /v1/log-groups/{id}
    { "clear_retention": true }
    ```
  </Tab>
</Tabs>

### Ingest

```bash theme={null}
POST /v1/logs
{
  "logs": [
    {
      "log_group": "app/prod/api",
      "log_stream": "api-host-07",
      "severity": "INFO",
      "body": "request completed status=200 dur=12ms",
      "attributes": { "route": "/v1/users", "status": "200" },
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "span_id": "00f067aa0ba902b7"
    }
  ]
}
```

`log_group`, `log_stream` and `body` are required per record. `log_stream` is
free-form and conventionally identifies the producer — a host, a container, a
task. `timestamp` is optional and defaults to ingest time.

<Warning>
  **A `202` does not mean every record landed.** The status reports that the
  batch was accepted at the wire level. Read `accepted`, `rejected` and
  `errors` in the body — a record naming a group that does not exist, or
  failing validation, is dropped individually while the rest of the batch
  flows:

  ```json theme={null}
  { "accepted": 998, "rejected": 2,
    "errors": ["record 3: log group \"app/prod/unregistered\" does not exist in this account (create it first)"] }
  ```
</Warning>

Two limits bound one call: at most **1000 records** per batch, and a **4 MiB**
request body.

### Timestamps you supply are bounded

A caller-supplied `timestamp` is accepted only between **2000-01-01** and
**24 hours ahead** of the receiving server's clock. The skew allowance covers a
drifting producer clock and a batching exporter.

<Info>
  The ceiling exists because your timestamp decides both which retention
  partition a record lands in and when expiry drops it. A record stamped in
  2200 would sit alone in a partition nothing queries and outlive its retention
  window by however far ahead it was stamped.
</Info>

Two attribute keys are reserved: `basaltic_account_id` and `basaltic_org_id`
are stripped from whatever you send and set from your signed identity. A
workload with shell access on one of your instances cannot label its logs as
someone else's.

### Search

```bash theme={null}
GET /v1/logs?from=2026-01-15T00:00:00Z&to=2026-01-16T00:00:00Z
    &log_group=app/prod/api
    &min_severity=WARN
    &q=status=500
    &attr.route=/v1/users
```

`from` and `to` are **required**, and the window must be at most **31 days** —
an unbounded search would turn into a full-retention scan. Results come back
newest-first with an opaque `marker` cursor.

| Parameter                  | Effect                                                                      |
| -------------------------- | --------------------------------------------------------------------------- |
| `q`                        | Case-insensitive substring against the record body                          |
| `attr.<key>=<value>`       | Exact match on one attribute; repeat for more                               |
| `min_severity`             | Records at or above the band: `TRACE` `DEBUG` `INFO` `WARN` `ERROR` `FATAL` |
| `log_group` / `log_stream` | Narrow to a group, or a stream inside it                                    |
| `trace_id`                 | Every record correlated to one trace                                        |

`trace_id` is the useful one when you already have a trace: it pulls the log
lines emitted inside those spans, so you can read a request's logs and its
waterfall against each other.

### Deleting a group

Deleting removes the group's administrative record. Log records already written
keep their reference to it and keep expiring on their own schedule — but the
group name no longer resolves, so you cannot search them by group after the
delete.

<Tabs>
  <Tab title="Console">
    On **Telemetry → Log Groups**, each row carries a **Delete log group**
    action.
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    DELETE /v1/log-groups/{id}
    ```
  </Tab>
</Tabs>

## Metrics

### What "Prometheus-compatible" means here

Precisely two things, and it is worth being clear about the third:

<Columns cols={2}>
  <Card title="Compatible" icon="check">
    **Ingest** is the real `remote_write` protocol — a snappy-compressed
    protobuf `WriteRequest`, byte-for-byte what a Prometheus server sends.

    **Response envelopes** are Prometheus's — `status`, `data.resultType`
    (`matrix` or `vector`) and `data.result` — with sample values as strings
    and timestamps as unix seconds, so an existing chart renderer reads them
    unchanged.
  </Card>

  <Card title="Not compatible" icon="x">
    **The query language is not PromQL.** There is no `query=` expression
    parameter. You select a metric, filter it with label matchers, group it
    and aggregate it through structured parameters.

    Joins, `histogram_quantile`, and arbitrary expressions have no equivalent
    here. A Grafana Prometheus data source will not work against these
    endpoints.
  </Card>
</Columns>

### Ingest

```bash theme={null}
POST /v1/metrics/write
Content-Type: application/x-protobuf
<snappy-compressed prometheus WriteRequest>
```

Returns **`204`** on success, per the Prometheus specification. Every timeseries
is stamped with your account and organization on the way in, and any tenant
labels the payload already carried are stripped first — a producer cannot claim
another account's series.

### Querying

An instant query returns a vector — the aggregate over one lookback window
ending at `time`:

```bash theme={null}
GET /v1/metrics/query?metric=http_requests_total
    &agg=rate
    &match[]=job="api"
    &by[]=route
    &step=5m
```

A range query returns a matrix over buckets of width `step`:

```bash theme={null}
GET /v1/metrics/query_range?metric=http_requests_total
    &agg=rate&match[]=job="api"&by[]=route
    &start=2026-01-15T09:00:00Z&end=2026-01-15T10:00:00Z&step=15s
```

<ResponseField name="metric" type="required">
  The metric name. One metric per query.
</ResponseField>

<ResponseField name="agg" type="avg | sum | min | max | count | last | rate | increase">
  How samples collapse inside a bucket. Defaults to `avg` when omitted. `rate`
  and `increase` derive from successive counter samples and are reset-guarded,
  so a counter restart does not read as a spike.
</ResponseField>

<ResponseField name="match[]" type="repeated">
  Label matchers using PromQL's four operators — `=`, `!=`, `=~`, `!~`. For
  example `match[]=job="api"` and `match[]=route=~/v1/.*`.
</ResponseField>

<ResponseField name="by[]" type="repeated">
  Group-by label names. Omit it and you get one series per distinct label set.
</ResponseField>

<ResponseField name="step" type="duration">
  Bucket width on `query_range` (defaults to `60s`, minimum `1s`); lookback
  window on `query` (defaults to `5m`).
</ResponseField>

Every query endpoint also accepts `POST` with a form-encoded body, which is how
you send a matcher set too long to fit in a URL.

<Warning>
  A range query is capped at **11 000 points**. `start`/`end` divided by `step`
  above that is rejected with `query yields too many points; widen step or
      shorten the window` rather than materialising the matrix. Widen `step`
  first — it is almost always the wrong knob that got turned.
</Warning>

### Discovery

`GET /v1/metrics/names?start=…&end=…` returns the distinct metric names you
emitted in the window. `GET /v1/metrics/series?metric=…&start=…&end=…` returns
the distinct label sets for one metric. Together they are what a dashboard
builder needs to offer a picker instead of a blank text field.

### Metric retention is fixed at 30 days

Unlike logs and traces, metric retention is not per-tenant and not
configurable — samples expire **30 days** after their timestamp. That is also
why the query window is capped at 31 days: a longer window could only ever
return a partly-empty range.

## Traces

### Ingesting spans

```bash theme={null}
POST /v1/spans
{
  "spans": [
    {
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "span_id": "00f067aa0ba902b7",
      "parent_span_id": "a2fb4a1d1a96d312",
      "name": "GET /api/users",
      "kind": "SERVER",
      "service_name": "api-gateway",
      "start_time": "2026-01-15T09:30:00Z",
      "end_time": "2026-01-15T09:30:00.012Z",
      "status_code": "OK"
    }
  ]
}
```

Same batch shape as logs: up to 1000 spans, `202` with per-record `rejected`
and `errors`.

<Warning>
  **A span needs a service name even though the request schema does not mark
  it required.** It is taken from the top-level `service_name`, or from
  `resource["service.name"]` if that is absent. A span carrying neither is
  rejected — with no service, a trace cannot be attributed to anything on the
  read side.
</Warning>

`trace_id` is 32 lower-hex characters and `span_id` is 16, matching the
OpenTelemetry wire format. `parent_span_id` is empty for a root span.
`end_time` must not precede `start_time`. Span timestamps are bounded by the
same ingest window as logs.

`kind` defaults to `INTERNAL` and `status_code` to `UNSET`.

### Reading traces

```bash theme={null}
GET /v1/traces?from=2026-01-15T00:00:00Z&to=2026-01-15T01:00:00Z
    &service=api-gateway&status_code=ERROR&min_duration_ms=100
```

This returns one summary per distinct trace — root operation, root service,
duration, span count, error count, service count — which is the shape a trace
list renders from without fetching anything else.

`GET /v1/traces/{trace_id}` then returns every span in that trace, ordered by
`start_time` ascending, so it can be drawn as a waterfall directly.

The 31-day window cap applies here too, and `from`/`to` are required.

### Trace settings

Trace retention is set **once per account**, not per log group — one setting
decides the fate of everything the account ever traces.

```bash theme={null}
PUT /v1/trace-settings
{ "retention_days": 90 }
```

The same twenty-two retention values apply, for the same partitioning reason.

<Note>
  Trace settings are **API only**. The console's **Telemetry → Traces** page
  reads traces and nothing else — there is no control there for span retention
  or for the key spans are encrypted under, so a `PUT` is the only way to
  change either.
</Note>

<Warning>
  **Spans have no never-expire option**, and the two ways you might reach for
  one both reset you to the default instead:

  * `clear_retention: true` sets retention back to **30 days**. On a log group
    it means never expire; on trace settings it does not.
  * Omitting `retention_days` from the `PUT` body does the same — this is a
    `PUT`, so the body is the full intent, and a missing retention is not
    "leave it alone".

  Read the response back to confirm what you got.
</Warning>

A trace is the highest-volume signal the platform accepts — one row per
operation, carrying attributes, events and links. The ceiling, 3653 days, is
already past any horizon a trace is read at.

Like log groups, a retention change reaches only spans ingested after it. An
account that has never written settings reads back the default of 30 days,
which is exactly what its spans are being stamped with.

<Note>
  **There is no server-side sampling control.** Trace settings cover retention
  and key association, nothing else — the API stores every span you send it.
  Sample in your SDK or collector, before the data leaves your workload.
</Note>

## Encryption at rest

A log group and an account's trace settings each take an optional
`kms_key_crn`. When one is set, record bodies (and, for spans, the
customer-typed bag of name, status message, attributes, events and links) are
envelope-encrypted under that key before storage.

Indexed fields stay in the clear so search still works without unwrapping
anything: for spans that is `service_name`, `trace_id`, `span_id`, timing and
`status_code`.

<Warning>
  Associating or disassociating a key affects only data ingested **after** the
  change. Already-stored records keep whatever encryption state they were
  written with. Pass an empty string to disassociate.
</Warning>

<Note>
  A log group's key is settable in the console: **Encryption key** on **Create
  Log Group**, or on an existing group's **Settings** followed by **Save**. The
  span-side association lives on trace settings, which have no console page, so
  that one is API only.
</Note>

## OTLP

The OTLP receiver is a separate host serving the canonical OpenTelemetry paths,
so an SDK resolves them from one variable:

```bash theme={null}
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.sa-saopaulo-1.basaltic.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

| Signal  | Path               |
| ------- | ------------------ |
| Logs    | `POST /v1/logs`    |
| Traces  | `POST /v1/traces`  |
| Metrics | `POST /v1/metrics` |

<Warning>
  **Binary protobuf only.** `Content-Type` must be `application/x-protobuf`;
  the JSON protobuf encoding is rejected. Set
  `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`, not `http/json`.
</Warning>

Rejections come back in OTLP's own `partial_success` envelope with a
`rejected_log_records` count and a joined error message, rather than as an HTTP
error — the same per-record semantics as the native ingest, expressed in the
protocol's shape.

### Resolving the log group

OTLP has no log-group concept, so the receiver derives one from resource
attributes:

| Field      | Resolved from, in order                                                             |
| ---------- | ----------------------------------------------------------------------------------- |
| Log group  | `basaltic.log_group`, then `service.name`                                           |
| Log stream | `basaltic.log_stream`, then `service.instance.id`, then `host.name`, then `default` |

The resolved group still has to exist in the account. Create it before you
point a collector at the receiver, or every record comes back rejected.

Severity uses `severity_text` when the SDK set one; otherwise the numeric band
maps per the OpenTelemetry specification — 1-4 `TRACE`, 5-8 `DEBUG`, 9-12
`INFO`, 13-16 `WARN`, 17-20 `ERROR`, 21-24 `FATAL`.

### The authentication constraint

<Warning>
  **Every OTLP request must be signed**, exactly like every other Basaltic API
  call — `BASALTIC-HMAC-SHA256` over a canonical form of the request, with
  `X-Date` and a per-request `X-Nonce`, plus `X-Account-Id`. There is no static
  bearer token or API-key header that a stock exporter can be configured with.

  A signature is valid for five minutes, so this is not something you can
  precompute into a config file either. In practice you need an exporter that
  can sign each request, or a signing proxy between your collector and the
  receiver.
</Warning>

The same applies to `POST /v1/metrics/write`: a Prometheus server's
`remote_write` block has no way to produce this signature on its own. See
[Authentication](/authentication) for the full signing procedure.

## Permissions

Telemetry actions authorize against the CRN of the thing being touched, so a
policy can be scoped to one group or one naming convention. See
[policies](/iam/policies) for how the evaluation works.

| Action                                                                                                         | Resource CRN                                                          |
| -------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| `telemetry:CreateLogGroup` `telemetry:UpdateLogGroup` `telemetry:DeleteLogGroup` `telemetry:DescribeLogGroups` | `log-group/<name>`                                                    |
| `telemetry:WriteLogs`                                                                                          | `log-group/<name>` — checked once per distinct group in a batch       |
| `telemetry:ReadLogs`                                                                                           | `log-group/<name>` when the search names one, `log-group/*` otherwise |
| `telemetry:WriteSpans` `telemetry:ReadTraces`                                                                  | `trace/*`                                                             |
| `telemetry:GetTraceSettings` `telemetry:PutTraceSettings`                                                      | `trace-settings/default`                                              |
| `telemetry:WriteMetrics` `telemetry:ReadMetrics`                                                               | account-scoped                                                        |

A batch that touches several log groups is authorized per group. One denied
group fails only its own records — the rest of the batch is written.

## Limits

<ResponseField name="Batch size" type="1000 records">
  Per `POST /v1/logs` and per `POST /v1/spans`.
</ResponseField>

<ResponseField name="Request body" type="4 MiB">
  On both the native API and the OTLP receiver.
</ResponseField>

<ResponseField name="Search window" type="31 days">
  Required and capped on `GET /v1/logs`, `GET /v1/traces`, and every metric
  query.
</ResponseField>

<ResponseField name="Range query points" type="11 000">
  `(end - start) / step` on `query_range`.
</ResponseField>

<ResponseField name="Ingest timestamp window" type="2000-01-01 to now + 24h">
  Applies to log timestamps and to span `start_time` / `end_time`.
</ResponseField>

<ResponseField name="Metric retention" type="30 days">
  Fixed. Log and trace retention are yours to choose.
</ResponseField>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Ingest returns 202 but nothing is searchable" icon="triangle-alert">
    Read `rejected` and `errors` in the `202` body. The most common entry is a
    log group that was never created — ingest is strict, and naming an unknown
    group rejects that record rather than creating the group.

    The second most common is a timestamp outside the accepted window. Check
    the producer's clock: anything more than 24 hours ahead of ours is dropped
    per record.
  </Accordion>

  <Accordion title="Creating a log group returns 400 on the name" icon="slash">
    Names cannot start with `/`. `app/prod/api` is valid; `/app/prod/api` is
    not. Slashes elsewhere in the name are fine.

    The full rule is 1–512 characters of `A-Za-z0-9_./#-`, and the first
    character cannot be `/`.
  </Accordion>

  <Accordion title="retention_days is rejected" icon="calendar">
    Retention is a closed set, not a range: 1, 3, 5, 7, 14, 30, 60, 90, 120,
    150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653.
    A value like 45 or 3650 is refused even though it falls between valid
    entries.
  </Accordion>

  <Accordion title="Trace retention keeps reverting to 30 days" icon="rotate-ccw">
    `PUT /v1/trace-settings` replaces the whole settings document. Omitting
    `retention_days` — or sending `clear_retention: true` hoping for
    never-expire — resets to the 30-day default, because spans have no
    never-expire option.

    Send the retention you want on every `PUT`, and read the response back.
  </Accordion>

  <Accordion title="A metric query returns an empty result" icon="chart-line">
    Confirm the metric name with `GET /v1/metrics/names` for the same window —
    a name that was never emitted returns success with an empty result, not an
    error. Then check the label set with `GET /v1/metrics/series`: a matcher
    against a label the series does not carry filters everything out.

    Also check the window against retention. Metrics expire after 30 days, so
    a query near the edge of the 31-day cap can be reading past the data.
  </Accordion>

  <Accordion title="A Grafana Prometheus data source will not connect" icon="plug">
    It cannot. The read endpoints share Prometheus's response envelope but not
    its query language or its URL layout — there is no `/api/v1/query` and no
    `query=<promql>` parameter, and the endpoints require request signing that
    the data source cannot produce.

    Ingest is the compatible half: `remote_write` from Prometheus or an agent
    works, given a way to sign the request.
  </Accordion>

  <Accordion title="A collector gets 401 on every export" icon="key">
    OTLP is authenticated exactly like the rest of the API, and a static header
    will not satisfy it. You need per-request `BASALTIC-HMAC-SHA256` signing
    with a fresh `X-Date` and `X-Nonce`, plus `X-Account-Id` for the account
    you are writing to.

    If the signature is right and you still see `401`, check clock skew first —
    a signature is valid for five minutes from its `X-Date`.
  </Accordion>
</AccordionGroup>

## Next

<CardGroup cols={2}>
  <Card title="Authentication" icon="key" href="/authentication">
    The signing procedure every ingest and query request needs.
  </Card>

  <Card title="Policies" icon="shield" href="/iam/policies">
    Scoping a policy to a log group subtree.
  </Card>

  <Card title="Regions" icon="globe" href="/regions">
    Which host to call, and why telemetry is regional.
  </Card>

  <Card title="API reference" icon="code" href="/api-reference/introduction">
    Every telemetry operation, with request and response schemas.
  </Card>
</CardGroup>