Logs, metrics and alarms

Charts are for looking; alarms are for being told. This page covers the three tools in Activity → Logs and alerts: Logs, Metrics and Alarms.

Logs

Activity → Logs searches two kinds of line together:

Your flows’ logs: what Log blocks wrote, with their level (info, warning, error), tagged with the flow (flow:send-receipt).
Access logs: one line per API request, with its level taken from its status (errors for 5xx, warnings for 4xx).

Choose a time range (from the last hour up to 14 days), type text to search for, and read how many lines match. Each line links to the request it belongs to, so you can open its trace.

curl "$PAWABASE_URL/platform/v1/envs/production/logs?q=refund&minutes=1440&limit=200" \
  -H "apikey: $PAWABASE_SECRET_KEY"

const lines = await manage(pawabase).observability.logs({ q: "refund", minutes: 1440 });

minutes is 1 to 20,160 (14 days), limit 1 to 1,000. Pass after (an ISO time) to page forward.

Write useful logs

In a flow, add a Log block where something worth knowing happens: a branch taken, a partner’s unusual answer, a skipped record. Include ids, not personal data or secrets. How long logs are kept depends on your plan (Plans and limits).

Metrics

Activity → Metrics charts any metric over time. Choose the metric, a bucket size (how much time each point covers) and a time range, and optionally compare with the previous period to see whether something changed.

Metric	Unit	What it counts
Requests	count	API requests
Server errors	count	Responses with a 5xx status
Client errors	count	Responses with a 4xx status
Error rate	%	Server errors as a share of requests
Latency, average	ms	Average response time
Latency, 95th percentile	ms	The time 95% of requests finish within
Failed jobs	count	Background jobs that failed
Job failure rate	%	Failed jobs as a share of jobs
Jobs waiting	count	Jobs queued right now (a current value)
Flow runs	count	Flow runs
Failed flow runs	count	Flow runs that failed
Events	count	Events emitted
Mail sent	count	Emails sent
Mail failed	count	Emails that failed
Failed webhook deliveries	count	Outbound deliveries that failed
Workers alive	count	Background workers running now (a current value)
Secrets due for rotation	count	Secrets past their rotation reminder (a current value)

Counts are measured over a window; Jobs waiting, Workers alive and Secrets due for rotation are read as they are right now.

Alarms

An alarm checks a metric every minute and emails people when it crosses a line, and again when it recovers.

Activity → Alarms → New alarm

Give it a Name that says what’s wrong, such as Checkout errors, and a Note that says what to check first.

Choose the metric and the line

Pick a Metric, and Alarm when it is above, at or above, below or at or below a threshold.

Choose the window

Measured over (minutes): how much recent time the metric covers, 1 to 1,440. Server errors over the last 5 minutes.

Choose how sure to be

Checks in a row: how many consecutive minute-checks must cross the line before it fires, 1 to 60. Two or three stops a single blip from waking anyone.

Choose who to tell

Email these people: up to 20 addresses. Remind every (minutes): email again while it stays bad, 0 for never, up to a week.

Save, then test

Send test emails the recipients the current wording, whatever the state, so you know it arrives.

How an alarm decides

State	Meaning
unknown	Not checked yet, or there’s no data
ok	The last check was on the right side of the line
alarm	It crossed the line for the required checks in a row

Each minute the alarm measures the metric. If the value crosses the line, it counts one more breach; once the count reaches Checks in a row, the state becomes alarm and recipients get an email headed [ALARM]. As soon as a check is back on the right side, the count resets, the state becomes ok, and recipients get [OK] … has recovered. The very first check going from unknown to ok sends nothing. Every change of state also emits the alarm.state_changed event, so you can send alerts to chat or anywhere else with a subscription or webhook.

Mute, check, history

Mute silences an alarm’s emails for a number of minutes (up to 30 days) during planned work. It keeps checking and recording; it just doesn’t email. Muting for 0 minutes un-mutes.
Check now evaluates it immediately instead of waiting for the next minute.
History lists every change of state with the value that caused it and whether an email went out (or why not: muted, no recipients).
Enabled off stops checking altogether.

Alarm emails go through your mail delivery. If delivery isn’t set up or is paused, alarms still change state and emit their event, but nobody is emailed.

From code

const m = manage(pawabase);
await m.alarms.create({
  name: "Checkout errors",
  metric: "errors_5xx",
  comparison: "gt",
  threshold: 10,
  windowMinutes: 5,
  breachesNeeded: 2,
  recipients: ["ops@example.com"],
  renotifyMinutes: 30,
});

curl -X POST "$PAWABASE_URL/platform/v1/envs/production/alarms" \
  -H "apikey: $PAWABASE_SECRET_KEY" -H "content-type: application/json" \
  -d '{"name":"Checkout errors","metric":"errors_5xx","comparison":"gt","threshold":10,
       "window_minutes":5,"breaches_needed":2,"recipients":["ops@example.com"],"renotify_minutes":30}'

Metric names for the API: requests, errors_5xx, errors_4xx, error_rate, latency_avg, latency_p95, jobs_failed, job_failure_rate, jobs_waiting, flow_runs, flow_failures, events, mail_sent, mail_failed, webhook_failures, workers_alive, secrets_due. Comparisons: gt, gte, lt, lte.

Alarms worth having

Alarm	Metric	Line	Window	Checks
The API is failing	Error rate	above 2%	5 min	2
It’s slow	Latency, 95th percentile	above 1,500 ms	10 min	3
Background work is failing	Failed jobs	above 5	15 min	1
Work is piling up	Jobs waiting	above 500	now	5
Nothing is processing jobs	Workers alive	below 1	now	2
Email is failing	Mail failed	above 3	30 min	1
A partner is down	Failed webhook deliveries	above 10	30 min	1
Keys are getting old	Secrets due for rotation	above 0	now	1

Tune the numbers to your traffic: a line that’s crossed every day gets ignored.

Good practice

Alarm on what users notice, not on every number.
Use Checks in a row to avoid noise, and Remind every for alarms that matter at night.
Put the first thing to check in the Note.
After every incident, add the alarm you wish you’d had.

Environments

Activity

Secrets and backups

Pawabase Cloud

Logs, metrics and alarms

Logs

Write useful logs

Metrics

Alarms

How an alarm decides

Mute, check, history

From code

Alarms worth having

Good practice

​Logs

​Write useful logs

​Metrics

​Alarms

​How an alarm decides

​Mute, check, history

​From code

​Alarms worth having

​Good practice

Logs

Write useful logs

Metrics

Alarms

How an alarm decides

Mute, check, history

From code

Alarms worth having

Good practice