Automations fail: a partner API is down, a provider rejects a key, a bug slips through. Pawabase is built so that failures are visible and recoverable, but whether recovering is safe depends on how you design your flows. This page brings the rules together.

​
The guarantees

“At least once” means a reaction can, rarely, run twice: for example, if a worker restarts part-way through. “Once per event per endpoint” means Pawabase never deliberately sends the same event to the same webhook twice; retries after a failure are the same delivery, with the same event id.

​
Design rule 1: make reactions safe to repeat

A reaction is idempotent when running it twice has the same effect as running it once. Ways to get there:

​
Design rule 2: retry only what’s safe and transient

Flows get one attempt because a blind retry could charge a card or email someone twice. Add a Retry block around the specific step that can fail for a passing reason and is safe to repeat, usually an HTTP request to another service. It runs its attempt branch up to 10 times with a doubling wait (starting at Base delay, at most 30 seconds between tries). Don’t wrap a whole flow in Retry. Don’t retry a step that already succeeded at the other end: if a partner’s API timed out, it may still have done the work, so use its idempotency key.

​
Design rule 3: answer fast, work in the background

  • A route or flow your app calls should do the minimum, then Queue flow the slow part and return.
  • A webhook receiver of yours should store the event, answer 2xx within 15 seconds, and process afterwards.
  • A flow has a timeout (60 seconds by default, up to 900). Long loops over many items should queue one job per item instead.

​
Design rule 4: don’t depend on order

Reactions run in the background and may finish in any order. If orders.updated is handled before orders.created, your flow must still do the right thing. Load the current record by id rather than trusting the event’s copy, and compare states rather than assuming a sequence.

​
Design rule 5: keep payloads small

Events and job inputs should carry ids and a little context. A job that runs later should read fresh data, and small payloads are cheaper to store and retry.

​
Recovering from an outage

When a dependency was down (your mail provider, a partner API, your own webhook receiver):
  1. Find the scope. In Activity, filter failures by time. Read two or three errors to confirm they share a cause.
  2. Fix the cause. Replace the expired key, wait for the partner to recover, deploy the fix.
  3. Preview the recovery. Every bulk action has a dry run:
  4. Run it, then watch queues drain and spot-check results.
  5. Write down what happened, and add an alarm so you hear about it sooner next time.
Because replay re-runs work with its original input, rules 1 and 4 are what make recovery safe.

​
Know before your users do

See Logs, metrics and alarms.

​
A checklist for every automation

  • It’s safe to run twice (it checks, uses unique keys, or records the event id).
  • Steps that call other services are wrapped in Retry, and use idempotency keys where offered.
  • Every exit is connected, including missing, false and failed.
  • It reads ids from the event and loads current data.
  • It finishes well inside its timeout, or queues the slow part.
  • Failures alert someone.
  • You’ve tried recovering it: fail it on purpose in development, then replay.