Event Notifications

Webhook System 💎

Super acts as an intelligent observer. Instead of just logging errors, it can actively push events to external systems like Slack, Microsoft Teams, or your company’s internal IM tools.

Configuration

Notifications are configured in a separate file conf/notify.toml (sibling to super.toml). This separation allows you to hot-reload alerting rules without restarting your processes.

File location

Notifications are configured in a separate file conf/notify.toml (sibling to super.toml). This separation allows you to hot-reload alerting rules without restarting your processes.

$SUPER_ROOT/
├── conf/
│   ├── super.toml       # Main daemon config
│   └── notify.toml      # Webhooks + inhibition rules (this file)
├── run/                 # Optional pidfile when using superd --daemon
├── plugins/
│   └── notify.so        # Required plugin

If notify.toml does not exist, the plugin starts with zero channels and no notifications are sent.

$SUPER_ROOT is the instance root resolved from the SUPER_ROOT environment variable (falling back to the binary layout / working directory).

Configuration Reference

Channel schema

Each channel is defined as a [[channels]] block in TOML (conf/notify.toml):

# notify.toml
[[channels]]
id = "unique-channel-id"          # Required: unique identifier
name = "Human-readable name"      # Required: display name
type = "webhook"                  # Required: channel type (see presets below)
triggers = ["process_fatal", "*"] # Required: event types to listen for
include_log_tail = true           # Optional: attach recent logs to notifications (default: false)

[channels.config]                 # Required: webhook-specific settings
url = "https://example.com/webhook"
secret = "hmac-signing-secret"    # Optional: HMAC-SHA256 signature key
headers = { Authorization = "Bearer token" }  # Optional: custom HTTP headers

[channels.config]

Built-in Presets

You do not need to write complex JSON templates for popular platforms. Super includes built-in rich-text templates. Just set the type field to one of the supported presets:

Slack

# notify.toml
[[channels]]
id = "slack-ops"
name = "Ops Team Slack"
type = "slack"
triggers = ["process_fatal", "process_backoff"]
include_log_tail = true

[channels.config]
url = "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"

Renders a Slack Block Kit message with a header, markdown body, and hostname context footer.

DingTalk

# notify.toml
[[channels]]
id = "dingtalk-alert"
name = "DingTalk Ops Group"
type = "dingtalk"
triggers = ["*"]
include_log_tail = true

[channels.config]
url = "https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN"
secret = "SEC..."

Renders a markdown message with title Superd Alert: {program_name}. When secret is set, requests carry the standard X-Super-Signature header (see Payload Signing).

Lark / Feishu

# notify.toml
[[channels]]
id = "lark-alert"
name = "Backend On-Call"
type = "lark"
triggers = ["process_fatal", "system_startup", "system_shutdown"]

[channels.config]
url = "https://open.feishu.cn/open-apis/bot/v2/hook/YOUR_HOOK_ID"

Renders an interactive card with red header, markdown body, and hostname/version footer.

WeCom / WeChat

# notify.toml
[[channels]]
id = "wecom-ops"
name = "WeCom Ops Channel"
type = "wecom"
triggers = ["*"]

[channels.config]
url = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_KEY"

Renders a markdown message via the WeCom msgtype: markdown format.

Microsoft Teams

# notify.toml
[[channels]]
id = "teams-alerts"
name = "Teams Ops Channel"
type = "teams"
triggers = ["process_fatal"]
include_log_tail = true

[channels.config]
url = "https://outlook.office.com/webhook/YOUR_WEBHOOK_URL"

Renders a MessageCard with summary, hostname subtitle, and markdown body.

Generic Webhook with HMAC

# notify.toml
[[channels]]
id = "internal-monitoring"
name = "Internal Monitoring Hub"
type = "webhook"
triggers = ["*"]

[channels.config]
url = "https://api.my-company.com/v1/alerts"
secret = "my-super-secret-key-888"
headers = { Authorization = "Bearer sk-123456", X-Custom-Auth = "super-admin" }

Sends the raw Super JSON envelope (see Default Envelope below).

Supported Events

Use these strings in the triggers field. Full payload reference: System Events.

Default Envelope (webhook type)

When type = "webhook", Super sends this structured JSON payload:

{
  "id": "uuid-of-notification",
  "timestamp": "2026-07-22T10:00:00Z",
  "event": "process_fatal",
  "system": {
    "hostname": "prod-server-1",
    "version": "1.4.0"
  },
  "summary": "[Fatal] worker on prod-server-1: killed by SIGKILL (9)",
  "markdown": "### Process Fatal Alert\n- Service: worker\n- Program ID: `550e8400-e29b-41d4-a716-446655440000`\n- Host: prod-server-1\n- Signal: SIGKILL (9)\n- Cause: Killed by SIGKILL (may be a kernel/cgroup OOM kill; check resource_limits and system logs)\n- Exit Code: None\n- Reason: Stopped after 3 retries.\n\n---\nSent from Super · host `prod-server-1`\n",
  "data": {
    "type": "ProcessFatal",
    "payload": {
      "program_id": "550e8400-e29b-41d4-a716-446655440000",
      "program_name": "worker",
      "pid": 12345,
      "uptime_secs": 502,
      "exit_code": null,
      "signal": 9,
      "msg": "Stopped after 3 retries.",
      "log_tail": "Error: Connection refused..."
    }
  },
  "log_tail": "Error: Connection refused..."
}

The data field is the tagged System Event envelope — event fields live under data.payload.*.

Payload Signing

When secret is configured, Super signs the outgoing request body with a single, uniform scheme for all channel types (webhook, Slack, DingTalk, Lark, WeCom, Teams):

• Algorithm: HMAC-SHA256(secret, request_body)
• Delivered as a request header: X-Super-Signature: sha256=<hex-digest>

Platform-native signing conventions (DingTalk timestamp+sign, Lark sign field, …) are not generated automatically — secret produces the X-Super-Signature header only, and platform-specific verification is up to your receiver.

import hmac, hashlib

def verify(secret: str, body: bytes, header: str) -> bool:
    expected = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    received = header.replace("sha256=", "")
    return hmac.compare_digest(expected, received)

Platforms without verification

Slack, Teams, and WeCom native incoming webhooks cannot be configured to validate a custom header — for those, the URL itself is the credential. Setting secret still adds the X-Super-Signature header, but the platform will ignore it; use a custom type = "webhook" receiver when you need signature verification.

Custom Templates

For advanced use cases, override the default payload with a Handlebars template (conf/notify.toml):

# notify.toml
[[channels]]
id = "custom-integration"
name = "Custom Integration"
type = "webhook"
triggers = ["process_fatal"]

template = """
{
  "service": "{{ payload.program_name }}",
  "severity": "critical",
  "message": {{ json_quote rendered_summary }},
  "host": "{{ system_hostname }}"
}
"""

[channels.config]
url = "https://api.example.com/alerts"

Template variables

Event-specific payload fields:

json_quote helper

When embedding rendered text inside JSON strings, use {{ json_quote ... }} to escape special characters (newlines, quotes, backslashes) so the output remains valid JSON:

{
  "text": {{ json_quote rendered_markdown }}
}

Without json_quote, raw markdown would break JSON syntax.

Template priority

When multiple template sources exist, Super resolves in this order:

1. template field (explicit custom template)
2. type preset (e.g., type = "slack")
3. Default Super envelope

Hot Reload

Configuration changes take effect without restarting the daemon:

super reload

The plugin re-reads conf/notify.toml and applies the new channel list immediately.

Management API

Read configuration

curl -H "Authorization: Bearer $TOKEN" \
  http://127.0.0.1:9002/api/v1/system/notify

Sensitive fields (secret, token, key, authorization, etc.) are automatically masked as ******** for Viewer.

Update configuration

curl -X PUT http://127.0.0.1:9002/api/v1/system/notify \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "channels": [
      {
        "id": "slack-ops",
        "name": "Slack",
        "type": "slack",
        "triggers": ["*"],
        "config": { "url": "https://hooks.slack.com/..." }
      }
    ]
  }'

Test a channel

curl -X POST http://127.0.0.1:9002/api/v1/system/notify/test \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "test",
    "name": "Test",
    "type": "webhook",
    "triggers": ["*"],
    "config": { "url": "https://webhook.site/your-id" }
  }'

Sends a system_startup event with hostname = "TEST-MODE".

Metrics

Prometheus metrics are exposed at /metrics (Bearer required when API auth is on):

# HELP super_notify_sent_total Total number of notifications sent.
# TYPE super_notify_sent_total counter
super_notify_sent_total{status="success"} 142
super_notify_sent_total{status="failed"} 3

Complete Example

A production-ready conf/notify.toml with multiple channels:

# conf/notify.toml
# Alert on crashes to Slack
[[channels]]
id = "slack-critical"
name = "Slack Critical Alerts"
type = "slack"
triggers = ["process_fatal", "system_shutdown"]
include_log_tail = true

[channels.config]
url = "https://hooks.slack.com/services/T00/B00/CRITICAL"


# All events to DingTalk
[[channels]]
id = "dingtalk-all"
name = "DingTalk All Alerts"
type = "dingtalk"
triggers = ["*"]

[channels.config]
url = "https://oapi.dingtalk.com/robot/send?access_token=TOKEN"
secret = "SEC..."


# Internal monitoring with HMAC + custom headers
[[channels]]
id = "internal-hub"
name = "Internal Monitoring Hub"
type = "webhook"
triggers = ["process_fatal", "process_backoff", "process_recovered"]

[channels.config]
url = "https://monitoring.internal/api/alerts"
secret = "hmac-secret-key"
headers = { "X-Source" = "superd", Authorization = "Bearer internal-token" }

Storm Suppression 💎

When a process crashes and restarts rapidly, each process_fatal and process_backoff event triggers a notification. Without safeguards, a flapping service can flood your webhooks — and if the remote platform rate-limits you (HTTP 429), important alerts may be lost.

Super’s storm suppression system prevents notification floods with two complementary mechanisms:

In the licensed Dashboard, these map to:

Delivery Strategy

Each webhook channel defines a strategy that governs notification pacing for that destination only.

    flowchart TB
  EV["Incoming events<br/>(matched triggers)"] --> MODE{{"strategy.mode"}}

  MODE -->|"immediate"| IMM["Send every event"]
  MODE -->|"cooldown"| CD["Send first of each event type<br/>Skip same-type repeats for cooldown_secs"]
  MODE -->|"batch"| BAT["Buffer for window_secs<br/>Flush one summary (early at max_events)"]

  IMM --> WH["Webhook destination"]
  CD --> WH
  BAT --> WH
  

# notify.toml
[[channels]]
id = "slack-ops"
name = "Ops Slack"
type = "slack"
triggers = ["*"]

[channels.strategy]
mode = "cooldown"      # immediate | cooldown | batch
cooldown_secs = 60     # only for mode = "cooldown"
window_secs = 30        # only for mode = "batch"
max_events = 10         # only for mode = "batch"

Strategy Modes

The dashboard pre-fills these recommended values when you switch modes so fields are never left empty.

Cooldown in Detail

When mode = "cooldown":

• The first event of a given type (process_fatal, process_backoff, etc.) fires immediately.
• Subsequent events of the same type are suppressed until cooldown_secs seconds have elapsed.
• Different event types do not share a cooldown — a process_fatal does not block a system_startup.

# notify.toml — [channels.strategy]
[channels.strategy]
mode = "cooldown"
cooldown_secs = 60      # At most 1 notification per event type per minute

Batch in Detail

When mode = "batch":

• Events are accumulated in an in-memory buffer during the window_secs window.
• A global ticker (1-second resolution) checks all batching channels and flushes windows that have elapsed.
• If the queue reaches max_events first, the batch flushes immediately instead of waiting for the window.
• The flush sends a single summary webhook: total event/host counts in summary, plus a markdown table with one row per event.

# notify.toml — [channels.strategy]
[channels.strategy]
mode = "batch"
window_secs = 30        # Collect events for 30 seconds
max_events = 10          # Flush early when 10 events are queued

Batch payload example:

{
  "id": "uuid-of-batch-notification",
  "timestamp": "2026-07-24T10:00:30Z",
  "event": "batch_summary",
  "system": {
    "hostname": "prod-server-1",
    "version": "1.4.0"
  },
  "summary": "[Super] 5 events on 1 host(s)",
  "markdown": "### [Super] 5 events in the last window\n\n| Program | Program ID | Host | Event | Detail |\n|---------|------------|------|-------|--------|\n| worker-1 | `550e8400-e29b-41d4-a716-446655440000` | prod-server-1 | 💥 Fatal | exit code 137 – Stopped after 3 retries. |\n| worker-2 | `660e8400-e29b-41d4-a716-446655440001` | prod-server-1 | ⚠️ Backoff | retry #2 exit_code=Some(1) |\n\n2 program(s) affected on 1 host(s)\n\n---\nSent from Super · host `prod-server-1`\n",
  "batch": true
}

summary aggregates total events and affected host count; markdown is a rendered table with one row per queued event (including Program ID).

Inhibition

While Delivery Strategy controls per-webhook pacing, Inhibition prevents redundant notifications across all webhooks based on event semantics (similar to Alertmanager inhibit_rules).

Classic example: If process_fatal fires for worker, there is little value in also sending process_backoff for worker — the fatal alert already covers the failure. Inhibition expresses that relationship.

Mental model (dashboard)

Configure each rule in order:

1. When — events that start (or refresh) the mute window → config field sources
2. Mute targets — events to silence while the window is active → config field targets
3. For — how long the window lasts → config field ttl_secs

Matching is limited to the same program (match_on = ["program_name"] in the dashboard).

    flowchart LR
  subgraph scope["Same program_name · all webhooks"]
    direction LR
    W["1. When<br/>sources<br/>e.g. process_fatal"] --> M["2. Mute targets<br/>targets<br/>e.g. process_backoff"]
    M --> F["3. For<br/>ttl_secs<br/>e.g. 300s"]
  end

  style scope stroke-dasharray: 5 5
  

Schema

# notify.toml
[[inhibition_rules]]
id = "fatal-suppresses-backoff"   # Unique identifier
sources = ["process_fatal"]       # When: events that trigger inhibition
targets = ["process_backoff"]     # Mute targets: suppressed while a source is active
match_on = ["program_name"]       # Fields that must match between source and target
ttl_secs = 300                    # For: how long inhibition lasts (seconds)

How it Works

1. A When event fires (e.g., process_fatal for program worker).
2. Super creates an active inhibitor scoped to match_on values — here program_name = "worker".
3. For ttl_secs (e.g., 300 seconds), any Mute target event (process_backoff) for the same worker is suppressed globally across all webhooks.
4. After the TTL expires, the inhibitor is cleared. If a When event fires again, the inhibitor is refreshed.

Overlap is allowed: an event type may appear in both sources and targets (for example When = Fatal or Restarting, Mute targets = Restarting). Evaluation order is: check suppression → send if allowed → then record the event as a source. So the first matching event still notifies; further repeats of the mute target are silenced for the duration.

Example: Fatal → Restarting

# notify.toml
[[inhibition_rules]]
id = "fatal-scenarios"
sources = ["process_fatal"]              # When
targets = ["process_backoff"]            # Mute targets
match_on = ["program_name"]
ttl_secs = 300                           # For: 5 minutes

Scenario:

Audit Logging

Skipped deliveries (cooldown / inhibition) and HTTP outcomes are written to:

• $SUPER_ROOT/logs/notify.log — append-only ops tail (OK / FAIL / SKIP)
• $SUPER_ROOT/data/notify_delivery.db — queryable history (see Delivery history)

Inhibition skips store detail=rule_id=… on the delivery row (and a SKIP | outcome=inhibited line in the log).

Choosing the Right Combination

Configuration in notify.toml

All storm suppression settings live in the same conf/notify.toml:

# conf/notify.toml
# --- Webhooks with delivery strategies ---

[[channels]]
id = "slack-critical"
name = "Slack Critical"
type = "slack"
triggers = ["process_fatal", "system_shutdown"]
include_log_tail = true

[channels.config]
url = "https://hooks.slack.com/services/T00/B00/CRITICAL"

[channels.strategy]
mode = "cooldown"
cooldown_secs = 60


[[channels]]
id = "dingtalk-batch"
name = "DingTalk Summary"
type = "dingtalk"
triggers = ["*"]

[channels.config]
url = "https://oapi.dingtalk.com/robot/send?access_token=TOKEN"

[channels.strategy]
mode = "batch"
window_secs = 30
max_events = 10


# --- Global inhibition (When → Mute targets → For) ---

[[inhibition_rules]]
id = "fatal-blocks-backoff"
sources = ["process_fatal"]
targets = ["process_backoff"]
match_on = ["program_name"]
ttl_secs = 300

Hot Reload

Like webhook configuration, inhibition rules take effect on super reload — no daemon restart needed.

super reload

Delivery history

Successful sends, failures, cooldown skips, and inhibition skips are stored in SQLite under the instance root (default $SUPER_ROOT/data/notify_delivery.db). This is separate from process event history (events.db).

Optional top-level keys in conf/notify.toml:

delivery_keep_days = 14                 # 0 = never prune; default 14
delivery_db = "data/notify_delivery.db" # relative to SUPER_ROOT

Browse and filter in the dashboard: Notification Settings → Delivery (/settings/notify/delivery). Outcome chips (OK / Fail / Cooldown / Inhibited) toggle filters; table columns are sortable. Or query the API:

curl -H "Authorization: Bearer $TOKEN" \
  'http://127.0.0.1:9002/api/v1/system/notify/deliveries?outcome=fail&sort_by=time&order=desc&limit=20'

Query params: outcome, channel_id, program_id, event_type, q, limit, offset, sort_by (time | outcome | channel | event | program | detail), order (asc | desc).

Alert body identity

Program-scoped webhook markdown always includes Program ID (UUID) next to the service name, and ends with a source footer Sent from Super · host \…`. Override the host label with [SUPER_HOSTNAME`](/docs/06-internals/environment-variables#super_hostname) when the OS hostname is not meaningful (containers).

Troubleshooting