Resource Isolation

Cgroups Integration 💎

Super integrates directly with the Linux Kernel’s Control Groups (v2) to provide hardware-level isolation for managed processes.

Configuration Reference

Resource limits are defined per program — via stack files (TOML/JSON), the API, or CLI (super add --memory 512 --cpu 1.5). Example: services[] entry in a stack file (conf/conf.d/*.json / *.toml):

{
  "services": [
    {
      "name": "data-processor",
      "command": "./worker",
      "resource_limits": {
        "memory_limit": 1024,
        "cpu_quota": 2.0
      }
    }
  ]
}

• memory_limit — hard memory limit in MB (binary, 1 MB = 1024² bytes). If the process (and its children) exceed this, the OOM Killer terminates it.
• memory_warn_percent — optional; warn (emit a memory_pressure event) when live memory reaches this % of memory_limit. Default 80; 0 disables pre-kill warning (see Warning & visibility).
• memory_warn_headroom — optional; also warn when live memory comes within this many MB of memory_limit (absolute headroom; whichever threshold triggers first). 0 (default) disables.
• memory_high — optional; kernel soft limit in MB. When exceeded the kernel throttles the cgroup instead of killing it (see Tier 2 — memory.high (opt-in)). 0 (default) disables.
• cpu_quota — CPU quota in cores (1.0 = one full core, 0.5 = half a core, 2.0 = two cores; fractions allowed). The scheduler throttles the process if it exceeds this usage.

Warning & visibility (three-tier)

Super combines pre-kill warning, an opt-in kernel throttle, and post-kill OOM confirmation around the hard cap. All three surface as SystemEvents — recorded in the event history (super events <name>), forwarded by OSS event hooks, and alerted by licensed event notifications.

Tier 1 — pre-kill warning (default on)

A background watcher polls each limited cgroup’s anonymous memory (memory.stat → anon, page cache is excluded because the kernel can reclaim it and it never causes OOM) every 10 s.

• When anon ≥ threshold (memory_warn_percent% of the limit, or within memory_warn_headroom of it), Super emits memory_pressure once per 5-minute cooldown per program. The process keeps running.
• A program only re-arms the warning after it drops back below threshold − 5 pp (hysteresis), preventing flapping at the boundary.
• memory_warn_percent = 0 and memory_warn_headroom = 0 disable this tier.

What a warning buys you: with a leak rate of ~100 MB/s, the default 80% threshold on a 1 GiB cap leaves ~200 MB / ~2 s of reaction time. It is best-effort — a fast OOM between polls still kills; the kill is then confirmed by Tier 3.

Tier 2 — memory.high (opt-in)

Setting memory_high writes the cgroup v2 soft limit (memory.high). When the cgroup exceeds it the kernel throttles reclaim (slows allocation) instead of OOM-killing. Useful to hold a process under its hard cap long enough for Tier 1 to fire reliably.

• This is off by default because throttling can slow memory-healthy processes that briefly spike (GC pre-collection heap growth, cache warmup). Only enable it if you accept kernel-level throttling as a trade for earlier, kernel-accurate pressure signals.
• The memory_pressure event fires on the same thresholds when the cgroup is under throttle.

Tier 3 — OOM confirmation (always on)

Super watches each limited cgroup’s memory.events → oom_kill counter. When it increments, Super emits memory_oom_kill with a usage snapshot (memory.current, memory.max, anon). This makes an OOM kill distinguishable from a manual kill -9: instead of a generic signal 9 exit, you get an explicit “memory cap exceeded” event with the exact usage at kill time.

Recommended setup

One program entry in a stack file (stack.json services[] or stack.toml [[services]]):

{
  "name": "worker",
  "command": "/usr/local/bin/worker",
  "resource_limits": {
    "memory_limit": 1024,
    "memory_warn_percent": 80,
    "memory_warn_headroom": 0,
    "memory_high": 0
  }}

The defaults are conservative: Tier 1 warning on, Tier 2 off, Tier 3 always on. Raise/lower memory_warn_percent to trade earlier warning against false positives from normal fluctuation; add memory_high only when you want kernel-enforced throttling before the hard cap.

Requirements

• OS: Linux only.
• Kernel: Cgroups v2 enabled (Standard on Ubuntu 22.04+, Debian 11+, Fedora).
• Privileges: The superd daemon usually requires root privileges to create and manage cgroups (writing to /sys/fs/cgroup).

Monitoring Limits

You can check if Cgroups are being enforced via the metrics endpoint:

curl -H "Authorization: Bearer $SUPER_TOKEN" http://localhost:9002/metrics | grep cgroup
# super_cgroup_enforced_total 5

Superd logs include [isolation] Applying limits for '…' (PID: …) on start. When limits change on a running process, the cgroup files (cpu.max, memory.max) are updated in place (see Hot-update limits).

Verify & adjust

Pre-flight

• Run superd with the isolation plugin loaded on Linux, with permission to write under /sys/fs/cgroup (typically root on bare metal).
• Containers: cgroup mounts are often read-only unless you provide a writable cgroup namespace or run privileged. Limits will not apply if Super logs a read-only cgroup warning at startup.
• Confirm cgroup v2: mount | grep cgroup2 (or stat -fc %T /sys/fs/cgroup/ shows cgroup2fs).

Confirm a program is in a cgroup

Each managed program gets a directory named by its UUID:

# Replace <id> with the program UUID from `super list` or the API
ls /sys/fs/cgroup/super/<id>/
cat /sys/fs/cgroup/super/<id>/cpu.max      # e.g. 50000 100000 ≈ 50% of one core
cat /sys/fs/cgroup/super/<id>/memory.max   # hard memory cap in bytes

When the process stops, Super removes the cgroup directory (after_stop cleanup).

Hot-update limits (no restart for CPU quota)

Change limits on a running program without restarting it (CPU quota is updated in place):

# CLI (requires isolation plugin on Linux)
super update data-processor --cpu 1.5
super update data-processor --memory 512

# API
curl -X PUT http://127.0.0.1:9002/api/v1/programs/<id> \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"resource_limits": {"cpu_quota": 1.5, "memory_limit": 512}}'

Re-check cpu.max / memory.max under /sys/fs/cgroup/super/<id>/ and watch process CPU with top -p <pid>.

Troubleshooting