Signal9Signal9 — home
online
all lessons

Sandbox a systemd Service: Confinement Directives and systemd-analyze security

Intermediate35 min7 steps
What you'll learn
  • Explain what service sandboxing protects against: what an attacker who owns a service can reach
  • Build filesystem walls with ProtectSystem= (true, full, strict), ProtectHome=, PrivateTmp= and the Read*/Inaccessible*/StateDirectory= paths
  • Remove privilege with NoNewPrivileges=, an empty CapabilityBoundingSet= and User=/DynamicUser=, and say what each one stops
  • Narrow the kernel surface with ProtectKernel*=, RestrictAddressFamilies= and SystemCallFilter=@system-service
  • Measure a unit plus its drop-ins with systemd-analyze security --offline=true, read the table, and lower the overall exposure
  • Recognise a sandbox that breaks the service (read-only state, hidden /home, private /tmp, missing capability) and loosen the one directive at fault
  1. The third question about a service

    Frame confinement as the layer between "the service runs" and "the service owns the box".

    You already know how to ask two questions about a service: is it running? (systemctl status) and how much may it use? (CPUQuota=, MemoryMax=). This lesson is about the third: if this service is compromised, what can the attacker reach?

    By default the answer is "everything the service's user can reach", and a lot of services run as root. A bug in a report worker that only ever needed to read one directory and write another hands over every file on the box, every device, every kernel interface, the network, and the chance to escalate further through setuid binaries.

    systemd can shrink that, per service, with directives in the unit's [Service] section. Behind the scenes it starts the service in its own mount namespace (so it can make parts of the filesystem read-only or invisible), drops capabilities, sets the kernel's no_new_privs flag, and installs a seccomp system-call filter. You write one line per wall; systemd builds the wall at start-up. You add them the way the overrides lesson taught: in a drop-in, so the vendor's unit file stays untouched.

    systemd-analyze --version | head -n 1
    systemd 252 (252-67.el9_8.6.rocky.0.1)
    --version
    print the systemd release and its compile-time features
    head -n 1
    keep only the first line
    The systemd release decides which directives and which systemd-analyze options exist. 252 is new enough for everything in this lesson. The text in brackets is the distribution's build and will differ from box to box.
    systemd-analyze security sshd.service 2>&1
    System has not been booted with systemd as init system (PID 1). Can't operate.
    Failed to connect to bus: Host is down
    The online form asks the running service manager over D-Bus, and there isn't one here. That's the container, not a mistake. On a normal server this prints the table you're about to see. 2>&1 just shows the error in the same stream.
    printf '%s\n' '[Unit]' 'Description=Demo nightly report worker' '' '[Service]' 'ExecStart=/usr/bin/sleep infinity' > /etc/systemd/system/demo-report.service
    A minimal unit to experiment on: no User=, so it would run as root, and no hardening at all. That's what most hand-written units look like. printf '%s\n' writes each quoted argument as one line, which is an easy way to write a small file without an editor.
    systemd-analyze security --offline=true demo-report.service | tail -n 1
    → Overall exposure level for demo-report.service: 9.6 UNSAFE 😨
    --offline=true
    analyse the unit file and its drop-ins from disk; no running systemd needed
    The verdict: an exposure level from 0.0 (tightly confined) to 10.0 (nothing confined). 9.6 is roughly where every unhardened root service lands. The arrow and face depend on your terminal: a UTF-8 terminal shows → and an emoji, a plain one shows -> and :-{. The number is the part to read.
    systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'
    Overall exposure level for demo-report.service: 9.6
    --offline=true
    analyse the files on disk, no running systemd needed
    grep -o 'Overall.*: [0-9.]*'
    print only the matching part: the label and the number, without the glyphs
    The same line with the glyphs and the face cut off: grep -o prints only the part that matches, up to the last digit. You'll use this form to watch the number move as you add walls.

    Check yourself

    A service scores 9.6 in systemd-analyze security. Does that mean the service's code has a vulnerability?

  2. Filesystem walls

    Make the OS read-only for the service, hide what it never needs, and hand back exactly the paths it writes.

    The filesystem walls decide what the service can see and change. All of them work by giving the service its own mount namespace, so the rest of the system is unaffected.

    • `ProtectSystem=` has three strengths, and learners routinely pick the weakest and assume it covered /etc:

    - true: /usr, /boot and /efi are read-only. /etc is still writable.

    - full: the same plus /etc.

    - strict: the entire filesystem is read-only, except the API filesystems /dev, /proc and /sys, and /home, /root and /run/user, which are left to ProtectHome=. This includes /var and /tmp, so the service's own state too.

    • `ProtectHome=`: yes makes /home, /root and /run/user empty and inaccessible; read-only leaves them readable; tmpfs mounts an empty, read-only tmpfs over them, onto which you can bind just the paths the service needs.
    • `PrivateTmp=yes` gives the service its own empty /tmp and /var/tmp, invisible to everyone else. That stops /tmp races and snooping on other services' temp files.
    • Paths: ReadWritePaths=, ReadOnlyPaths= and InaccessiblePaths= adjust single paths. A deeper path overrides a shallower one, so ProtectSystem=strict plus ReadWritePaths=/var/lib/demo is "everything read-only except that directory". The one exception: below an inaccessible path, nothing comes back.
    • `StateDirectory=NAME` is the tidy way to hand back state: systemd creates /var/lib/NAME, makes it writable inside the sandbox, and owns it to the service's user.
    mkdir -p /etc/systemd/system/demo-report.service.d
    -p
    create parent directories as needed; no error if it already exists
    The drop-in directory from the overrides lesson. Every *.conf file in it is merged over the unit, in file-name order, so you can split the hardening into themed files.
    printf '%s\n' '[Service]' 'ProtectSystem=strict' 'StateDirectory=demo-report' 'ProtectHome=yes' 'PrivateTmp=yes' > /etc/systemd/system/demo-report.service.d/10-filesystem.conf
    The filesystem walls in their own drop-in: everything read-only, /var/lib/demo-report handed back for state, home directories hidden, and a private /tmp. Nothing prints on success.
    systemd-analyze security --offline=true demo-report.service | grep -E 'ProtectSystem=|ProtectHome=|PrivateTmp='
    ✓ ProtectSystem=                                              Service has strict read-only access to the OS file hierarchy
    ✓ PrivateTmp=                                                 Service has no access to other software's temporary files
    ✓ ProtectHome=                                                Service has no access to home directories
    --offline=true
    analyse the files on disk, no running systemd needed
    grep -E
    extended regex, so | separates alternatives
    The analyser read the unit and the drop-in: all three rows are now ticked. A tick shows as ✓ in a UTF-8 terminal and + in a plain one (a cross is ✗ or -). The description column tells you which strength it found, so ProtectSystem=full would read "very limited write access" instead of "strict".
    systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'
    Overall exposure level for demo-report.service: 9.0
    9.6 down to 9.0. The filesystem walls matter enormously to a real attacker, but the score weights them lightly. The big numbers come in the next two steps. Don't read a small drop as "not worth it".

    Check yourself

    Which ProtectSystem= value makes /etc read-only but leaves /var writable?

  3. Privilege walls

    Stop the service from gaining privilege, and take away the root powers it never uses.

    `NoNewPrivileges=yes` is the cheapest line in the file. It sets the kernel's no_new_privs flag on the service, and the flag is inherited by every child and survives every execve(). From then on, running a setuid or setgid binary, or one with file capabilities, gives the process nothing extra. The hardening lesson's SUID audit finds the binaries that could be abused; this directive makes all of them useless to the confined service, including ones you haven't found yet. Some directives imply it, for example DynamicUser=yes.

    `CapabilityBoundingSet=` limits which of root's capabilities (the pieces root's power is split into: CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_DAC_OVERRIDE, and about forty more) the service can ever hold. List the ones to keep, or prefix with ~ to list the ones to drop. An empty value, CapabilityBoundingSet=, keeps none: root in name only. It can still write files it owns, but it can't override permissions, load modules, change the network, or bind ports below 1024.

    `User=` runs the service as an ordinary account, and `DynamicUser=yes` invents a throwaway user for each run. DynamicUser= also switches on ProtectSystem=strict, ProtectHome=read-only, PrivateTmp=yes and NoNewPrivileges=yes, which is powerful and easy to forget.

    grep NoNewPrivs /proc/self/status
    NoNewPrivs:	0
    The kernel flag itself. /proc/self/status describes the process reading it, here grep. 0 means a setuid binary run from this shell would still gain its owner's rights.
    setpriv --no-new-privs grep NoNewPrivs /proc/self/status
    NoNewPrivs:	1
    --no-new-privs
    set the no_new_privs flag, then run the command
    setpriv sets the same flag systemd sets for NoNewPrivileges=yes, then runs grep, which reports 1. Nothing can clear it again for that process or its children. That's exactly what the service gets.
    printf '%s\n' '[Service]' 'NoNewPrivileges=yes' 'CapabilityBoundingSet=' > /etc/systemd/system/demo-report.service.d/20-privileges.conf
    A second drop-in for the privilege walls. The empty CapabilityBoundingSet= is deliberate: an empty value means "no capabilities", not "unchanged".
    systemd-analyze security --offline=true demo-report.service | grep -E 'NoNewPrivileges=|~CAP_SYS_ADMIN|~CAP_NET_\('
    ✓ NoNewPrivileges=                                            Service processes cannot acquire new privileges
    ✓ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has no elevated networking privileges
    ✓ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has no administrator privileges
    The analyser has one row per capability group, read as "the bounding set excludes (~) these". With an empty set every group is ticked. Note the networking group: CAP_NET_BIND_SERVICE is gone, so this service could no longer listen on port 80.
    systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'
    Overall exposure level for demo-report.service: 6.4
    9.0 to 6.4 from two lines. The capability rows carry heavy weights because each one is a separate road from "compromised service" to "compromised kernel or host".

    Check yourself

    A confined service has NoNewPrivileges=yes and finds a setuid-root copy of bash on the box. What does running it give the service?

  4. Kernel and system-call walls

    Take the kernel's control knobs, unneeded socket types and dangerous system calls away from the service.

    The last group narrows what the service can ask the kernel to do. You don't need to understand seccomp to use these; you need to know what each one closes.

    • `ProtectKernelTunables=yes` makes /proc/sys and similar knobs read-only, so the service can't change sysctls.
    • `ProtectKernelModules=yes` denies loading or reading kernel modules. `ProtectControlGroups=yes` makes the cgroup tree read-only.
    • `RestrictAddressFamilies=` lists the socket types the service may open. A worker that only talks to local daemons needs AF_UNIX; a network service adds AF_INET AF_INET6. Raw packet and netlink sockets, favourites for attackers, are then refused.
    • `SystemCallFilter=@system-service` installs a seccomp allow-list of the system calls ordinary services use. Mounting, rebooting, loading modules, swapping, changing the clock and other privileged calls are refused. @system-service is a named group; systemd-analyze syscall-filter shows what is in it.
    • `SystemCallArchitectures=native` blocks the old 32-bit call path on a 64-bit box, a classic way around filters.

    Keep this step shallow on purpose: the named groups are maintained by systemd for exactly this use, so @system-service is the right default and hand-written call lists are rarely worth it.

    systemd-analyze syscall-filter @system-service | head -n 6
    @system-service
        # General system service operations
        @aio
        @basic-io
        @chown
        @default
    syscall-filter @system-service
    list the system calls in that named group
    head -n 6
    show only the first six lines
    syscall-filter works offline: the groups are compiled into systemd. @system-service is built from smaller groups (asynchronous I/O, basic I/O, ownership changes, the defaults every program needs) plus individual calls. Drop the | head to see all of it.
    printf '%s\n' '[Service]' 'ProtectKernelTunables=yes' 'ProtectKernelModules=yes' 'ProtectControlGroups=yes' 'RestrictAddressFamilies=AF_UNIX' 'SystemCallFilter=@system-service' 'SystemCallArchitectures=native' > /etc/systemd/system/demo-report.service.d/30-kernel.conf
    The kernel walls in a third drop-in. The demo worker only needs local sockets, so AF_UNIX is the whole allow-list.
    systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'
    Overall exposure level for demo-report.service: 3.6
    6.4 to 3.6. The address families and the system-call filter are scored as a dozen separate rows, so getting them right moves the number a lot.

    Check yourself

    A service only talks to a local database over a Unix socket. What should RestrictAddressFamilies= list?

  5. Measure it

    Read the security table, use it to choose the next wall, and know what the number does and doesn't mean.

    systemd-analyze security turns hardening from a checklist into a measurement. Each row is one setting: a tick (✓) when the unit has it, a cross (✗) when it doesn't, with the exposure that row adds in the last column. Rows with no mark don't apply (for example SupplementaryGroups= for a root service). The overall level combines the rows; it is systemd's estimate, from 0.0 to 10.0, of how exposed the system is if this service is taken over.

    The demo unit now has three drop-ins, 10-filesystem.conf, 20-privileges.conf and 30-kernel.conf, merged in that order. Splitting by theme makes it obvious later which wall broke something.

    The workflow is a loop: measure, pick the expensive crosses, add a wall, measure again. It reads the unit file and every drop-in, from /etc/systemd/system/<unit>.d/, /run/systemd/system/<unit>.d/ and the vendor directories, exactly as systemd would merge them, so you are measuring what would actually run. You can name the unit (demo-report.service) or give its path (/etc/systemd/system/demo-report.service): both load the same drop-ins, so passing the file never means "ignore the overrides".

    Two limits to keep in mind. The score covers only systemd's own features, not the program. And it can't tell whether the service still works under the walls you added. That's the next step.

    systemd-analyze security --offline=true demo-report.service | awk '$NF ~ /^[0-9.]+$/ && $NF >= 0.3'
    ✗ User=/DynamicUser=                                          Service runs as root user                                                                                0.4
    ✗ RestrictNamespaces=~user                                    Service may create user namespaces                                                                       0.3
    ✗ PrivateNetwork=                                             Service has access to the host's network                                                                 0.5
    $NF ~ /^[0-9.]+$/
    the last field is a number (rows without one have nothing left to cost)
    $NF >= 0.3
    and it is worth at least 0.3
    Your shopping list: the remaining crosses that cost the most. Running as root, user namespaces and the network are left. The demo worker needs no network, so PrivateNetwork=yes and RestrictNamespaces=yes would be next; User= or DynamicUser= needs more thought, because it changes who owns the files. Column widths depend on the longest description, so the spacing varies.

    Check yourself

    You hardened a unit with a drop-in, then ran systemd-analyze security --offline=true /etc/systemd/system/app.service. Does the score include the drop-in?

  6. The trap: a sandbox that breaks the service

    Recognise the failure each wall causes, and loosen the one directive at fault instead of deleting the block.

    Confinement breaks services, and it breaks them in recognisable ways. The service starts, then fails at the first write or read the sandbox now forbids, and the journal says why:

    • `ProtectSystem=strict` on a service that writes its own state: Read-only file system (EROFS) on a path under /var/lib, /var/spool or /tmp. Fix: StateDirectory= or ReadWritePaths= for that path, not ProtectSystem=full.
    • `ProtectHome=yes` on a service that reads a user's file: No such file or directory for something you can plainly see in /home. Fix: ProtectHome=read-only.
    • `PrivateTmp=yes` on a service that hands files to another one through `/tmp`: no error at all. It writes into its private /tmp, and the other service never sees the files. Fix: turn it off, and don't use DynamicUser=, which implies it.
    • An empty `CapabilityBoundingSet=` on a service that binds port 80: bind: Permission denied. Fix: keep CAP_NET_BIND_SERVICE.

    On a live host, read it with journalctl -u UNIT -n 20 after a failed start. The rule is the same every time: loosen the one directive that causes the error and keep the rest. Deleting the hardening block because one line hurt throws away nine walls to fix one.

    mkdir -p /var/lib/demo-report && unshare -m sh -c 'mount -o bind,ro /var/lib/demo-report /var/lib/demo-report && touch /var/lib/demo-report/last-run'
    touch: cannot touch '/var/lib/demo-report/last-run': Read-only file system
    unshare -m
    run the command in a new, private mount namespace
    mount -o bind,ro
    bind-mount the directory over itself, read-only
    This is what ProtectSystem=strict does to a state directory that wasn't handed back, done by hand: a private mount namespace with the directory mounted read-only. The write fails with Read-only file system, the exact error a broken service logs. The read-only mount exists only inside that namespace.
    touch /var/lib/demo-report/last-run && ls /var/lib/demo-report
    last-run
    Outside the namespace the same directory is writable. That's why a sandboxing bug never shows up when you test the program from your shell: only the service sees the walls.
    printf '%s\n' '[Service]' 'ProtectSystem=strict' 'ProtectHome=yes' 'PrivateTmp=yes' > /etc/systemd/system/demo-report.service.d/10-filesystem.conf
    Now break the demo unit the same way: rewrite the filesystem drop-in without the StateDirectory= line that handed its state directory back. With ProtectSystem=strict still in place, the worker could no longer write /var/lib/demo-report.
    systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'
    Overall exposure level for demo-report.service: 3.6
    Still 3.6. The analyser can't tell that you just broke the service: StateDirectory= and ReadWritePaths= don't count either way. So a good score is necessary but not sufficient. After every wall, check that the service can still reach what it uses.
    rm -rf /etc/systemd/system/demo-report.service /etc/systemd/system/demo-report.service.d /var/lib/demo-report
    -r
    remove directories and their contents
    -f
    no prompts, no error for missing files
    Cleans up the demo unit, its drop-ins and its state directory.

    Check yourself

    After you add a hardening drop-in, a service fails with 'Read-only file system' writing /var/spool/app. Which one line fixes it while keeping ProtectSystem=strict?

  7. Try it in your live terminal

    Confine a fresh unit with a drop-in, measure it offline, and hand back what it needs.

    Build one from scratch in your lab terminal, with a different name so nothing collides with the demo. The goal: an overall exposure under 4.5, with /var/lib/try-sbx still writable for the service. Predict each new number before you run the measurement. When you're done, the lab systemd-service-sandbox gives you a real worker with its own ceiling, state directory and one more duty to keep, and grades the merged result with this same analyser.

    printf '%s\n' '[Unit]' 'Description=Sandbox practice' '' '[Service]' 'ExecStart=/usr/bin/sleep infinity' > /etc/systemd/system/try-sbx.service
    An unhardened unit. Measure it first so you know the starting number.
    systemd-analyze security --offline=true try-sbx.service | grep -o 'Overall.*: [0-9.]*'
    The baseline. Expect it near the top of the scale.
    mkdir -p /etc/systemd/system/try-sbx.service.d && printf '%s\n' '[Service]' 'NoNewPrivileges=yes' 'ProtectSystem=strict' 'StateDirectory=try-sbx' 'CapabilityBoundingSet=' 'RestrictAddressFamilies=AF_UNIX' 'SystemCallFilter=@system-service' > /etc/systemd/system/try-sbx.service.d/sandbox.conf
    One drop-in with a wall from each family, and the state directory handed back.
    systemd-analyze security --offline=true try-sbx.service | awk '$NF ~ /^[0-9.]+$/ && $NF >= 0.2'
    The expensive crosses still left. Add the ones this do-nothing service obviously doesn't need to sandbox.conf and measure again until you are under 4.5.
    systemd-analyze security --offline=true try-sbx.service | grep -E 'NoNewPrivileges=|ProtectSystem='
    Both should be ticked. Then confirm the drop-in still hands back /var/lib/try-sbx (grep StateDirectory /etc/systemd/system/try-sbx.service.d/*.conf).
    rm -rf /etc/systemd/system/try-sbx.service /etc/systemd/system/try-sbx.service.d
    Clean up the practice unit.
Recap
  • Sandboxing answers "if this service is compromised, what can the attacker reach?". Put the walls in a drop-in so the unit file stays the vendor's.
  • Filesystem: ProtectSystem= true/full/strict (only strict covers /var and /tmp; true leaves /etc writable), ProtectHome=, PrivateTmp=, and ReadWritePaths=/StateDirectory= to hand back exactly what the service writes.
  • Privilege: NoNewPrivileges=yes makes setuid binaries useless to the service and its children; an empty CapabilityBoundingSet= removes root's powers; DynamicUser= implies strict, read-only home, private /tmp and no_new_privs.
  • Kernel: ProtectKernelTunables/Modules/ControlGroups, RestrictAddressFamilies= and SystemCallFilter=@system-service close most of the remaining surface.
  • Measure with systemd-analyze security --offline=true on a box without PID-1 systemd. It merges every drop-in. Read the number on the Overall line, not the face.
  • The score can't see a broken service. When a wall breaks something (Read-only file system, a hidden /home, a private /tmp, a missing capability), loosen that one directive and keep the rest.