Sandbox a systemd Service: Confinement Directives and systemd-analyze security
- Explain what service sandboxing protects against: what an attacker who owns a service can reach
- Build filesystem walls with ProtectSystem= (true, full, strict), ProtectHome=, PrivateTmp= and the Read*/Inaccessible*/StateDirectory= paths
- Remove privilege with NoNewPrivileges=, an empty CapabilityBoundingSet= and User=/DynamicUser=, and say what each one stops
- Narrow the kernel surface with ProtectKernel*=, RestrictAddressFamilies= and SystemCallFilter=@system-service
- Measure a unit plus its drop-ins with systemd-analyze security --offline=true, read the table, and lower the overall exposure
- Recognise a sandbox that breaks the service (read-only state, hidden /home, private /tmp, missing capability) and loosen the one directive at fault
The third question about a service
Frame confinement as the layer between "the service runs" and "the service owns the box".
You already know how to ask two questions about a service: is it running? (
systemctl status) and how much may it use? (CPUQuota=,MemoryMax=). This lesson is about the third: if this service is compromised, what can the attacker reach?By default the answer is "everything the service's user can reach", and a lot of services run as root. A bug in a report worker that only ever needed to read one directory and write another hands over every file on the box, every device, every kernel interface, the network, and the chance to escalate further through setuid binaries.
systemd can shrink that, per service, with directives in the unit's
[Service]section. Behind the scenes it starts the service in its own mount namespace (so it can make parts of the filesystem read-only or invisible), drops capabilities, sets the kernel's no_new_privs flag, and installs a seccomp system-call filter. You write one line per wall; systemd builds the wall at start-up. You add them the way the overrides lesson taught: in a drop-in, so the vendor's unit file stays untouched.systemd-analyze --version | head -n 1systemd 252 (252-67.el9_8.6.rocky.0.1)
- --version
- print the systemd release and its compile-time features
- head -n 1
- keep only the first line
The systemd release decides which directives and which systemd-analyzeoptions exist. 252 is new enough for everything in this lesson. The text in brackets is the distribution's build and will differ from box to box.systemd-analyze security sshd.service 2>&1System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is down
The online form asks the running service manager over D-Bus, and there isn't one here. That's the container, not a mistake. On a normal server this prints the table you're about to see. 2>&1just shows the error in the same stream.printf '%s\n' '[Unit]' 'Description=Demo nightly report worker' '' '[Service]' 'ExecStart=/usr/bin/sleep infinity' > /etc/systemd/system/demo-report.serviceA minimal unit to experiment on: no User=, so it would run as root, and no hardening at all. That's what most hand-written units look like.printf '%s\n'writes each quoted argument as one line, which is an easy way to write a small file without an editor.systemd-analyze security --offline=true demo-report.service | tail -n 1→ Overall exposure level for demo-report.service: 9.6 UNSAFE 😨
- --offline=true
- analyse the unit file and its drop-ins from disk; no running systemd needed
The verdict: an exposure level from 0.0 (tightly confined) to 10.0 (nothing confined). 9.6 is roughly where every unhardened root service lands. The arrow and face depend on your terminal: a UTF-8 terminal shows →and an emoji, a plain one shows->and:-{. The number is the part to read.systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'Overall exposure level for demo-report.service: 9.6
- --offline=true
- analyse the files on disk, no running systemd needed
- grep -o 'Overall.*: [0-9.]*'
- print only the matching part: the label and the number, without the glyphs
The same line with the glyphs and the face cut off: grep -oprints only the part that matches, up to the last digit. You'll use this form to watch the number move as you add walls.Check yourself
A service scores 9.6 in systemd-analyze security. Does that mean the service's code has a vulnerability?
Filesystem walls
Make the OS read-only for the service, hide what it never needs, and hand back exactly the paths it writes.
The filesystem walls decide what the service can see and change. All of them work by giving the service its own mount namespace, so the rest of the system is unaffected.
- `ProtectSystem=` has three strengths, and learners routinely pick the weakest and assume it covered
/etc:
-
true:/usr,/bootand/efiare read-only./etcis still writable.-
full: the same plus/etc.-
strict: the entire filesystem is read-only, except the API filesystems/dev,/procand/sys, and/home,/rootand/run/user, which are left toProtectHome=. This includes/varand/tmp, so the service's own state too.- `ProtectHome=`:
yesmakes/home,/rootand/run/userempty and inaccessible;read-onlyleaves them readable;tmpfsmounts an empty, read-only tmpfs over them, onto which you can bind just the paths the service needs. - `PrivateTmp=yes` gives the service its own empty
/tmpand/var/tmp, invisible to everyone else. That stops/tmpraces and snooping on other services' temp files. - Paths:
ReadWritePaths=,ReadOnlyPaths=andInaccessiblePaths=adjust single paths. A deeper path overrides a shallower one, soProtectSystem=strictplusReadWritePaths=/var/lib/demois "everything read-only except that directory". The one exception: below an inaccessible path, nothing comes back. - `StateDirectory=NAME` is the tidy way to hand back state: systemd creates
/var/lib/NAME, makes it writable inside the sandbox, and owns it to the service's user.
mkdir -p /etc/systemd/system/demo-report.service.d- -p
- create parent directories as needed; no error if it already exists
The drop-in directory from the overrides lesson. Every *.conffile in it is merged over the unit, in file-name order, so you can split the hardening into themed files.printf '%s\n' '[Service]' 'ProtectSystem=strict' 'StateDirectory=demo-report' 'ProtectHome=yes' 'PrivateTmp=yes' > /etc/systemd/system/demo-report.service.d/10-filesystem.confThe filesystem walls in their own drop-in: everything read-only, /var/lib/demo-reporthanded back for state, home directories hidden, and a private/tmp. Nothing prints on success.systemd-analyze security --offline=true demo-report.service | grep -E 'ProtectSystem=|ProtectHome=|PrivateTmp='✓ ProtectSystem= Service has strict read-only access to the OS file hierarchy ✓ PrivateTmp= Service has no access to other software's temporary files ✓ ProtectHome= Service has no access to home directories
- --offline=true
- analyse the files on disk, no running systemd needed
- grep -E
- extended regex, so | separates alternatives
The analyser read the unit and the drop-in: all three rows are now ticked. A tick shows as ✓in a UTF-8 terminal and+in a plain one (a cross is✗or-). The description column tells you which strength it found, soProtectSystem=fullwould read "very limited write access" instead of "strict".systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'Overall exposure level for demo-report.service: 9.0
9.6 down to 9.0. The filesystem walls matter enormously to a real attacker, but the score weights them lightly. The big numbers come in the next two steps. Don't read a small drop as "not worth it". Check yourself
Which ProtectSystem= value makes /etc read-only but leaves /var writable?
- `ProtectSystem=` has three strengths, and learners routinely pick the weakest and assume it covered
Privilege walls
Stop the service from gaining privilege, and take away the root powers it never uses.
`NoNewPrivileges=yes` is the cheapest line in the file. It sets the kernel's
no_new_privsflag on the service, and the flag is inherited by every child and survives everyexecve(). From then on, running a setuid or setgid binary, or one with file capabilities, gives the process nothing extra. The hardening lesson's SUID audit finds the binaries that could be abused; this directive makes all of them useless to the confined service, including ones you haven't found yet. Some directives imply it, for exampleDynamicUser=yes.`CapabilityBoundingSet=` limits which of root's capabilities (the pieces root's power is split into:
CAP_SYS_ADMIN,CAP_NET_ADMIN,CAP_DAC_OVERRIDE, and about forty more) the service can ever hold. List the ones to keep, or prefix with~to list the ones to drop. An empty value,CapabilityBoundingSet=, keeps none: root in name only. It can still write files it owns, but it can't override permissions, load modules, change the network, or bind ports below 1024.`User=` runs the service as an ordinary account, and `DynamicUser=yes` invents a throwaway user for each run.
DynamicUser=also switches onProtectSystem=strict,ProtectHome=read-only,PrivateTmp=yesandNoNewPrivileges=yes, which is powerful and easy to forget.grep NoNewPrivs /proc/self/statusNoNewPrivs: 0
The kernel flag itself. /proc/self/statusdescribes the process reading it, heregrep. 0 means a setuid binary run from this shell would still gain its owner's rights.setpriv --no-new-privs grep NoNewPrivs /proc/self/statusNoNewPrivs: 1
- --no-new-privs
- set the no_new_privs flag, then run the command
setprivsets the same flag systemd sets forNoNewPrivileges=yes, then runsgrep, which reports 1. Nothing can clear it again for that process or its children. That's exactly what the service gets.printf '%s\n' '[Service]' 'NoNewPrivileges=yes' 'CapabilityBoundingSet=' > /etc/systemd/system/demo-report.service.d/20-privileges.confA second drop-in for the privilege walls. The empty CapabilityBoundingSet=is deliberate: an empty value means "no capabilities", not "unchanged".systemd-analyze security --offline=true demo-report.service | grep -E 'NoNewPrivileges=|~CAP_SYS_ADMIN|~CAP_NET_\('✓ NoNewPrivileges= Service processes cannot acquire new privileges ✓ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has no elevated networking privileges ✓ CapabilityBoundingSet=~CAP_SYS_ADMIN Service has no administrator privileges
The analyser has one row per capability group, read as "the bounding set excludes ( ~) these". With an empty set every group is ticked. Note the networking group:CAP_NET_BIND_SERVICEis gone, so this service could no longer listen on port 80.systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'Overall exposure level for demo-report.service: 6.4
9.0 to 6.4 from two lines. The capability rows carry heavy weights because each one is a separate road from "compromised service" to "compromised kernel or host". Check yourself
A confined service has NoNewPrivileges=yes and finds a setuid-root copy of bash on the box. What does running it give the service?
Kernel and system-call walls
Take the kernel's control knobs, unneeded socket types and dangerous system calls away from the service.
The last group narrows what the service can ask the kernel to do. You don't need to understand seccomp to use these; you need to know what each one closes.
- `ProtectKernelTunables=yes` makes
/proc/sysand similar knobs read-only, so the service can't change sysctls. - `ProtectKernelModules=yes` denies loading or reading kernel modules. `ProtectControlGroups=yes` makes the cgroup tree read-only.
- `RestrictAddressFamilies=` lists the socket types the service may open. A worker that only talks to local daemons needs
AF_UNIX; a network service addsAF_INET AF_INET6. Raw packet and netlink sockets, favourites for attackers, are then refused. - `SystemCallFilter=@system-service` installs a seccomp allow-list of the system calls ordinary services use. Mounting, rebooting, loading modules, swapping, changing the clock and other privileged calls are refused.
@system-serviceis a named group;systemd-analyze syscall-filtershows what is in it. - `SystemCallArchitectures=native` blocks the old 32-bit call path on a 64-bit box, a classic way around filters.
Keep this step shallow on purpose: the named groups are maintained by systemd for exactly this use, so
@system-serviceis the right default and hand-written call lists are rarely worth it.systemd-analyze syscall-filter @system-service | head -n 6@system-service # General system service operations @aio @basic-io @chown @default- syscall-filter @system-service
- list the system calls in that named group
- head -n 6
- show only the first six lines
syscall-filterworks offline: the groups are compiled into systemd.@system-serviceis built from smaller groups (asynchronous I/O, basic I/O, ownership changes, the defaults every program needs) plus individual calls. Drop the| headto see all of it.printf '%s\n' '[Service]' 'ProtectKernelTunables=yes' 'ProtectKernelModules=yes' 'ProtectControlGroups=yes' 'RestrictAddressFamilies=AF_UNIX' 'SystemCallFilter=@system-service' 'SystemCallArchitectures=native' > /etc/systemd/system/demo-report.service.d/30-kernel.confThe kernel walls in a third drop-in. The demo worker only needs local sockets, so AF_UNIXis the whole allow-list.systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'Overall exposure level for demo-report.service: 3.6
6.4 to 3.6. The address families and the system-call filter are scored as a dozen separate rows, so getting them right moves the number a lot. Check yourself
A service only talks to a local database over a Unix socket. What should RestrictAddressFamilies= list?
- `ProtectKernelTunables=yes` makes
Measure it
Read the security table, use it to choose the next wall, and know what the number does and doesn't mean.
systemd-analyze securityturns hardening from a checklist into a measurement. Each row is one setting: a tick (✓) when the unit has it, a cross (✗) when it doesn't, with the exposure that row adds in the last column. Rows with no mark don't apply (for exampleSupplementaryGroups=for a root service). The overall level combines the rows; it is systemd's estimate, from 0.0 to 10.0, of how exposed the system is if this service is taken over.The demo unit now has three drop-ins,
10-filesystem.conf,20-privileges.confand30-kernel.conf, merged in that order. Splitting by theme makes it obvious later which wall broke something.The workflow is a loop: measure, pick the expensive crosses, add a wall, measure again. It reads the unit file and every drop-in, from
/etc/systemd/system/<unit>.d/,/run/systemd/system/<unit>.d/and the vendor directories, exactly as systemd would merge them, so you are measuring what would actually run. You can name the unit (demo-report.service) or give its path (/etc/systemd/system/demo-report.service): both load the same drop-ins, so passing the file never means "ignore the overrides".Two limits to keep in mind. The score covers only systemd's own features, not the program. And it can't tell whether the service still works under the walls you added. That's the next step.
systemd-analyze security --offline=true demo-report.service | awk '$NF ~ /^[0-9.]+$/ && $NF >= 0.3'✗ User=/DynamicUser= Service runs as root user 0.4 ✗ RestrictNamespaces=~user Service may create user namespaces 0.3 ✗ PrivateNetwork= Service has access to the host's network 0.5
- $NF ~ /^[0-9.]+$/
- the last field is a number (rows without one have nothing left to cost)
- $NF >= 0.3
- and it is worth at least 0.3
Your shopping list: the remaining crosses that cost the most. Running as root, user namespaces and the network are left. The demo worker needs no network, so PrivateNetwork=yesandRestrictNamespaces=yeswould be next;User=orDynamicUser=needs more thought, because it changes who owns the files. Column widths depend on the longest description, so the spacing varies.Check yourself
You hardened a unit with a drop-in, then ran systemd-analyze security --offline=true /etc/systemd/system/app.service. Does the score include the drop-in?
The trap: a sandbox that breaks the service
Recognise the failure each wall causes, and loosen the one directive at fault instead of deleting the block.
Confinement breaks services, and it breaks them in recognisable ways. The service starts, then fails at the first write or read the sandbox now forbids, and the journal says why:
- `ProtectSystem=strict` on a service that writes its own state:
Read-only file system(EROFS) on a path under/var/lib,/var/spoolor/tmp. Fix:StateDirectory=orReadWritePaths=for that path, notProtectSystem=full. - `ProtectHome=yes` on a service that reads a user's file:
No such file or directoryfor something you can plainly see in/home. Fix:ProtectHome=read-only. - `PrivateTmp=yes` on a service that hands files to another one through `/tmp`: no error at all. It writes into its private
/tmp, and the other service never sees the files. Fix: turn it off, and don't useDynamicUser=, which implies it. - An empty `CapabilityBoundingSet=` on a service that binds port 80:
bind: Permission denied. Fix: keepCAP_NET_BIND_SERVICE.
On a live host, read it with
journalctl -u UNIT -n 20after a failed start. The rule is the same every time: loosen the one directive that causes the error and keep the rest. Deleting the hardening block because one line hurt throws away nine walls to fix one.mkdir -p /var/lib/demo-report && unshare -m sh -c 'mount -o bind,ro /var/lib/demo-report /var/lib/demo-report && touch /var/lib/demo-report/last-run'touch: cannot touch '/var/lib/demo-report/last-run': Read-only file system
- unshare -m
- run the command in a new, private mount namespace
- mount -o bind,ro
- bind-mount the directory over itself, read-only
This is what ProtectSystem=strictdoes to a state directory that wasn't handed back, done by hand: a private mount namespace with the directory mounted read-only. The write fails withRead-only file system, the exact error a broken service logs. The read-only mount exists only inside that namespace.touch /var/lib/demo-report/last-run && ls /var/lib/demo-reportlast-run
Outside the namespace the same directory is writable. That's why a sandboxing bug never shows up when you test the program from your shell: only the service sees the walls. printf '%s\n' '[Service]' 'ProtectSystem=strict' 'ProtectHome=yes' 'PrivateTmp=yes' > /etc/systemd/system/demo-report.service.d/10-filesystem.confNow break the demo unit the same way: rewrite the filesystem drop-in without the StateDirectory=line that handed its state directory back. WithProtectSystem=strictstill in place, the worker could no longer write/var/lib/demo-report.systemd-analyze security --offline=true demo-report.service | grep -o 'Overall.*: [0-9.]*'Overall exposure level for demo-report.service: 3.6
Still 3.6. The analyser can't tell that you just broke the service: StateDirectory=andReadWritePaths=don't count either way. So a good score is necessary but not sufficient. After every wall, check that the service can still reach what it uses.rm -rf /etc/systemd/system/demo-report.service /etc/systemd/system/demo-report.service.d /var/lib/demo-report- -r
- remove directories and their contents
- -f
- no prompts, no error for missing files
Cleans up the demo unit, its drop-ins and its state directory. Check yourself
After you add a hardening drop-in, a service fails with 'Read-only file system' writing /var/spool/app. Which one line fixes it while keeping ProtectSystem=strict?
- `ProtectSystem=strict` on a service that writes its own state:
Try it in your live terminal
Confine a fresh unit with a drop-in, measure it offline, and hand back what it needs.
Build one from scratch in your lab terminal, with a different name so nothing collides with the demo. The goal: an overall exposure under 4.5, with
/var/lib/try-sbxstill writable for the service. Predict each new number before you run the measurement. When you're done, the lab systemd-service-sandbox gives you a real worker with its own ceiling, state directory and one more duty to keep, and grades the merged result with this same analyser.printf '%s\n' '[Unit]' 'Description=Sandbox practice' '' '[Service]' 'ExecStart=/usr/bin/sleep infinity' > /etc/systemd/system/try-sbx.serviceAn unhardened unit. Measure it first so you know the starting number. systemd-analyze security --offline=true try-sbx.service | grep -o 'Overall.*: [0-9.]*'The baseline. Expect it near the top of the scale. mkdir -p /etc/systemd/system/try-sbx.service.d && printf '%s\n' '[Service]' 'NoNewPrivileges=yes' 'ProtectSystem=strict' 'StateDirectory=try-sbx' 'CapabilityBoundingSet=' 'RestrictAddressFamilies=AF_UNIX' 'SystemCallFilter=@system-service' > /etc/systemd/system/try-sbx.service.d/sandbox.confOne drop-in with a wall from each family, and the state directory handed back. systemd-analyze security --offline=true try-sbx.service | awk '$NF ~ /^[0-9.]+$/ && $NF >= 0.2'The expensive crosses still left. Add the ones this do-nothing service obviously doesn't need to sandbox.confand measure again until you are under 4.5.systemd-analyze security --offline=true try-sbx.service | grep -E 'NoNewPrivileges=|ProtectSystem='Both should be ticked. Then confirm the drop-in still hands back /var/lib/try-sbx(grep StateDirectory /etc/systemd/system/try-sbx.service.d/*.conf).rm -rf /etc/systemd/system/try-sbx.service /etc/systemd/system/try-sbx.service.dClean up the practice unit.
- Sandboxing answers "if this service is compromised, what can the attacker reach?". Put the walls in a drop-in so the unit file stays the vendor's.
- Filesystem: ProtectSystem= true/full/strict (only strict covers /var and /tmp; true leaves /etc writable), ProtectHome=, PrivateTmp=, and ReadWritePaths=/StateDirectory= to hand back exactly what the service writes.
- Privilege: NoNewPrivileges=yes makes setuid binaries useless to the service and its children; an empty CapabilityBoundingSet= removes root's powers; DynamicUser= implies strict, read-only home, private /tmp and no_new_privs.
- Kernel: ProtectKernelTunables/Modules/ControlGroups, RestrictAddressFamilies= and SystemCallFilter=@system-service close most of the remaining surface.
- Measure with systemd-analyze security --offline=true on a box without PID-1 systemd. It merges every drop-in. Read the number on the Overall line, not the face.
- The score can't see a broken service. When a wall breaks something (Read-only file system, a hidden /home, a private /tmp, a missing capability), loosen that one directive and keep the rest.
