SELinux Troubleshooting: From AVC Denial to Root-Cause Fix
- Recognise the signature of an SELinux denial and confirm it with permissive mode
- Find and read an AVC denial in the audit log (scontext, tcontext, tclass, permission)
- Walk the troubleshooting decision tree: wrong label vs boolean vs port vs genuine policy gap
- Fix the root cause with restorecon/semanage/setsebool — and use audit2allow only as a last resort
- Run audit2why and audit2allow against a saved log and the on-disk policy, scope permissive mode to one domain, and verify persistent changes with semanage -l -C
The signature: permissions are fine, access still fails
Recognise when a failure is SELinux rather than DAC.
SELinux denials wear a disguise: the application reports a plain Permission denied, but the file's owner, group, and mode all look correct. That mismatch — DAC says yes, the app says no — is the tell. A web server that can read
/var/www/htmlbut returns 403 for a page you moved in from your home directory; a service that won't bind a non-standard port; a container that can't write a volume — all classic SELinux, not Unix permissions.The reason is that SELinux is Mandatory Access Control layered on top of discretionary permissions. Even after the mode/owner check passes, the kernel consults the policy: does the process's type (its domain) have permission to act on the target's type? If not, it's denied and — in enforcing mode — logged as an AVC (Access Vector Cache) denial. Your job in troubleshooting is to turn that denial into a specific, minimal fix rather than the reflexive
setenforce 0that disables the whole protection.Confirm it with permissive mode
Use permissive as a probe to prove SELinux is the cause.
Before you spend time chasing labels, prove SELinux is actually the culprit. Permissive mode applies the policy in name only: it logs what it would deny but allows everything. So if a failure disappears in permissive and returns in enforcing, SELinux owns it.
Example — not runnable on your lab box
sudo setenforce 0 # permissive: log, don't block # ... reproduce the failure ... sudo setenforce 1 # back to enforcingsetenforce 0/1flips the runtime mode instantly (it does not touch/etc/selinux/config, so a reboot restores the configured mode). If the operation succeeds while permissive, SELinux was blocking it — and, crucially, the attempt is now recorded as an AVC you can read. If it still fails in permissive, look elsewhere: DAC permissions, the firewall, or the app's own config.Find the denial in the audit log
Locate AVC denials with ausearch and the SELinux log helpers.
AVC denials are written by the kernel and captured by auditd in
/var/log/audit/audit.log. Don't grep that file by hand —ausearchfilters it for you, and-ts(time-start) narrows to the window you care about.Example — not runnable on your lab box
sudo ausearch -m avc -ts recenttype=AVC msg=audit(...): avc: denied { read } for pid=1234 comm="httpd" name="report.html" dev="vda1" ino=98765 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0-m avcselects AVC records;-ts recentlimits to roughly the last ten minutes (-ts todayor a timestamp also work). One denial line carries the whole story: comm (the program), scontext (the source/process context), tcontext (the target's context), tclass (file, tcp_socket, dir…), and the permission in braces (read,write,name_bind…).On a system with setroubleshoot installed,
sealert -a /var/log/audit/audit.log(or the desktop alert) translates each denial into plain English and suggests candidate fixes — a useful cross-check, though you should still understand the fix before applying it.Read the AVC: what exactly was denied
Decode scontext/tcontext to see which label is wrong.
A context is
user:role:type:level. For troubleshooting, the type (the_tfield) is almost always what matters. In the example above the source type ishttpd_t(the web server's domain) and the target type isuser_home_t— a file still labelled as if it lived in someone's home directory. The policy simply has no rule lettinghttpd_treaduser_home_t, so: denied.Read it as a sentence: 'a process of type `httpd_t` was denied `read` on a `file` of type `user_home_t`.' Once you can say that sentence, the fix follows from which half is wrong — the target's label, or the policy that governs the source.
Example — not runnable on your lab box
ls -Z /var/www/html/report.html; ps -eZ | grep httpdunconfined_u:object_r:user_home_t:s0 /var/www/html/report.html system_u:system_r:httpd_t:s0 1234 ? 00:00:00 httpd
ls -Zshows a file's context andps -eZa process's. Here the file's type (user_home_t) doesn't match whathttpd_tis allowed to read (httpd_sys_content_t) — a textbook wrong-label denial caused by moving a file withmv(which preserves the old context) instead of copying or relabelling it.To decide quickly whether a label is wrong, compare the file's actual context against the default the policy expects for that path.
matchpathcon /var/www/html/report.html(or the newerselabel_lookup) prints what the label should be; if that differs from whatls -Zshows, you have a relabel problem and the fix isrestorecon, not a boolean or a module. When the actual and expected already agree yet access is still denied, the label is fine and you move on down the tree — the cause is a boolean, a port, or a genuine policy gap. Building that actual-versus-expected reflex is what separates a five-minute fix from an afternoon of guesswork.The decision tree: label, boolean, port, or policy
Route the denial to the right kind of fix.
Almost every denial resolves to one of four causes — check them in this order:
1. Wrong file label (most common). The target's type doesn't match its location/role. Fix by relabelling:
restorecon(restore the default) orsemanage fcontext(define a new default) thenrestorecon.2. A boolean is off. The policy already supports what you want, but it's gated behind a tunable — e.g.
httpd_can_network_connect. Fix withsetsebool -P.3. A non-standard port. A daemon is binding a port the policy doesn't associate with its type (
name_bindon atcp_socket). Fix withsemanage port -a.4. A genuine policy gap. Nothing above fits and the action is legitimately novel. Only now consider a custom module via
audit2allow.Check yourself
An AVC shows httpd_t denied name_bind on tcp_socket for port 8088. Which branch of the tree is this?
Fix #1 — the wrong label (restorecon & semanage fcontext)
Relabel correctly and make it survive a relabel/reboot.
If the file simply has the wrong label for where it lives,
restoreconresets it to the policy's default for that path. If you've put content in a new location the policy doesn't know about, first teach the policy the default withsemanage fcontext, then apply it withrestorecon.Example — not runnable on your lab box
sudo restorecon -Rv /var/www/html # custom location? define the default first, then apply: sudo semanage fcontext -a -t httpd_sys_content_t "/srv/web(/.*)?" sudo restorecon -Rv /srv/webRelabeled /var/www/html/report.html from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:httpd_sys_content_t:s0
restorecon -Rvrecurses (-R) and prints what it changes (-v).semanage fcontext -aadds a persistent rule mapping a path regex to a type — it edits the policy's file-context database, so a future full relabel (restoreconortouch /.autorelabel; reboot) keeps your setting. Usingchconinstead would fix it now but be wiped by the next relabel.Fix #2 — a boolean or a port (and audit2allow as last resort)
Flip the right tunable, or generate a scoped module only when nothing else fits.
When the label is right but the policy still says no, a boolean usually gates the behaviour. List candidates with
getsebool -a | grep httpd(orsemanage boolean -lfor descriptions), then set persistently with-P. For a daemon on a non-standard port, register the port with the daemon's port type.Example — not runnable on your lab box
sudo setsebool -P httpd_can_network_connect on sudo semanage port -a -t http_port_t -p tcp 8088setsebool -Pwrites the boolean to persistent storage so it survives reboot (drop the-Pand it reverts — a notorious 'works today, breaks after the next restart' trap).semanage port -a -t http_port_t -p tcp 8088tells the policy that TCP 8088 is a legitimate http port, sohttpd_tmayname_bindit.Only when the denial is genuinely new — no label, boolean, or port fits — do you build a custom module.
ausearch -m avc -ts recent | audit2allow -M mymodulegenerates a.te/.ppfrom the logged denials;semodule -i mymodule.ppinstalls it. Read the generated rules first.audit2allowwill happily grant whatever was denied, including denials caused by a compromise or a wrong label you should have fixed instead. It is the tool of last resort, not the first.On the lab box: audit2why, per-domain permissive, and proving the fix persisted
Diagnose a saved denial offline, confine permissive mode to one domain, and read back exactly what you changed.
The Signal9 SELinux labs hand you denials as saved AVC records (
/root/case/avc.log) on a box whose kernel runs with SELinux disabled. Nothing is enforced live, so the work is the part that persists: the policy store that an enforcing box loads at boot. Three tools carry that work, and each takes the policy file explicitly when there is no live kernel policy to ask.`audit2why` reads an AVC and tells you which branch of the decision tree it is on. Point it at the on-disk policy with
-pand at the log with-i.Example — not runnable on your lab box
P=$(ls /etc/selinux/targeted/policy/policy.*) audit2why -p "$P" -i /root/case/avc.logtype=AVC msg=audit(...): avc: denied { open } for pid=1526 comm="logrotate" name="app.log" dev="vda1" ino=7762 scontext=system_u:system_r:logrotate_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0 Was caused by: Missing type enforcement (TE) allow rule. You can use audit2allow to generate a loadable module to allow this access.The verdict maps straight onto the tree. Missing type enforcement (TE) allow rule means no boolean covers it: check the label first ( matchpathcon, step 4), and only if that is right is it theaudit2allowcase, which takes the same-p/-iflags. The boolean … was set incorrectly is the boolean branch, and it names thesetsebool -Pthat fixes it. One of the following booleans lists several: take the one scoped to this access, not a catch-all likehttpd_unified. After a fix, re-run it: Unknown - would be allowed by active policy means the policy now permits that record.`semanage permissive` is the scoped version of the permissive probe from step 2. Instead of
setenforce 0, which drops protection for every service, it makes one domain permissive: that process's denials are logged but allowed, and everything else stays enforced. You need the domain (the process type, likesquid_t), not the type of its binary on disk (squid_exec_t).matchpathconon the binary gives you the exec type, and the domain is usually the same name ending in_t.Example — not runnable on your lab box
matchpathcon /usr/sbin/squid semanage permissive -a squid_t semanage permissive -l | tail -3 # done diagnosing? put it back: semanage permissive -d squid_t/usr/sbin/squid system_u:object_r:squid_exec_t:s0 Customized Permissive Types squid_t
-aadds the domain,-llists the built-in permissive types followed by your Customized ones, and-dremoves it.semanage permissive -a squid_exec_tis accepted without complaint, but no process runs in a file type, so it changes nothing. Read the listing back before you trust it.Prove the change persisted.
semanagewrites every fix (file-context rules, port labels, booleans) to the policy store, and adding-Clists only the local customizations, which is exactly what your fix changed. It's the quickest check that you changed one thing and not three.Example — not runnable on your lab box
semanage fcontext -l -C semanage port -l -C semanage boolean -l -C matchpathcon /srv/shop/index.htmlSELinux fcontext type Context /srv/shop(/.*)? all files system_u:object_r:httpd_sys_content_t:s0 SELinux Port Type Proto Port Number http_port_t tcp 8691 SELinux boolean State Default Description httpd_enable_homedirs (on , on) Allow httpd to enable homedirs /srv/shop/index.html system_u:object_r:httpd_sys_content_t:s0
Each listing is empty until you customize something, so an extra line means an extra change (a second port, a /srv(/.*)?rule that also labels the directory next door).matchpathconasks the policy what label a path will get, which is whatrestoreconapplies, so you can check a file-context rule before relabeling anything. An equivalence rule (semanage fcontext -a -e /var/www /srv/shop) is another correct way to give a new web root the/var/wwwlabels, and it shows up in the same-l -Clisting.
- The SELinux signature is DAC-says-yes-app-says-no; confirm it by reproducing under permissive (setenforce 0), which also logs the AVC.
- Read the AVC with ausearch -m avc: scontext/tcontext types and the permission in braces tell you exactly what was denied to whom.
- Walk the tree in order — wrong label (restorecon/semanage fcontext) → boolean (setsebool -P) → port (semanage port) → genuine gap (audit2allow, last resort).
- Prefer semanage fcontext + restorecon over chcon, and always setsebool -P, so fixes survive a relabel and a reboot.
