Signal9Signal9 — home
online
all lessons

SELinux Troubleshooting: From AVC Denial to Root-Cause Fix

Advanced40 min8 steps
What you'll learn
  • Recognise the signature of an SELinux denial and confirm it with permissive mode
  • Find and read an AVC denial in the audit log (scontext, tcontext, tclass, permission)
  • Walk the troubleshooting decision tree: wrong label vs boolean vs port vs genuine policy gap
  • Fix the root cause with restorecon/semanage/setsebool — and use audit2allow only as a last resort
  • Run audit2why and audit2allow against a saved log and the on-disk policy, scope permissive mode to one domain, and verify persistent changes with semanage -l -C
  1. The signature: permissions are fine, access still fails

    Recognise when a failure is SELinux rather than DAC.

    SELinux denials wear a disguise: the application reports a plain Permission denied, but the file's owner, group, and mode all look correct. That mismatch — DAC says yes, the app says no — is the tell. A web server that can read /var/www/html but returns 403 for a page you moved in from your home directory; a service that won't bind a non-standard port; a container that can't write a volume — all classic SELinux, not Unix permissions.

    The reason is that SELinux is Mandatory Access Control layered on top of discretionary permissions. Even after the mode/owner check passes, the kernel consults the policy: does the process's type (its domain) have permission to act on the target's type? If not, it's denied and — in enforcing mode — logged as an AVC (Access Vector Cache) denial. Your job in troubleshooting is to turn that denial into a specific, minimal fix rather than the reflexive setenforce 0 that disables the whole protection.

  2. Confirm it with permissive mode

    Use permissive as a probe to prove SELinux is the cause.

    Before you spend time chasing labels, prove SELinux is actually the culprit. Permissive mode applies the policy in name only: it logs what it would deny but allows everything. So if a failure disappears in permissive and returns in enforcing, SELinux owns it.

    Example — not runnable on your lab box

    sudo setenforce 0   # permissive: log, don't block
    # ... reproduce the failure ...
    sudo setenforce 1   # back to enforcing
    setenforce 0/1 flips the runtime mode instantly (it does not touch /etc/selinux/config, so a reboot restores the configured mode). If the operation succeeds while permissive, SELinux was blocking it — and, crucially, the attempt is now recorded as an AVC you can read. If it still fails in permissive, look elsewhere: DAC permissions, the firewall, or the app's own config.
  3. Find the denial in the audit log

    Locate AVC denials with ausearch and the SELinux log helpers.

    AVC denials are written by the kernel and captured by auditd in /var/log/audit/audit.log. Don't grep that file by hand — ausearch filters it for you, and -ts (time-start) narrows to the window you care about.

    Example — not runnable on your lab box

    sudo ausearch -m avc -ts recent
    type=AVC msg=audit(...): avc:  denied  { read } for  pid=1234 comm="httpd" name="report.html" dev="vda1" ino=98765 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
    -m avc selects AVC records; -ts recent limits to roughly the last ten minutes (-ts today or a timestamp also work). One denial line carries the whole story: comm (the program), scontext (the source/process context), tcontext (the target's context), tclass (file, tcp_socket, dir…), and the permission in braces (read, write, name_bind…).

    On a system with setroubleshoot installed, sealert -a /var/log/audit/audit.log (or the desktop alert) translates each denial into plain English and suggests candidate fixes — a useful cross-check, though you should still understand the fix before applying it.

  4. Read the AVC: what exactly was denied

    Decode scontext/tcontext to see which label is wrong.

    A context is user:role:type:level. For troubleshooting, the type (the _t field) is almost always what matters. In the example above the source type is httpd_t (the web server's domain) and the target type is user_home_t — a file still labelled as if it lived in someone's home directory. The policy simply has no rule letting httpd_t read user_home_t, so: denied.

    Read it as a sentence: 'a process of type `httpd_t` was denied `read` on a `file` of type `user_home_t`.' Once you can say that sentence, the fix follows from which half is wrong — the target's label, or the policy that governs the source.

    Example — not runnable on your lab box

    ls -Z /var/www/html/report.html; ps -eZ | grep httpd
    unconfined_u:object_r:user_home_t:s0 /var/www/html/report.html
    system_u:system_r:httpd_t:s0    1234 ?  00:00:00 httpd
    ls -Z shows a file's context and ps -eZ a process's. Here the file's type (user_home_t) doesn't match what httpd_t is allowed to read (httpd_sys_content_t) — a textbook wrong-label denial caused by moving a file with mv (which preserves the old context) instead of copying or relabelling it.

    To decide quickly whether a label is wrong, compare the file's actual context against the default the policy expects for that path. matchpathcon /var/www/html/report.html (or the newer selabel_lookup) prints what the label should be; if that differs from what ls -Z shows, you have a relabel problem and the fix is restorecon, not a boolean or a module. When the actual and expected already agree yet access is still denied, the label is fine and you move on down the tree — the cause is a boolean, a port, or a genuine policy gap. Building that actual-versus-expected reflex is what separates a five-minute fix from an afternoon of guesswork.

  5. The decision tree: label, boolean, port, or policy

    Route the denial to the right kind of fix.

    Almost every denial resolves to one of four causes — check them in this order:

    1. Wrong file label (most common). The target's type doesn't match its location/role. Fix by relabelling: restorecon (restore the default) or semanage fcontext (define a new default) then restorecon.

    2. A boolean is off. The policy already supports what you want, but it's gated behind a tunable — e.g. httpd_can_network_connect. Fix with setsebool -P.

    3. A non-standard port. A daemon is binding a port the policy doesn't associate with its type (name_bind on a tcp_socket). Fix with semanage port -a.

    4. A genuine policy gap. Nothing above fits and the action is legitimately novel. Only now consider a custom module via audit2allow.

    Check yourself

    An AVC shows httpd_t denied name_bind on tcp_socket for port 8088. Which branch of the tree is this?

  6. Fix #1 — the wrong label (restorecon & semanage fcontext)

    Relabel correctly and make it survive a relabel/reboot.

    If the file simply has the wrong label for where it lives, restorecon resets it to the policy's default for that path. If you've put content in a new location the policy doesn't know about, first teach the policy the default with semanage fcontext, then apply it with restorecon.

    Example — not runnable on your lab box

    sudo restorecon -Rv /var/www/html
    # custom location? define the default first, then apply:
    sudo semanage fcontext -a -t httpd_sys_content_t "/srv/web(/.*)?"
    sudo restorecon -Rv /srv/web
    Relabeled /var/www/html/report.html from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:httpd_sys_content_t:s0
    restorecon -Rv recurses (-R) and prints what it changes (-v). semanage fcontext -a adds a persistent rule mapping a path regex to a type — it edits the policy's file-context database, so a future full relabel (restorecon or touch /.autorelabel; reboot) keeps your setting. Using chcon instead would fix it now but be wiped by the next relabel.
  7. Fix #2 — a boolean or a port (and audit2allow as last resort)

    Flip the right tunable, or generate a scoped module only when nothing else fits.

    When the label is right but the policy still says no, a boolean usually gates the behaviour. List candidates with getsebool -a | grep httpd (or semanage boolean -l for descriptions), then set persistently with -P. For a daemon on a non-standard port, register the port with the daemon's port type.

    Example — not runnable on your lab box

    sudo setsebool -P httpd_can_network_connect on
    sudo semanage port -a -t http_port_t -p tcp 8088
    setsebool -P writes the boolean to persistent storage so it survives reboot (drop the -P and it reverts — a notorious 'works today, breaks after the next restart' trap). semanage port -a -t http_port_t -p tcp 8088 tells the policy that TCP 8088 is a legitimate http port, so httpd_t may name_bind it.

    Only when the denial is genuinely new — no label, boolean, or port fits — do you build a custom module. ausearch -m avc -ts recent | audit2allow -M mymodule generates a .te/.pp from the logged denials; semodule -i mymodule.pp installs it. Read the generated rules first. audit2allow will happily grant whatever was denied, including denials caused by a compromise or a wrong label you should have fixed instead. It is the tool of last resort, not the first.

  8. On the lab box: audit2why, per-domain permissive, and proving the fix persisted

    Diagnose a saved denial offline, confine permissive mode to one domain, and read back exactly what you changed.

    The Signal9 SELinux labs hand you denials as saved AVC records (/root/case/avc.log) on a box whose kernel runs with SELinux disabled. Nothing is enforced live, so the work is the part that persists: the policy store that an enforcing box loads at boot. Three tools carry that work, and each takes the policy file explicitly when there is no live kernel policy to ask.

    `audit2why` reads an AVC and tells you which branch of the decision tree it is on. Point it at the on-disk policy with -p and at the log with -i.

    Example — not runnable on your lab box

    P=$(ls /etc/selinux/targeted/policy/policy.*)
    audit2why -p "$P" -i /root/case/avc.log
    type=AVC msg=audit(...): avc:  denied  { open } for  pid=1526 comm="logrotate" name="app.log" dev="vda1" ino=7762 scontext=system_u:system_r:logrotate_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0
    
    	Was caused by:
    		Missing type enforcement (TE) allow rule.
    
    		You can use audit2allow to generate a loadable module to allow this access.
    The verdict maps straight onto the tree. Missing type enforcement (TE) allow rule means no boolean covers it: check the label first (matchpathcon, step 4), and only if that is right is it the audit2allow case, which takes the same -p/-i flags. The boolean … was set incorrectly is the boolean branch, and it names the setsebool -P that fixes it. One of the following booleans lists several: take the one scoped to this access, not a catch-all like httpd_unified. After a fix, re-run it: Unknown - would be allowed by active policy means the policy now permits that record.

    `semanage permissive` is the scoped version of the permissive probe from step 2. Instead of setenforce 0, which drops protection for every service, it makes one domain permissive: that process's denials are logged but allowed, and everything else stays enforced. You need the domain (the process type, like squid_t), not the type of its binary on disk (squid_exec_t). matchpathcon on the binary gives you the exec type, and the domain is usually the same name ending in _t.

    Example — not runnable on your lab box

    matchpathcon /usr/sbin/squid
    semanage permissive -a squid_t
    semanage permissive -l | tail -3
    # done diagnosing? put it back:
    semanage permissive -d squid_t
    /usr/sbin/squid	system_u:object_r:squid_exec_t:s0
    Customized Permissive Types
    
    squid_t
    -a adds the domain, -l lists the built-in permissive types followed by your Customized ones, and -d removes it. semanage permissive -a squid_exec_t is accepted without complaint, but no process runs in a file type, so it changes nothing. Read the listing back before you trust it.

    Prove the change persisted. semanage writes every fix (file-context rules, port labels, booleans) to the policy store, and adding -C lists only the local customizations, which is exactly what your fix changed. It's the quickest check that you changed one thing and not three.

    Example — not runnable on your lab box

    semanage fcontext -l -C
    semanage port -l -C
    semanage boolean -l -C
    matchpathcon /srv/shop/index.html
    SELinux fcontext                                   type               Context
    
    /srv/shop(/.*)?                                    all files          system_u:object_r:httpd_sys_content_t:s0
    SELinux Port Type              Proto    Port Number
    
    http_port_t                    tcp      8691
    SELinux boolean                State  Default Description
    
    httpd_enable_homedirs          (on   ,   on)  Allow httpd to enable homedirs
    /srv/shop/index.html	system_u:object_r:httpd_sys_content_t:s0
    Each listing is empty until you customize something, so an extra line means an extra change (a second port, a /srv(/.*)? rule that also labels the directory next door). matchpathcon asks the policy what label a path will get, which is what restorecon applies, so you can check a file-context rule before relabeling anything. An equivalence rule (semanage fcontext -a -e /var/www /srv/shop) is another correct way to give a new web root the /var/www labels, and it shows up in the same -l -C listing.
Recap
  • The SELinux signature is DAC-says-yes-app-says-no; confirm it by reproducing under permissive (setenforce 0), which also logs the AVC.
  • Read the AVC with ausearch -m avc: scontext/tcontext types and the permission in braces tell you exactly what was denied to whom.
  • Walk the tree in order — wrong label (restorecon/semanage fcontext) → boolean (setsebool -P) → port (semanage port) → genuine gap (audit2allow, last resort).
  • Prefer semanage fcontext + restorecon over chcon, and always setsebool -P, so fixes survive a relabel and a reboot.