User Resource Limits: ulimit and pam_limits

Intermediate25 min5 steps

What you'll learn

Explain soft versus hard limits, and who may change each
Read the current limits of a shell with ulimit, and of a running process via /proc
Write persistent pam_limits rules for a user and for a group in /etc/security/limits.d/
Verify limits from a fresh login session, including users the rule must not affect
Know when limits.conf doesn't apply (systemd services) and what to use instead

The mental model: a ceiling and a starting point
Understand what a resource limit is, and the difference between soft and hard, before touching any file.
Every process on Linux runs with a set of resource limits: how many files it may have open at once, how many processes its user may run, how large a core dump may be, how much memory it may lock, and more. The kernel enforces them, so a runaway program hits a wall instead of taking the whole machine down with it.
Each limit comes as a pair, and the pair is the whole idea:
- The soft limit is the value a process actually lives under right now.
- The hard limit is the ceiling. An ordinary user may raise their own soft limit up to the hard limit, or lower either one, but never raise the hard limit. Only root can do that.
Picture a hotel thermostat. The guest can turn the room temperature up or down (soft limit), but management has set a maximum the dial physically won't pass (hard limit). Guests get flexibility; the building stays safe.
Limits are inherited: a child process starts with its parent's limits. That's why they're applied at the very start of a login session, so that the shell and everything started from it carries them.
Why it matters
Two incidents every sysadmin meets sooner or later: a database or web server that fails under load with "Too many open files" because the default open-files limit is too low, and a script stuck in a fork loop that starts processes until the box stops responding. The first needs a limit raised for one account; the second needs a ceiling for a group of users. Both are LFCS tasks.
Check yourself
A user's soft open-files limit is 1024 and the hard limit is 4096. Can they run ulimit -n 3000 in their own shell?
Reading the limits you have
Inspect your current soft and hard limits with ulimit.
ulimit is a shell builtin that reads and sets the limits of the current shell. On its own it's terse, so you'll mostly use it with a letter for the resource and -S or -H to pick soft or hard. The two you'll use most are -n (open files, the kernel's RLIMIT_NOFILE) and -u (processes per user, RLIMIT_NPROC).
```
$ ulimit -Sn; ulimit -Hn
```
```
1024
10240
```
-S
Show or set the soft limit.
-H
Show or set the hard limit.
-n
The open-files limit (file descriptors).
The first number is your soft open-files limit and the second is the hard ceiling. Container and distro defaults vary a lot, so don't be surprised if yours differ from someone else's. What matters is the relationship: soft is never above hard.
```
$ ulimit -a | head -8
```
```
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 64300
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
```
-a
Print all current (soft) limits with their option letters.
-a lists every limit for the current shell with its letter in brackets, so you can look up the flag for a resource you don't use often. unlimited means the kernel imposes no ceiling for that resource.
```
$ ulimit -Sn 512 && ulimit -Sn
```
```
512
```
-Sn 512
Set the soft open-files limit for this shell to 512.
Lowering your own soft limit always works and takes effect immediately, but only for this shell and the processes it starts. Open a new login and it's back to the default. That's the point of the next step: making a limit persistent for an account.
Note
ulimit only ever shows the limits of the shell you typed it in. To see the limits of a running service, read /proc/PID/limits for its process ID. It lists soft and hard values side by side for every resource.
Check yourself
You set ulimit -Sn 512 in one terminal. Does a second terminal you already had open get the new limit?
Making limits persistent with pam_limits
Write limits.conf rules for a user and for a group.
Persistent per-account limits are applied by a PAM module, pam_limits, when a session opens: at console login, over SSH, and with su - or sudo. It reads /etc/security/limits.conf and then every *.conf file in /etc/security/limits.d/, in alphabetical order. Putting your rules in a drop-in file under limits.d keeps them separate from the distribution's file, so package upgrades never conflict with your changes.
Each rule is one line of four fields: domain, type, item, value.
```
$ cat /etc/security/limits.conf | grep -E '^#<domain>|^#\*|^#@' | head -5
```
```
#<domain>        <type>  <item>  <value>
#<domain> can be:
#<domain>      <type>  <item>         <value>
#*               soft    core            0
#*               hard    rss             10000
```
The stock file is all comments, and those comments are the best quick reference you have. <domain> can be a user name, @group for every member of a group, or * for everyone. The type is soft, hard, or - for both at once.
```
$ mkdir -p /etc/security/limits.d && printf 'webapp  soft  nofile  2048\nwebapp  hard  nofile  4096\n@interns  hard  nproc  30\n' > /etc/security/limits.d/50-demo.conf && cat /etc/security/limits.d/50-demo.conf
```
```
webapp  soft  nofile  2048
webapp  hard  nofile  4096
@interns  hard  nproc  30
```
nofile
Maximum number of open file descriptors.
nproc
Maximum number of processes the user may run.
@group
Apply the rule to every member of that group.
Three rules in one drop-in. The service account webapp starts every session with 2048 open files and may raise that to 4096, no further. Every member of the group interns gets a process ceiling of 30 they can't raise. Note the @: without it, the rule would apply to a user literally named interns.
Soft forgotten, @ forgotten
Two slips account for most broken limits files. First, setting only hard nofile 4096 and expecting sessions to get 4096: they still start at the default soft value, and the application has to raise it itself. Set soft too, or use - for both. Second, writing interns instead of @interns, which silently matches nobody.
Warning
Be careful with * rules. A low * hard nproc also applies to service accounts and can stop daemons from starting new workers. Target the user or group that actually needs the limit, and leave root alone unless you mean it.
Check yourself
Which line gives every member of the group devs a hard limit of 100 processes?
Verifying from a fresh session
Prove the limits apply where they should and nowhere else.
Because pam_limits runs when a session opens, editing the file changes nothing for sessions that already exist, including your own shell. The only honest test is a brand-new login session as the target user. su - USER opens a real login session through PAM, so it's exactly what you need. As root it doesn't ask for a password.
```
$ id webapp >/dev/null 2>&1 || useradd -m -s /bin/bash webapp; su - webapp -c 'ulimit -Sn; ulimit -Hn'
```
```
2048
4096
```
su - USER
Start a login session as USER, running the full PAM session stack.
-c 'CMD'
Run one command in that session and exit.
This creates the demo account if it doesn't exist, then opens a login session as webapp and prints its open-files limits. You should see 2048 and 4096: pam_limits read the drop-in when the session opened.
```
$ getent group interns >/dev/null || groupadd interns; id ana >/dev/null 2>&1 || useradd -m -s /bin/bash -G interns ana; su - ana -c 'ulimit -Hu'
```
```
30
```
-Hu
Show the hard limit on the number of processes.
ana is a member of interns, so the group rule applies and the hard process limit is 30. Run the same check as a user outside the group and you'd see the system default instead. That check matters just as much as the first, because it proves the rule isn't broader than intended.
Tip
Services started by systemd don't go through a PAM login, so limits.conf doesn't apply to them. For a systemd service, set LimitNOFILE= or LimitNPROC= in the unit or in a drop-in made with systemctl edit. Use limits.conf for people and login sessions, and unit directives for daemons.
Check yourself
You added a limits.d rule for webapp, but ulimit -n in your already-open root shell shows no change. Is the rule broken?
Now do it — in your live terminal
Configure and verify a limit yourself on the sandbox to your right.
Time to put it together. Run these in the terminal, predict what each ulimit will print before you press Enter, then compare. The loop of writing the rule, opening a fresh session and checking a user it should not affect is exactly how a limits lab is graded.
```
$ useradd -m -s /bin/bash builder; su - builder -c 'ulimit -Sn; ulimit -Hn'
```
Note builder's default limits before you change anything.
```
$ printf 'builder  -  nofile  2048\n' > /etc/security/limits.d/60-builder.conf && su - builder -c 'ulimit -Sn; ulimit -Hn'
```
With - as the type, both soft and hard become 2048. Both numbers should now match.
```
$ su - builder -c 'ulimit -Hn 4096'
```
As builder, try to go above the hard limit. You should get an error that the operation isn't permitted, which is the ceiling doing its job.

Recap

Every limit is a pair: soft is what a process lives under, hard is the ceiling. Users can move soft up to hard; only root raises hard.
ulimit -S/-H with -n (open files) or -u (processes) reads and sets the current shell's limits, and changes last only for that shell and its children.
Persistent limits live in /etc/security/limits.conf and limits.d/*.conf as domain, type, item, value. Prefix groups with @, and set soft as well as hard.
pam_limits applies the rules when a session opens, so verify with a fresh su - user, and check a user the rule should not touch.
systemd services skip PAM logins. Use LimitNOFILE= or LimitNPROC= in the unit for daemons.

Limits at a glance
Item / command	Meaning
`ulimit -Sn` / `ulimit -Hn`	Soft / hard open-files limit of this shell
`ulimit -Hu`	Hard limit on processes for this user
`/proc/PID/limits`	Soft and hard limits of a running process
`user soft nofile 2048`	A user's sessions start with 2048 open files
`user hard nofile 4096`	...and can never raise past 4096
`@group hard nproc 30`	Every group member capped at 30 processes
`user - core 0`	`-` sets soft and hard together (here: no core dumps)
`LimitNOFILE=` (systemd unit)	The equivalent for a service, which limits.conf doesn't reach

User Resource Limits: ulimit and pam_limits

The mental model: a ceiling and a starting point

Reading the limits you have

Making limits persistent with pam_limits

Verifying from a fresh session

Now do it — in your live terminal