xargs, Named Pipes & File Descriptors

Advanced35 min5 steps

What you'll learn

Drive a command over many inputs with xargs, safely (-0) and in parallel (-P)
Place each argument precisely with xargs -I{} instead of appending at the end
Wire two processes together through a named pipe (mkfifo) and explain why the writer blocks
Open, redirect, and close custom file descriptors with exec
Recognise when a plain pipe isn't enough and one of these tools is the right reach

When a plain pipe isn't enough
See the three jobs a single | can't do
A pipe a | b is one of Unix's best ideas: connect one command's output to another's input. But it has limits. It can't turn a list of items into one-command-per-item. It can't run those invocations in parallel. And it only connects two processes in a straight line — it can't fan output to a background producer or open a second private channel.
Three tools cover those gaps: `xargs` builds and runs commands from a list (optionally in parallel), named pipes (mkfifo) wire processes together through the filesystem, and file descriptors (exec) give you extra numbered I/O channels beyond stdin/stdout/stderr.
These are the 'guru' plumbing tools
You can go a long way without them — and then you hit a task where you're processing 10,000 files (needs parallelism), or coordinating a producer and consumer (needs a FIFO), or a tool wants to read config from a fixed descriptor (needs exec). Knowing these exist turns a shell script from a toy into a small concurrent system.
Check yourself
Name one thing a plain 'a | b' pipe cannot do.
xargs: turn a list into commands
Build per-item commands safely and place the argument where you want it
xargs reads items from stdin and runs a command with them as arguments. Two habits make it safe and precise:
- `-0` (with find -print0) separates items by NUL instead of whitespace, so filenames with spaces or newlines survive.
- `-I{}` names a placeholder, so the item lands exactly where you put {} — not just appended at the end. -I{} also runs one command per item.
```
$ printf '%s\n' web1 db1 gw | xargs -I{} echo "host={} status=?"
```
```
host=web1 status=?
host=db1 status=?
host=gw status=?
```
-I{}
placeholder for the item; runs the command once per item
One command per line of input, with the item placed mid-string via {}. Without -I{}, xargs would append all three to a single echo. -I{} is what lets you build a real per-item command instead of a flat argument dump.
```
$ find . -name '*.log' -print0 | xargs -0 -P4 -I{} gzip {}
```
-0
items are NUL-separated — pair with find -print0 for space-safe filenames
-P4
run up to 4 invocations in parallel
The production pattern: find emits NUL-separated paths (-print0), xargs -0 reads them safely (spaces in names are fine), -P4 compresses up to 4 files at once, and -I{} places each path as gzip's argument. On a directory of thousands of logs, -P4 is roughly a 4x speedup over one-at-a-time.
Two more flags earn their keep. `-n N` caps how many items go into each command: -n1 forces exactly one per call even without -I{}, and -n50 batches fifty at a time (fewer processes than one-per-item, still bounded). `-r` (--no-run-if-empty) skips the command entirely when stdin is empty — without it, echo -n '' | xargs rm runs a bare rm and errors. They combine: find . -name '*.tmp' -print0 | xargs -0 -r -n50 rm removes matches fifty per rm and does nothing at all when there are none.
Whitespace and the missing -0
ls | xargs rm is a classic footgun: a file named my report.txt becomes two arguments (my and report.txt) and you delete the wrong things — or everything. The fix is always the pair find ... -print0 | xargs -0. And when you parallelise with -P, remember each job's stdout can interleave — collect results by redirecting xargs' output once, not by each child appending to a shared file.
Check yourself
Which xargs flag lets you place the input item in the MIDDLE of the command instead of at the end?
Named pipes: wire two processes together
Build a producer/consumer with mkfifo and understand the block
A named pipe (or FIFO) is a pipe with a name on the filesystem. mkfifo path creates it; one process writes to it, another reads from it, and the kernel streams bytes between them — just like |, but the two ends can be separate commands you start independently. The catch that trips everyone the first time: a FIFO has no buffer of its own, so opening it for writing blocks until a reader connects (and vice-versa). That's why the producer runs in the background.
```
$ mkfifo /tmp/p
wc -l < /etc/passwd > /tmp/p &
read count < /tmp/p
echo "passwd has $count lines"
rm /tmp/p
```
```
passwd has 34 lines
```
mkfifo path
create a named pipe on the filesystem
cmd > fifo &
producer writes into the pipe, backgrounded so it can't deadlock
read x < fifo
consumer connects and unblocks the writer
mkfifo makes the pipe. The producer (wc -l) writes its result INTO the pipe, backgrounded with & — because the write blocks until someone reads. The consumer (read) then connects and unblocks it, receiving the count. Foreground the producer instead of backgrounding it and the script deadlocks: the writer waits for a reader that never gets to run.
The deadlock: forgetting the &
Run the producer in the foreground (wc -l < file > /tmp/p with no &) and the script hangs forever: opening the FIFO for writing blocks until a reader attaches, but your reader line never executes because the shell is stuck on the writer. Background the producer with &, then read. And clean up the FIFO (rm /tmp/p) so a re-run doesn't hit 'file exists'.
Check yourself
Why must the process that writes into a named pipe usually be backgrounded with &?
File descriptors: extra I/O channels with exec
Open, use, and close a numbered descriptor beyond 0/1/2
Every process starts with three descriptors: 0 stdin, 1 stdout, 2 stderr. You already redirect them (>, 2>, <). But you can open more — descriptor 3, 4, … — with `exec`, and keep them open across many commands. That's how you write to a logfile and the terminal at once, or hold a file open for a whole loop instead of re-opening it each iteration.
```
$ exec 3> /tmp/audit.log
echo "run started" >&3
echo "visible on screen"
echo "run finished" >&3
exec 3>&-
cat /tmp/audit.log
```
```
visible on screen
run started
run finished
```
exec 3> file
open file descriptor 3 for writing to a file, kept open
>&3
redirect this command's output to fd 3
exec 3>&-
close file descriptor 3
exec 3> opens fd 3 to a file and leaves it open. Now >&3 sends a line to the log while normal echo still goes to the screen — two independent channels. exec 3>&- closes fd 3 when done. The final cat shows both audit lines landed in the file, in order, without re-opening it each time.
Why a custom fd beats reopening
Appending to a log with >> file inside a tight loop reopens the file every iteration. Open it once with exec 3>> file, write with >&3 all loop long, and close it at the end — fewer syscalls and a single, consistent handle. Custom descriptors also let a tool that expects input on a specific fd (some interactive programs, coprocesses) get exactly the channel it wants.
Descriptors work for input too. exec 3< file opens fd 3 for reading, and read -u 3 line pulls one line from it — while stdin (fd 0) stays free for something else, like a prompt. This is the fix for a classic trap: a while read line; do ...; done < file loop can't also read a y/n answer from the keyboard, because the loop already owns stdin. Move the file onto fd 3 and the keyboard stays on fd 0.
```
$ printf 'one\ntwo\n' > /tmp/f
exec 3< /tmp/f
read -u 3 a
read -u 3 b
echo "$a then $b"
exec 3<&-
rm /tmp/f
```
```
one then two
```
exec 3< file
open file descriptor 3 for reading from a file, kept open
read -u 3
read one line from fd 3 (not stdin)
exec 3<&-
close the reading file descriptor 3
exec 3< opens the file on fd 3 for reading and keeps it open, so two successive read -u 3 calls advance through the file line by line — first 'one', then 'two' — without stdin ever being touched. exec 3<&- closes the reading end. Swap read -u 3 for a plain read and it would block on the keyboard instead.
Check yourself
What do exec 3> file and exec 3>&- do, respectively?
Now do it — in your live terminal
Parallelise with xargs, pass data through a FIFO, and log via a custom fd
Exercise all three: a parallel map with xargs -P, a producer/consumer through a FIFO, and a run-log on descriptor 3. These are the exact moves the xargs+FIFO and file-descriptor labs will grade.
```
$ printf '%s\n' alpha bravo charlie delta | xargs -P4 -I{} sh -c 'echo "{} has ${#1} chars" ' _ {}
```
-P4 -I{}
up to 4 parallel workers, one per item at {}
A parallel map: four items, up to four workers, each printing the item and its length. Wrapping the work in sh -c '...' _ {} lets you compute per-item (here ${#1}, the length). Run it, then change -P4 to -P1 and notice the ordering settle — proof the parallelism was real.
```
$ mkfifo /tmp/q
( seq 1 5 | paste -sd+ | bc ) > /tmp/q &
read total < /tmp/q
echo "sum 1..5 = $total"
rm /tmp/q
```
( ... ) > fifo &
backgrounded producer writing into the named pipe
A producer/consumer: the backgrounded subshell computes 1+2+3+4+5 and writes the result into the FIFO; the reader picks it up. Remove the & and watch it deadlock (then Ctrl-C and put it back) — the fastest way to feel why the writer must be backgrounded.

Recap

xargs turns a list into commands: -I{} places each item precisely and runs one command per item; -P N runs N in parallel; -0 with find -print0 is space-safe.
A named pipe (mkfifo) wires two independently-started processes together; it has no buffer, so the writer blocks until a reader connects — background the producer with & or it deadlocks.
exec opens custom file descriptors that stay open across commands: exec 3> file to write (>&3) or exec 3< file to read (read -u 3), closed with exec 3>&- / 3<&- — cheaper than reopening, and reading on fd 3 keeps stdin free for a prompt.
Beyond -I{}/-P/-0, xargs -n N caps items per command and -r (--no-run-if-empty) skips the command entirely on empty input — so a batch delete does nothing when nothing matches.
Reach for these when a plain pipe can't: per-item/parallel work (xargs), producer/consumer coordination (FIFO), or a second private I/O channel (custom fd).
The recurring hazards are whitespace (always find -print0 | xargs -0) and the FIFO deadlock (background the writer, clean up the pipe).

Concept → syntax
Concept	Syntax
Per-item command	`... \| xargs -I{} cmd {}`
Parallel / space-safe	`find . -print0 \| xargs -0 -P4 -I{} cmd {}`
Create a named pipe	`mkfifo /tmp/p`
Producer / consumer	`cmd > /tmp/p &` · `read x < /tmp/p`
Batch / skip-if-empty	`... \| xargs -r -n50 cmd`
Open / write / close fd	`exec 3> file` · `>&3` · `exec 3>&-`
Read through a fd	`exec 3< file` · `read -u 3 x` · `exec 3<&-`

xargs, Named Pipes & File Descriptors

When a plain pipe isn't enough

xargs: turn a list into commands

Named pipes: wire two processes together

File descriptors: extra I/O channels with exec

Now do it — in your live terminal