My deploy script printed "up to date" and exited, while the new image sat there unused. The check that decided this was one line: docker compose up --dry-run | grep -q 'Recreate'. How can a grep that finds its match make an if take the "no match" branch? The answer is three Bash behaviors that are each reasonable on their own: grep -q stops reading early, the writer then gets SIGPIPE, and set -o pipefail reports that as the pipeline failing.
This post walks through the bug as it happened in the deploy scripts of my self-hosted VPS. It covers a small script that reproduces it, why it only fails some of the time, the fix, and what the fix costs.
The symptom: a deploy that said "up to date"
The script asks Compose what it would do, and skips the rollout if nothing would change. This is the check as it was (shortened from the deploy script):
set -euo pipefail
# ...
if ! docker compose up -d --dry-run "$SERVICE" 2>&1 | grep -qE 'Container .* (Recreate|Create) *$'; then
echo "==> $app is up to date"
exit 0
fi
Dry run mode prints the steps Compose would take. For up, that includes a Container ... Recreate line for every container whose configuration or image changed. A new image tag should produce exactly that line, grep should match, ! should make the condition false, and the deploy should continue.
It printed "up to date" instead, although the new image was a real change that Compose had to recreate a container for. The match happened, and the script still took the branch for "no match".
grep -q stops reading at the first match
The first piece is in the GNU grep manual: -q means "Exit immediately with zero status if any match is found". That is the point of -q: there is nothing to print, so there is no reason to read the rest of the input.
When grep exits, the read end of the pipe closes. If the program on the left of the pipe writes anything after that, the kernel sends it SIGPIPE. The pipe(7) man page puts it plainly: when all read ends are closed, a write "will cause a SIGPIPE signal to be generated for the calling process". The default action for SIGPIPE is to terminate, and Bash reports a process killed by signal N as exit status 128+N. SIGPIPE is 13, so the writer exits with 141.
Without pipefail this does not matter. A pipeline's status is the status of its last command, and the last command is grep, which succeeded.
pipefail turns a successful grep into a failed pipeline
The second piece is the option almost every careful Bash script turns on. From the Bash manual on pipelines: "If pipefail is enabled, the pipeline's return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully."
So with pipefail the pipeline's status is 141: the writer's death by SIGPIPE, not grep's success. Here is the whole chain in a script that fakes the Compose output: one matching line, a short pause, then one more line, the way a real program keeps writing while it works.
set -o pipefail
plan() { echo ' Container app-web-1 Recreate'; sleep 0.1; echo ' Container app-web-1 Recreated'; }
if ! plan 2>&1 | grep -qE 'Container .* (Recreate|Create) *$'; then
echo "==> up to date"
else
echo "==> deploying"
fi
==> up to date
grep matched the first line and exited 0. plan tried to write its second line, got SIGPIPE and exited 141. pipefail made the pipeline's status 141, ! turned that into true, and the script reported that nothing had changed.
Why set -e did not stop the script
I expected set -e to at least crash the script on a failed pipeline, which would have been a loud error instead of a quiet skip. It did not, and the Bash manual for set says why: the shell does not exit if the failing command is "part of the test in an if statement", or "if the command's return status is being inverted with !". This check was both.
That is the part that makes this bug expensive. Outside an if, the same pipeline would have killed the script with exit 141, and CI would have shown a red run. Inside if !, the failure is just a value, and the value happens to point to the harmless-looking branch. A deploy that skips itself looks exactly like a deploy with nothing to do.
It only fails when the writer writes after the match
The bug is a race, which is why a check like this can pass for a long time before it fails. SIGPIPE is only sent if the writer writes after grep has exited. If the writer produces its whole output in one write before grep gets to the match, nobody writes to a closed pipe and the pipeline succeeds. A reproduction with the cases side by side:
set -o pipefail
printf 'Container app-web-1 Recreate\n' | grep -qE 'Recreate *$'; echo "status=$?"
{ echo 'Container app-web-1 Recreate'; sleep 0.2; echo 'Container app-web-1 Recreated'; } | grep -qE 'Recreate *$'
echo "status=$? PIPESTATUS=${PIPESTATUS[*]}"
for i in $(seq 20); do seq 1 200000 | grep -q '^1$'; printf '%s ' $?; done; echo
status=0
status=141 PIPESTATUS=141 0
141 141 141 141 141 141 141 141 141 141 141 141 141 141 141 141 141 141 141 141
- One small write: the data is in the pipe before
grepreads it, the writer has nothing left to write aftergrepexits, so noSIGPIPE: status 0. - A write after the match:
PIPESTATUSshows the writer at 141 andgrepat 0, andpipefailpicks the 141. - Large output:
seqwrites about 1.3 MB, far more than a Linux pipe holds (16 pages, 65,536 bytes on a 4 KiB page system, per pipe(7)).grepmatches on line one and exits whileseqstill has most of its output to write. It failed 20 times out of 20.
A program like docker compose writes progress as it goes, so whether a given run fails depends on timing. That makes it the worst kind of check: correct in a quick test, wrong some of the time in CI.
head -n1 has the same shape. seq 1 200000 | head -n1 exits with status 141 under pipefail for the same reason, so first=$(something | head -n1) with set -e can kill a script at an assignment that looks harmless.
The fix: capture the output, then match it
The fix I applied to the scripts where this had bitten, or plainly could, is to separate the two steps. Run the command to completion and keep its output in a variable, then match the variable:
# (Output captured first: `| grep -q` under pipefail fails on SIGPIPE.)
plan=$(docker compose up -d --dry-run "$SERVICE" 2>&1 || true)
if ! grep -qE 'Container .* (Recreate|Create) *$' <<<"$plan"; then
echo "==> $app is up to date"
exit 0
fi
grep now reads from a here-string, and nothing is left writing to a pipe when it exits early. The || true is deliberate: the dry run's own exit status is not what decides the branch, its output is. If you need both, keep the status in a variable instead of throwing it away.
Where the check is only "does this command print anything", there is no need for grep at all (container name changed):
# before: docker ps -q --filter name='^app$' | grep -q .
if [ -n "$(docker ps -q --filter name='^app$')" ]; then
And where the check is about one known line, a Bash pattern match on the captured text does the same job, like testing that a dump file ends with its completion marker:
[[ $(zcat "$1" | tail -1) == *'Dump completed'* ]]
The zcat | tail pipe is safe here because tail reads all of its input before it writes anything.
Other ways out, and why I did not use them
There are a few other fixes. Each one works, but has a cost I did not want:
| Option | Effect | Why not |
|---|---|---|
grep -E ... >/dev/null (drop -q) |
grep reads all input, so no SIGPIPE |
Works, but the next person to "clean up" the line puts -q back |
set +o pipefail around the check |
Pipeline status is grep's again |
Easy to forget to turn it back on; hides real failures of the writer |
|| true after the pipeline |
Never fails | Also hides the match: the if can no longer tell the cases apart |
Check PIPESTATUS for 141 |
Precise | Hard to read, and every check needs the same few lines |
| Capture, then match | grep reads a string, not a pipe |
Holds the whole output in memory (see below) |
Capturing won because it reads like what it means: get the plan, then look at it. A comment above the deploy checks explains why it is written that way, so it does not get "simplified" back into a pipe.
What this fix does not cover
- Output size. A captured variable holds everything in memory. That is fine for a dry-run plan or a list of container IDs, and wrong for a command that streams gigabytes or never ends (
yes,tail -f). For those, stopping early is the whole point, and you need to handle 141 explicitly instead. - The writer's own failures.
|| truekeeps a failing dry run from aborting the script, so a broken Compose file now shows up as "would recreate nothing" plus whatever it printed to stderr. That trade was acceptable for a skip check. It is not acceptable for a check where an error must stop the deploy. - New code. The fix is a convention, not a rule a tool enforces. Writing this post, I searched the scripts again and found five more
| grep -qunderpipefailthat I had missed, in backup-restore and host-setup checks, each with a writer that prints several lines. I fixed them the same way. Nothing stops the next one from being written. Searching for the pattern (grep -rn '| *grep -q') after enablingpipefailis cheap, and worth doing.
Summary
grep -qexits at the first match. A writer that keeps writing getsSIGPIPEand exits 141.- With
set -o pipefail, that 141 becomes the pipeline's status even thoughgrepsucceeded. - Inside
ifor behind!,set -edoes not stop the script, so the failure silently picks a branch. In my deploy script it picked "up to date". - It is a race: small outputs written at once pass, output written after the match fails. Large outputs fail every time.
head -n1behaves the same way underpipefail.- The fix I use: capture the output in a variable, then match it with
grep <<<"$var",[[ ... == *pattern* ]]or[ -n "$(...)" ], with a comment saying why.
If you've been bitten by pipefail in a different way, or found a linter that catches this pattern, I'd love to compare notes. My email is on the home page.
