$ Linux One Liners
Back to problems

problem hub

Read-only first

systemd service failed

Read service state, result fields, current-boot logs, restart loops, and unit drop-ins before restarting or editing a service.

Safest first command

systemctl status app-worker --no-pager --lines=50

Before you run this

Expected output: Service state, Result, exit status, recent journal lines, and unit file hints for the failed service.

When not to use it: Do not restart or reset-failed before capturing the failure state and current-boot log lines. The first failure can disappear under later restart attempts.

Expected output example

app-worker.service - Background job worker
   Active: failed (Result: exit-code)
  Process: 2081 ExecStart=/srv/app/bin/worker (code=exited, status=1/FAILURE)
worker[2081]: ERROR redis connection refused at 127.0.0.1:6379

How to read the result

Start with Active and Result, then look for the first application or systemd line that explains why the unit failed. A restart counter is not the cause; it is evidence that the service kept failing.

read the result

What to check next

Active is failed and Result is exit-code

Means: The service command exited unsuccessfully.

Next step: Read the current-boot journal for the first application error before restarting.

Read Current-Boot Logs for One Service

Restart counter keeps increasing

Means: systemd is retrying the same failing command.

Next step: Build a restart-loop timeline and find the first useful failure line.

Build a Restart Loop Timeline

Status points at a user, path, or ExecStart problem

Means: The effective unit or a drop-in may differ from the file you expected.

Next step: Inspect the unit and drop-ins together before editing.

Inspect the Unit File and Drop-ins Together

systemd service failure decision tree

Capture the failed state before changing it. If status names an exit code, read current-boot logs for the first application error. If restarts keep repeating, build a timeline so the first failure is not buried by later retries. If status mentions users, paths, environment files, or ExecStart, inspect the effective unit and drop-ins before editing anything.

  1. systemctl status app-worker --no-pager --lines=50
  2. journalctl -u app-worker -b --no-pager -n 80
  3. systemctl show app-worker --property=Result,ExecMainCode,ExecMainStatus,NRestarts --no-pager
  4. journalctl -u app-worker -b --no-pager -o short-iso | grep -E 'Started|Failed|Scheduled restart|Main process exited'
  5. systemctl cat app-worker

Inspect dependencies before restart

When the first useful log line points at a dependency, prove that dependency from the same host. For connection refused, inspect the target listener. For permission or path errors, inspect ownership and working directory. For environment failures, compare systemctl cat with the files the unit actually loads.

  1. systemctl cat app-worker
  2. systemctl show app-worker --property=User,Group,WorkingDirectory,EnvironmentFiles --no-pager
  3. ss -ltnp
  4. namei -l /srv/app/bin/worker

Bad fixes to avoid

Do not restart repeatedly, clear failed state too early, edit unit files from memory, chmod application trees, or delete logs before you have the first failure line and the current effective unit. Do not disable restart policies just to quiet the alert unless you also capture the service owner and rollback plan.

Common causes

  • Application config or dependency failure
  • Wrong service user or missing working directory
  • Bad ExecStart path or environment file
  • Restart loop hiding the first failure
  • Drop-in override changing runtime behavior

What not to change yet

  • Do not run systemctl restart until the first failure line is captured.
  • Do not run systemctl reset-failed as diagnosis.
  • Do not edit unit files before checking systemctl cat.
  • Do not paste logs publicly without redacting secrets and hostnames.

Stop and escalate if

  • The next step could interrupt users, remove data, or lock out access.
  • The output includes secrets, customer data, or private infrastructure details.
  • You cannot explain the blast radius of the repair command.

platform notes

Distro and service notes

systemd

Use current-boot logs with -b when old failures would confuse the timeline.

Security

Journal lines can include tokens, paths, usernames, and hostnames; redact before sharing.

Rollback

If you change a unit or drop-in later, keep a copy and run systemctl daemon-reload deliberately.

supporting commands

Command path

Guides and drills