Server Hygiene

Periodic health and hygiene review for servers and VPS instances. A calm, repeatable ritual for detecting drift, bloat, and silent degradation before they become incidents.

Mindset: Server hygiene embodies /pb-design-rules thinking: Robustness (catch degradation before failure), Transparency (make server state visible and explainable), and Simplicity (predictable cleanups beat clever automation). Apply /pb-preamble thinking to challenge assumptions about what’s “probably fine.”

Resource Hint: sonnet (procedural, well-defined scope)

This is not firefighting. This is the periodic physical exam that prevents the emergency room visit.

When to Use This Command

• Monthly hygiene pass - Routine review of a running server
• Quarterly full audit - Deep drift analysis and capacity planning
• After a period of neglect - Server hasn’t been reviewed in months
• Before scaling or migration - Understand current state before changes
• Post-incident verification - Confirm the server is clean after recovery
• Onboarding to an inherited server - Build a mental model of what’s running

Quick Reference

Execution Flow

Phase 1: SNAPSHOT ──► Phase 2: HEALTH ──► Phase 3: DRIFT ──► Phase 4: CLEANUP ──► Phase 5: READINESS
  (inventory)         (signals)           (bloat detection)   (safe actions)       (future-proof)
       └── Weekly: phases 2-3 only ──┘
       └── Monthly: phases 1-4 ───────────────────────────┘
       └── Quarterly: all phases ──────────────────────────────────────────────────────────────────┘

Phase 1: Snapshot Reality

Goal: know exactly what the server is today. If you can’t explain the server in 5 minutes, it’s already drifting.

Server Inventory

# System identity
hostname && uname -a
head -4 /etc/os-release
uptime

# Resources
nproc && free -h && df -h

Application Footprint

Configuration Sources

Deliverable: A short server manifest. Write it down - even a few bullet points in a markdown file beats nothing.

Phase 2: Health Signals

Goal: detect slow degradation before users feel it.

Resource Trends

Look at trends, not just current values. A server at 60% disk today that was at 40% last month is a problem. Compare with your previous server manifest - if you don’t have one, record today’s numbers. That’s where trends start.

# Disk usage by mount
df -h

# Largest directories
du -sh /* 2>/dev/null | sort -hr | head -10

# Memory with swap
free -h

# CPU load (1, 5, 15 min averages)
uptime

# Disk IO wait (if iostat available)
iostat -x 1 3 2>/dev/null

Thresholds:

Process Health

# Long-running processes sorted by memory
ps aux --sort=-%mem | head -15

# Zombie processes
ps aux | awk '$8 ~ /Z/ {print}'

# Failed systemd units
systemctl --failed

# OOM killer history
dmesg | grep -i "out of memory" | tail -5
journalctl -k | grep -i "oom" | tail -5

Ask: Is anything slowly leaking memory? Are there zombie processes? Has the OOM killer fired recently?

Application Health

System Health

# Kernel warnings
dmesg --level=err,warn | tail -10

# Time sync
timedatectl status | grep "synchronized"

# Pending security updates (Debian/Ubuntu)
apt list --upgradable 2>/dev/null | grep -i security

Rule of thumb: If something spikes periodically, find out why. If something slowly rises, that’s a leak or accumulation.

Phase 3: Drift and Bloat Detection

This is where most server rot happens. Things quietly accumulate until one day the disk is full or a forgotten service gets exploited.

Disk Bloat

# Log sizes
du -sh /var/log/ /var/log/journal/

# Docker waste
docker system df
docker images -f "dangling=true" -q | wc -l
docker volume ls -f "dangling=true" -q | wc -l

# Old build artifacts, temp files, core dumps
find /tmp -type f -mtime +30 | head -20
find / -name "core" -type f 2>/dev/null | head -5

Service Bloat

Config Drift

• Hand-edited config files with no source of truth
• Inconsistent environment variables across applications
• One-off fixes never documented (“I’ll remember why I changed this”)
• Secrets duplicated in multiple places

Ask: Could you rebuild this server’s configuration from version control alone? If not, what’s missing?

Security Drift

# Users with shell access
grep -v "nologin\|false" /etc/passwd

# SSH keys - do you recognize all of them?
for user_home in /home/*/; do
  [ -f "$user_home.ssh/authorized_keys" ] && echo "=== $(basename $user_home) ===" && cat "$user_home.ssh/authorized_keys"
done

# Packages not updated recently
apt list --upgradable 2>/dev/null | wc -l

# TLS certificate expiry
openssl s_client -connect localhost:443 -servername $(hostname) </dev/null 2>/dev/null | openssl x509 -noout -dates

Deliverable: Two lists: “safe to remove now” and “needs planning before removal.”

Phase 4: Hygiene Actions

Golden rule: no “clever” changes during hygiene. Predictable beats smart. Only safe, reversible actions during routine reviews.

Safe Cleanups

Inspect before acting. Review output, then confirm.

# Rotate and prune journal logs
journalctl --vacuum-time=30d
journalctl --vacuum-size=500M

# Show removable packages, then clean
apt --dry-run autoremove
apt autoremove && apt clean

# Show what Docker would prune (images, containers, build cache)
docker system prune --dry-run
docker system prune

Requires judgment - these can destroy data if containers are temporarily stopped:

# Review temp files before deleting
find /tmp -type f -mtime +30 | head -20
# Only delete after reviewing: find /tmp -type f -mtime +30 -delete

# List unused volumes - verify none belong to stopped services you intend to restart
docker volume ls -f "dangling=true"
# Only prune after reviewing: docker volume prune

Stability Improvements

Performance Tuning

Only if measurements justify it. Don’t tune what you haven’t measured.

Phase 5: Future Readiness

This is where the ritual pays off long-term.

Backup Verification

The question is not “do you have backups” but “can you restore them.”

For comprehensive backup and recovery planning, see /pb-dr.

Monitoring Coverage

• Resource metrics (CPU, RAM, disk) - collected and retained
• Application error rates - visible and trended
• Uptime checks - external, not self-reported
• Log visibility - searchable, not just stored
• Alerts - fire when needed, reach someone who can act

For monitoring design guidance, see /pb-observability.

Scaling Headroom

• Current capacity: How much headroom before hitting limits?
• First bottleneck: What resource runs out first?
• Single points of failure: What has no redundancy?
• Growth trajectory: At current growth rate, when do you hit limits?

Disaster Questions

Answer honestly:

1. How long to rebuild this server from scratch?
2. What steps are manual vs automated?
3. What secrets would block recovery if lost?
4. Who else knows how this server works?

If rebuild takes more than a few hours, the system is fragile. See /pb-dr for disaster recovery planning.

Server Manifest Template

Maintain a living document per server. Even a few lines beats nothing.

# Server: [hostname]

**Provider:** [e.g., DigitalOcean, Hetzner, AWS]
**Size:** [CPU, RAM, disk]
**OS:** [distro and version]
**Last review:** [date]

## Services Running
- [service 1] - [purpose] - [deployment method]
- [service 2] - [purpose] - [deployment method]

## Access
- SSH: [who has keys]
- Firewall: [ports open]

## Backups
- [what, where, how often, last tested]

## Known Issues
- [things to watch or fix next time]

Quick Commands

Red Flags

Signs the server needs a hygiene pass now:

• “We’ll deal with it when it becomes a problem”
• Deploys are getting slower with no code changes
• Memory usage “mysteriously” grows between deploys
• Nobody knows what’s safe to delete
• A restart broke something that was working
• Last backup test was “never”

Related Commands

• /pb-maintenance - Strategic maintenance patterns and thinking triggers
• /pb-hardening - Initial server security setup (run before first deploy)
• /pb-dr - Disaster recovery planning and testing
• /pb-sre-practices - Toil reduction, error budgets, operational culture
• /pb-observability - Monitoring and alerting design

Last Updated: 2026-02-07 Version: 1.0.0

Production systems accumulate entropy. This ritual is how you pay down the interest before it compounds.