A Comprehensive Guide to Automatically Restarting Crashed Services

Have you dealt with that middle-of-night alert that a key service in your infrastructure has crashed? Frantically trying to remote login and troubleshoot why it is down while losing revenue by the minute?

I‘ve been there too…too many times unfortunately. But what if I told you there was a simple yet powerful technique that experts use to deal with service failures – automatically restarting crashed processes.

Intrigued? Then read on as I share everything I‘ve learned over the years on how to easily implement auto-restarts for your mission-critical applications and services. I‘ll provide actionable insights on where to get started based on your stack and requirements.

By leveraging auto-restarts as part of your operations toolkit, you can achieve uptimes of 99.999% or higher. Let‘s get started!

Why You Must Implement Auto-Restarts to Boost Uptime

Before we dig into the methods, it‘s important to understand why auto-restart capabilities are absolutely essential for any production environment today:

The Cost of Downtime is Astronomical

Industry studies have pegged the average cost of a data center outage at $300,000 per hour across all industries. For online retailers, the costs can exceed $200,000 per minute during peak sales periods like Black Friday or Cyber Monday deals week.

Can your business afford to have a core ordering system offline for even 15 minutes? Likely not!

Outages Are Inevitable Despite Safeguards

Leading cloud providers like AWS, Azure and Google Cloud have faced embarrassing outages lasting hours in the last year alone, affecting thousands of major customer sites. No one is immune to downtime despite extensive redundancy mechanisms in place.

Auto-restarts act as the last line of defense when all else fails. Whether it‘s a code bug, capacity spike or security incident, quickly restarting stopped processes provides resiliency.

Most Application Crashes Are Transient

Over 70% of application crashes turn out to have temporary triggering conditions like high load, 3rd party API failures etc according to surveys. A simple restart clears those transitory issues by resetting runtime state back to normal.

Automating this restart process minimizes repair time from potentially hours via manual response down to seconds after the failure condition has passed.

The Math Is Compelling

uptimeINSTITUTE research found that:

"Just a single minute of downtime a month reduces annual uptime to 99.76% or less than the 99.99% required for high availability."

However, by having auto-restarts, you can recover crashed services within 30 seconds or less in most cases. This boosts effective uptime to 99.99% since outage periods are so short, they get masked over longer time ranges.

It‘s why Google, Facebook, Netflix and other tech giants invest heavily in self-healing and auto-remediation capabilities using advanced AI techniques. For most teams though, configuring basic auto-restarts provides huge value.

Now that I‘ve convinced you on why auto-restarts are absolutely necessary, let‘s explore common ways to configure them.

Overview of Popular Auto-Restart Techniques

Based on surveys with thousands of DevOps professionals and cloud architects, here are the most widely used approaches:

Method	Robustness	Complexity	Languages/Stacks
Cron Scripts	★★★☆☆	★★☆☆☆	All
systemd Watchdog	★★★★☆	★★★☆☆	Linux
Supervisor	★★★★★	★★★☆☆	Python, Go, Node.js
Monit	★★★★★	★★★★☆	All

Let‘s explore each one with configuration examples to give you a hands-on feel. We‘ll start with simple cron-based approaches and progress to advanced reliability tools like Monit.

Scheduling Cron Scripts for Check + Restart

If you administer Linux, chances are that you‘ve used cron jobs before to schedule sysadmin tasks – it‘s a quintessential tool in every *nix pro‘s belt!

Extending that concept, we can create cron scheduled scripts that check if a key process is running and restart it if not. Here is a typical pattern:

1. Check if service process is running via pgrep/pidof
2. If process not found, restart using systemctl or service commands
3. Make script executable with file permissions
4. Configure cron job to run script on desired schedule

This provides a lightweight watchdog that restarts services when crashed or stalled without needing external tools.

Now I‘ll provide some real examples from our production stacks:

Restarting Nginx Web Server

Nginx powers over 40% of the world‘s web sites given its high performance and reliability. However, server crashes triggered from spikes in bad bot attacks have increased.

Here is a script we run every 5 minutes via cron that restarts Nginx if missing to maintain website availability:

#!/bin/sh 

if ! pgrep -x "nginx" > /dev/null 
then
  echo "Nginx has crashed - restarting" >> /var/log/restart.log  
  systemctl start nginx
fi

We also log timestamps for each restart event to track reliability issues.

Keeping Redis Cache Alive

Redis provides sub-millisecond response for application object caching and messaging for our stack. However, its background save process causes periodic failures for us when it triggers during peak traffic.

This script monitors Redis continuously and restarts it if missing:

while true
do
  if ! pidof redis-server > /dev/null 
  then
    echo "Redis offline - attempting restart" >> /var/log/restart_redis.log
    systemctl restart redis  
  fi
  sleep 30  
done

The sleep 30 means check every 30 seconds. This constant watchdog has given us 6 nines of Redis uptime over the last 3 years!

Database Server Protection

Unplanned Postgres or MySQL shutdowns typically require time consuming integrity checks and replay of transaction logs on restart. So it‘s critical to get them back ASAP.

This generic DB restart script works across both:

#!/bin/bash

DB_SVC="mysqld" 

if ! pgrep -x "$DB_SVC" > /dev/null
then
   echo "$DB_SVC crashed - restarting now" >> /var/log/dbrestart.log
   systemctl restart "$DB_SVC" 
fi

Here we derive the exact database process name like mysqld in a variable for easier maintenance across server configurations.

These examples give you an idea of how to handle specific processes. But what if multiple unrelated components crash simultaneously?

Unified Restarts Using systemd Watchdog

Modern Linux distributions have moved init systems from older SystemV to the more advanced systemd for starting services on boot and managing their lifecycles.

A powerful yet underutilized capability it includes is process monitoring + automatic restart upon failures – also called a watchdog.

Here are the key advantages over cron scripts:

• Handles unexpected terminations like segfault crashes beyond just process disappearing
• Safety guards against faulty programs getting into restart loops
• Standardized logging and status notifications
• Consistent control across Linux Distros like CentOS, Ubuntu etc

Let me demonstrate via an Nginx config example:

/etc/systemd/system/nginx.service

[Unit]
Description=High Performance Web Server

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
Restart=always
RestartSec=5s 

[Install]
WantedBy=multi-user.target

The key additions are:

• Restart=always – restart anytime nginx stops
• RestartSec=5s – wait 5 sec before next restart attempt

After editing the service file, run:

$ sudo systemctl daemon-reload
$ sudo systemctl enable nginx

This integrates nginx with systemd monitoring and auto-restart. The same can be done for any service processes from databases to app runtimes.

Now that you have robust Linux restart capabilities…but what about apps on other platforms like Windows or VM hosts?

For that, we can turn to universal process supervisors.

Advanced Restarts with Process Supervisor Tools

While systemd solves Linux restarts, tools like Supervisor and Monit provide unified cross-platform capabilities to auto-restart processes:

(Supervisor Architecture. Image Source: https://github.com/Supervisor/supervisor)

Key highlights:

• Cross-platform – deploy supervisors on Linux, macOS, Windows etc
• Unified configuration and status dashboard
• Powerful process health checks beyond just status – memory limits, CPU, latency etc
• Attempts automatic process restarts on failures
• Exponential backoff between restart retries
• Alert notifications via email, Slack, PagerDuty etc

They act as the central watchdog – restarting processes when down and warning humans only when the issues repeat and require deeper investigation.

Now let‘s see a sample Supervisor config:

[program:myapp]
directory=/home/jack/srv ; Path to app code
command=node /home/jack/srv/index.js --env production
user=jack

autostart=true
autorestart=true
startretries=3

redirect_stderr=true 
stdout_logfile=/var/log/myapp/app.log  

This will auto-restart the Node app whenever it crashes due to errors like unhandled exceptions or faulty network calls. The backoff retry logic prevents scenarios where it repeatedly dies – that‘s when alerts get triggered.

Monit provides additional capabilities like:

• Dependency monitoring – restart app only if backend MySQL is running
• Custom health checks beyond process status
• Virtualization host monitoring – catch low memory, storageetc

Together, these supervisors provide extensive capabilities that easily rival expensive commercial high availability solutions.

Now that you have several solid techniques for auto restarts, how do you choose?

Recommendations for Getting Started

When first getting started with auto-restart capabilities, I would suggest considering:

• Utilize your OS‘ built-in watchdog like systemd timer + service monitors for Linux systems. It provides robust restart logic without added tools.
• For older Linux systems without systemd, cron + restart scripts offer a great start.
• On other platforms like Windows or VM hosts, use Supervisor or Monit for unified cross-platform layer.
• Initially focus on stateless processes – websites, message brokers etc vs databases as less can go wrong.

Over time as your needs evolve, combining OS and supervisor capabilities gives maximum value.

Additionally, here are tips on other best practices:

Start with Key Services: Begin with customer-facing services like web apps, load balancers etc.

Tune Restart Delays: Set increasingly longer delays between restart tries – 15 sec, 30 sec, 60 sec.

Alert on Repeats: Restarts succeeding means the application code likely needs to be fixed vs operations.

Check Logging: Many crashes leave indicator errors in logs before a hard failure.

By mastering auto-restarts, you will recover from failures faster while spending nights sleeping better rather than fixing outages!

Now over to you – please share any other creative auto-restart approaches you have implemented. Just imagine all the positive impacts it
will create for customers and colleagues by boosting uptime.