Mastering Auto-Started Services on Reboot for Linux Servers

Initializing critical services automatically on server boot is crucial for Linux administrators. As enterprises continue adopting Linux for a growing range of workloads, understanding init systems and how to configure auto-start services is an essential skill. This in-depth guide will cover best practices for all major distributions.

The Evolving Linux Landscape

Over 31% of IT infrastructure now runs on Linux according to the 2021 State of Infrastructure Report. Usage spans a range of server types from web hosting and databases to enterprise applications and cloud-native microservices.

Handling service restarts across such heterogeneous Linux environments is a challening task for admins. Complicating management is the variety of init systems that have emerged over time:

  • System V (SysV) – Historically one of the first and widely used init systems for distributions like RHEL/CentOS 6, SLES 11, etc.
  • Upstart – Developed by Canonical as an alternative to SysV for Ubuntu 9.10 to 15.04.
  • Systemd – Modern native system and service manager for Linux. Adopted by RHEL 7, CentOS 7, Ubuntu 15.04+, Debian 9+, openSUSE 15+, and more.

To streamline administration, tools were developed in each ecosystem for configuring automatic service initialization on boot:

Init System Auto-Start Tool First Released
SysV chkconfig 1997
Upstart update-rc.d 2009
systemd systemctl 2010

Now let‘s explore the best practices for leveraging these tools across our most common Linux distributions and versions…

Auto-Start Services on RHEL and CentOS 6.x with chkconfig

The preferred mechanism for enabling services on older Red Hat Enterprise Linux and CentOS distributions is the chkconfig utility. This is part of the sysv-utils package and integrates with SysV init to manage scripts in /etc/init.d/.

For example, to have Apache HTTPD server start automatically on boot:

  1. Install httpd package if not already present
  2. Verify httpd script was added to /etc/init.d/
  3. Run chkconfig --add httpd
  4. Use chkconfig httpd on to enable for runlevels 2, 3, 4, and 5
  5. Check status with chkconfig --list httpd

Here we can see httpd will start on boot in multi-user, non-graphical/graphical, and rescue target modes:

httpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Of the still nearly 20% of servers running RHEL/CentOS 6, the vast majority rely on chkconfig for service initialization. However, upstream development of chkconfig ended in 2015 with the focus shifting to systemd.

Common chkconfig Usage Issues

When managing services with chkconfig, be aware of some potential gotchas:

  • The $network paramater – Some scripts expect networking enabled and can fail otherwise.
  • Permission problems – Scripts may fail to execute if wrong Linux permissions set.
  • Override runlevels – Double check that off states haven‘t been set for the target runlevels.

Consult the chkconfig man pages for advanced usage and sysv-rc compatibility details.

Leveraging systemd Service Units on RHEL and CentOS 7/8

The newest Red Hat Enterprise Linux versions (and related community CentOS builds) have fully adopted systemd as the init system and service manager. This requires a different approach to auto-start configuration.

Rather than monolithic init scripts, systemd uses modular unit files to control services. These unit files end in .service and reside in /usr/lib/systemd/system/ or /etc/systemd/system/:

php-fpm.service
mariadb.service 
nginx.service

To enable a unit across boots:

  1. Install associated server package (PHP, MySQL, Nginx, etc)
  2. Run systemctl enable nginx.service
  3. Check status with systemctl is-enabled nginx.service

This takes advantage of systemd‘s pre-defined dependency tree to ensure services start and stop in the necessary order.

Additional capabilities include custom directives in unit files, advanced logging, on-demand activation rather than always running daemons, and much more. Systemd provides a powerful framework for total Linux system management well beyond service initialization.

Optimizing Systemd Boot Performance

Due to the greater complexity, systemd boot times can lag behind more minimalist init systems. Utilize these optimizations to speed up initialization:

  • Set services to simple process tracking unless notify required
  • Leverage systemd-analyze to identify slow services
  • Increase DefaultTimeoutStartSec as makes sense
  • Enable early microcode and networking units

Confirm improvements with systemd-analyze time. Target under 300ms for typical web/app servers.

Auto-Starting on Ubuntu with Upstart and SysV

Ubuntu Linux has evolved its own init ecosystem separate from upstream systemd. Earlier versions used Upstart which was later replaced with systemd as well:

Ubuntu Version Init System Auto-Start Tool
9.10 – 15.04 Upstart update-rc.d
15.04 – Current Systemd systemctl

So Ubuntu servers can present hybrid environments. Modern hosts will follow the systemd guidance above, however older may need Upstart‘s update-rc.d instead.

For example, to enable the Nginx web service under Upstart:

  1. Install nginx package
  2. Copy init scripts from /etc/init/nginx.conf to /etc/init.d/nginx
  3. Run:
update-rc.d nginx defaults

This generates symlinks for auto-start. Test with a system reboot.

Common mistakes involve incorrect script headers causing immediate exit on Upstart execution. Refer to documentation like the Upstart Cookbook for troubleshooting.

Now that we‘ve covered techniques across the major distros, let‘s discuss some universal best practices as well…

General Tips for Auto-Started Services

While the commands differ between init systems, these guidelines apply broadly:

  • Favor automatic start only for essential, non-interactive services. Too many can slow boot.
  • Prefer later enablement rather than directly at install time.
  • Closely track logs on first initialization to catch issues.
  • Systematically disable unused services over time.
  • Utilize config management systems like Ansible, Salt, and Chef to further automate where possible.

Getting the right mix of services to load on boot and optimizing their ordering is more art than science. Continue monitoring your environments over time and adjusting as needs evolve.

The Future of Linux Initialization

Looking ahead, Linux developers continue innovating new solutions to the decades old boot problem. For example, systemd itself brings capabilities like:

  • systemd-networkd – For traditional and SDN networking configuration.
  • systemd-homed – User home directories mounted on demand.
  • systemd-resolved – Integrated DNS/mDNS resolver.

These components highlight how systemd aims to eventual subsume most core server functionality. Decentralized alternatives like elvish provide a simpler UX targeted towards ops generalists.

Most industry observers expect systemd to continue momentum for the near future. However the needs of cloud, containerization, and next generation app delivery may disrupt the status quo.

Key Takeaways and Resources

Linux services initialization has seen tremendous innovation but retains common administration pain points. Mastering techniques like chkconfig, Upstart, and systemctl will serve administrators well across any environment:

  • RHEL/CentOS 6 – Leverage chkconfig for SysV init scripts in /etc/init.d/.
  • RHEL/CentOS 7/8 – Manage *.service unit files with systemctl under systemd model.
  • Ubuntu – Hybrid Upstart and systemd usage depending on Ubuntu release.

Optimizing boot speed, failover handling, and automation should remain priorities too. As Linux continues growth into the exascale era, administrators have an exciting road ahead!

For further reading check out the documentation on systemd, chkconfig, Upstart, and Linux boot process fundamentals.

I hope this guide has demystified Linux service initialization for you across our major distributions. Please reach out in the comments with any questions or additional best practices!

Tags: