Server fails to start after Centos 7 update

The Problem

HTTPD failed after yum updated Centos 7.

SSH also failed.

There were over 400 updates or installations in over 700 steps during the update. A few that caught my eye just now in reviewing the log:

Updated: centos-release-7-4.1708.el7.centos.x86_64
Updated: firewalld-filesystem-0.4.4.4-6.el7.noarch
Updated: iptables-1.4.21-18.0.1.el7.centos.x86_64
Updated: 1:NetworkManager-libnm-1.8.0-9.el7.x86_64

Updated: device-mapper-persistent-data-0.7.0-0.1.rc6.el7.x86_64

Updated: initscripts-9.49.39-1.el7.x86_64

Updated: cronie-anacron-1.4.11-17.el7.x86_64

Installed: 1:NetworkManager-1.8.0-9.el7.x86_64
Installed: 1:NetworkManager-ppp-1.8.0-9.el7.x86_64

Updated: cloud-init-0.7.9-9.el7.centos.2.x86_64

Installed: 1:grub2-2.02-0.64.el7.centos.x86_64

Updated: 1:NetworkManager-tui-1.8.0-9.el7.x86_64
Updated: 1:NetworkManager-team-1.8.0-9.el7.x86_64

Installed: kernel-3.10.0-693.2.2.el7.x86_64

Fortunately, I was able to access the server through an emergency console via the Rackspace cloud server interface. The emergency console never let me down all through this process. I'm so grateful for that small mercy!

The boot.log showed the heart of the matter:

Starting LSB: Bring up/down netorking

Failed to start LSB: Bring up/down netowrking

It said to run systemctl status network.service for more details. The screen capture below shows the output from that command.

network.service failed to bring up networking because /etc/sysconfig/network-scripts/ifcfg-eth0 was changed from "BOOTPROTO static" to "BOOTPROTO dhcp". The crucial piece of information in the screen capture is that "Determining IP information for eth0... failed."

Sometime during the update, the configuration for eth0 was changed from static to dhcp. This is an old copy of ifcfg-eth0 from a server that I crashed in May of 2016. It tells eth0 that my IP address is "166.78.150.236". There is no way dhcp could figure that out by itself. It is a precious bit of information that must be fed into the system on boot.

# Automatically generated, do not edit

# Label public
DEVICE=eth0
BOOTPROTO=static
HWADDR=bc:76:4e:05:75:f4
IPADDR=166.78.150.236
NETMASK=255.255.255.0
DEFROUTE=yes
GATEWAY=166.78.150.1
IPV6INIT=yes
IPV6_AUTOCONF=no
IPV6ADDR=2001:4800:7812:0514:7cbc:4d9b:ff05:75f4/64
IPV6_DEFAULTGW=fe80::def%eth0
DNS1=72.3.128.241
DNS2=72.3.128.240
ONBOOT=yes
NM_CONTROLLED=no

I didn't save a copy of the mangled file. The crucial change was BOOTPROTO=dhcp. No IPADDR or GATEWAY or DNS servers were configured.

servicectl status network.service

network.service - LSB: Bring up/down networking

failed

No DHCPOFFERS received.

Determining IP information for eth0... failed

I think I may have had both network.service and NetworkManager running. I made a mess out of the data before I understood this distinction.

I eventually decided to disable network.service and do my best to learn how NetworkManager works. It seems to be what RedHat wants.

Failed to start The Apache HTTP Server

Route info failed

Unexpected error

I had set both eth0 and eth1 as "defroute."

The Solutions

use nmtui to examine eth1 and configure eth0

- "Network configuration using sysconfig files."

- nmcli connection reload

- nmcli dev disconnect interface-name

- nmcli con up interface-name

start httpd after NetworkManager configures network

The final problem was that

#!/bin/sh
# This is a NetworkManager dispatcher script to turn http on
# when the network is ready.  MXM
# /etc/NetworkManager/dispatcher.d/22-httpd
# https://wiki.archlinux.org/index.php/NetworkManager#Network_services_with_NetworkManager_dispatcher

if [ "$2" = "up" ]; then
	systemctl start httpd
fi

if [ "$2" = "down" ]; then
	systemctl stop httpd
fi

exit 0