Systemd dependencies

Do you want to make sure two separate services are connected in some way? That one service is up when the other is or even starts first? Thanks to systemd this is now super easy!

In the dark old days of SysV (/etc/init.d) services were started serially during boot and individually on demand. You could adjust the start and stop order by editing a special comment in the init file. For example

# chkconfig: - 85 15

would have the service start at priority 85 and stop at priority 15. Priorities are from 0 to 99 and are done in ascending order. So this would start pretty late in the boot and stop pretty early in the shutdown.

With directives like Requires, Wants, Before, and After you can give systemd more info on what you want. systemd can start things in parallel and will use these hints to build its dependency tree and start or stop everything in the right order.

Packages will install their systemd config files into /usr/lib/systemd/system/ . Don’t edit these files. If you want to make changes, copy them to /etc/systemd/system/ .

I’ll use mariadb and httpd as examples here. It makes sense that we want the database to be up and running before Apache so that we don’t serve broken pages. Lets explore our options and see how they each act under different circumstances.

Requires

This effectively makes one unit cascade the starting of another. If the one is enabled, the other is also effectively enabled. Both units will be started in parallel, but if one of the required units fails to start, the requiring unit will get stopped again.

We’ll add Requires=mariadb.service to the [Unit] section of /etc/systemd/system/httpd.service .

When we issue a start of httpd, mariadb is started in parallel.

When we issue a stop of httpd, mariadb keeps running.

If we break the mariadb config to prevent it from starting, then start httpd, it starts, but shows an error. I think despite what the documentation says, this is just a race condition since both are started in parallel. We’ll get into ordering in a bit.

If when both are running, we stop mariadb, httpd will stop automatically.

Wants

This is just a weaker version of Requires and is the recommended option. If the wanted service fails to start, the wanting service will still start.

When we issue a start of httpd, mariadb is started in parallel.

When we issue a stop of httpd, mariadb keeps running.

If we break the mariadb config to prevent it from starting, then start httpd, it starts with no error.

If when both are running, we stop mariadb, httpd will keep running.

After

Now we can get to some ordering! We’ll tell httpd to start After mariadb.

After=mariadb.service

On its own, it doesn’t enforce the starting of mariadb, just that if both happen to be starting at the same time, mariadb should be started first. And if you are stopping both at the same time, mariadb should be stopped last.

systemctl start httpd mariadb

will start them in order.

Before is just the inverse relationship. You could instead specify in the config for mariadb to start before httpd.

If we combine Requires and After we get what we want. httpd forcing mariadb to start first and fail to start if mariadb fails.

When we issue a start of httpd, mariadb is started first.

When we issue a stop of httpd, mariadb keeps running.

If we break the mariadb config to prevent it from starting, then start httpd, it will fail to start.

If when both are running, we stop mariadb, httpd will stop first.

More info:

Understanding Systemd Units and Unit Files

systemd.unit(5)

Advertisements

PXE Boot Kickstart

Normally to do a kickstart install, you need to first boot off an install disk (usually the NetBoot one) and then enter some extra params on the boot line. But what if you want a more automated process or don’t want to have to deal with disks? PXE boot to the rescue!

PXE boot is a process of having a computer fire up its network card, get an IP from DHCP, then pull down a bootable file.

TFTP

First we need to get a TFTP server set up, preferably on the same server as our installation source.

yum install tftp-server xinetd

Then we need to enable the TFTP server, as it comes disabled by default.

sed -i '/disable/s/yes/no/' /etc/xinetd.d/tftp

Allow TFTP through the firewall

firewall-cmd --add-service=tftp
firewall-cmd --permanent --add-service=tftp

tftp-server is a sub-service of the good old xinetd, so enable and start that.

systemctl start xinetd.service
systemctl enable xinetd.service

PXE Files

Next we need to get some PXE bootable image, preferably one with a menu system. Thankfully there already exists one! syslinux is a tool for installing a bootloader onto a FAT filesystem disk, but it comes with all the bits we need to boot from the network.

yum install syslinux

All the goodies we want are in /usr/share/syslinux/. We just need the pxelinux.0 boot image, and a menu displayer vesamenu.c32. Copy these to our TFTP server.

cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot/

Now we need to config the PXE menu. pxelinux.0 will look in pxelinux.cfg/ for a file called default. Let’s create that file.

mkdir /var/lib/tftpboot/pxelinux.cfg
vi /var/lib/tftpboot/pxelinux.cfg/default

Here is what we are going to put in it

# Use the ncurses menu
default menu.c32

# Show the prompt
prompt 0

# Don't timeout
timeout 0

menu title PXE Install CentOS

# Install CentOS 7
label centos7
 menu label Install CentOS 7
 # Use this as the default
 menu default
 kernel CentOS-7.2.1511/vmlinuz
 ipappend 2
 append initrd=CentOS-7.2.1511/initrd.img inst.ks=http://10.0.2.10/7.ks

menu separator

label rescue7
 menu label Rescure CentOS 7
 kernel CentOS-7.2.1511/vmlinuz
 ipappend 2
 append initrd=CentOS-7.2.1511/initrd.img inst.repo=http://10.0.2.10/CentOS-7.2.1511/ inst.lang=en_US.UTF-8 inst.keymap=us inst.rescue
# Boot the local disk
label local
 menu label Boot from local drive
 localboot 0xffff

Now we’ll copy in the kernel and init ramdisk from a install disk.

cp /media/CentOS-7.2.1511/images/pxeboot/vmlinuz /var/lib/tftpboot/CentOS-7.2.1511/
cp /media/CentOS-7.2.1511/images/pxeboot/initrd.img /var/lib/tftpboot/CentOS-7.2.1511/

DHCP Server

We need is a DHCP server to hand out an IP and pass along some pxeboot options. Your network might already have one, in which case you can simply configure it to point to your PXE server.

Let’s install and configure a DHCP server on our install server.

yum install  dhcp

Put something like the following into /etc/dhcp/dhcpd.conf

subnet 10.0.2.0 netmask 255.255.255.0 {
 option routers 10.0.2.1;
 range 10.0.2.100 10.0.2.200;
 next-server 10.0.2.10;
 filename "pxelinux.0";
}

Enable and start the DHCP server

systemctl enable dhcpd
systemctl start dhcpd

Showtime!

Now boot another server on the same network, and poke the appropriate BIOS button to get it to PXE boot. A sweet menu will show up and let you install CentOS with a single click!

VirtualBox_CS7_19_07_2016_21_24_27

References

Mandatory Packages in Kickstart

When building a %packages section in a kickstart file you may want to remove some packages you don’t need. But they might get installed anyways. It turns out Anaconda requires some packages be installed. So, what are these mandatory packages?

Looking at the code for RHEL7 we can see that some basic packages from Storage, Realm, AuthConfig, and FirewallD need to be included.

https://git.fedorahosted.org/cgit/anaconda.git/tree/pyanaconda/install.py?h=rhel7-branch#n210

    packages = storage.packages + ksdata.realm.packages
    packages += ksdata.authconfig.packages + ksdata.firewall.packages

    if not ksdata.bootloader.disabled:
        packages += storage.bootloader.packages

    if network.is_using_team_device:
        packages.append("teamd")
 

storage.packages comes from Blivet and will include packages needed to configure your storage. Things like lvm2, or mdadm, or device-mapper-multipath. You can dig through the source to see which packages might be included depending on what storage you are choosing to configure.

If you use the kickstart command realm to join an Active Directory domain then the realm package will be installed.

authconfig is simply a required package to configure authentication on the system.

firewalld is also a required package.

The bootloader is usually grub2 and so will install the needed package. If you are using a non-standard bootloader check the source of bootloader.py for what packages you will get.

If you are using network teaming or bonding, then you will get teamd installed.

So be aware of these extra packages not listed in the @core group of packages.

What is a minimal install?

In my post Basics of Kickstart I talked about doing a minimal install. So what is included in the @core group for each major version of CentOS? I’ll document here for the current versions as it’s possible these values will change.

CentOS 5.11

  • Mandatory Packages
    • SysVinit
    • authconfig
    • basesystem
    • bash
    • centos-release
    • coreutils
    • cpio
    • dhclient
    • dhcpv6-client
    • e2fsprogs
    • ed
    • file
    • filesystem
    • glibc
    • hdparm
    • hmaccalc
    • initscripts
    • iproute
    • iputils
    • kbd
    • kudzu
    • libgcc
    • libhugetlbfs
    • libtermcap
    • mkinitrd
    • openssh-server
    • passwd
    • policycoreutils
    • prelink
    • procps
    • readline
    • redhat-logos
    • rootfiles
    • rpm
    • selinux-policy-targeted
    • setools
    • setserial
    • setup
    • shadow-utils
    • sysklogd
    • termcap
    • util-linux
    • vim-minimal
    • yum
  • Default Packages
    • Deployment_Guide-en-US
    • grub
    • sysfsutils
    • udftools

CentOS 6.8

  • Mandatory Packages
    • acl
    • attr
    • audit
    • authconfig
    • basesystem
    • bash
    • coreutils
    • cpio
    • cronie
    • dhclient
    • e2fsprogs
    • filesystem
    • glibc
    • initscripts
    • iproute
    • iptables
    • iptables-ipv6
    • iputils
    • kbd
    • ncurses
    • openssh-server
    • passwd
    • policycoreutils
    • procps
    • rootfiles
    • rpm
    • rsyslog
    • selinux-policy-targeted
    • setup
    • shadow-utils
    • sudo
    • system-config-firewall-base
    • util-linux-ng
    • vim-minimal
    • yum
  • Default Packages
    • aic94xx-firmware
    • atmel-firmware
    • b43-openfwwf
    • bfa-firmware
    • efibootmgr
    • grub
    • ipw2100-firmware
    • ipw2200-firmware
    • ivtv-firmware
    • iwl100-firmware
    • iwl1000-firmware
    • iwl3945-firmware
    • iwl4965-firmware
    • iwl5000-firmware
    • iwl5150-firmware
    • iwl6000-firmware
    • iwl6000g2a-firmware
    • iwl6050-firmware
    • kernel-firmware
    • kexec-tools
    • libertas-usb8388-firmware
    • netxen-firmware
    • postfix
    • ql2100-firmware
    • ql2200-firmware
    • ql23xx-firmware
    • ql2400-firmware
    • ql2500-firmware
    • rdma
    • rt61pci-firmware
    • rt73usb-firmware
    • xorg-x11-drv-ati-firmware
    • zd1211-firmware

CentOS 7.2.1511

  • Mandatory Packages
    • audit
    • basesystem
    • bash
    • biosdevname
    • btrfs-progs
    • coreutils
    • cronie
    • curl
    • dhclient
    • e2fsprogs
    • filesystem
    • firewalld
    • glibc
    • hostname
    • initscripts
    • iproute
    • iprutils
    • iptables
    • iputils
    • irqbalance
    • kbd
    • kexec-tools
    • less
    • man-db
    • ncurses
    • openssh-clients
    • openssh-server
    • parted
    • passwd
    • plymouth
    • policycoreutils
    • procps-ng
    • rootfiles
    • rpm
    • rsyslog
    • selinux-policy-targeted
    • setup
    • shadow-utils
    • sudo
    • systemd
    • tar
    • tuned
    • util-linux
    • vim-minimal
    • xfsprogs
    • yum
  • Default Packages
    • NetworkManager
    • NetworkManager-team
    • NetworkManager-tui
    • aic94xx-firmware
    • alsa-firmware
    • dracut-config-rescue
    • ivtv-firmware
    • iwl100-firmware
    • iwl1000-firmware
    • iwl105-firmware
    • iwl135-firmware
    • iwl2000-firmware
    • iwl2030-firmware
    • iwl3160-firmware
    • iwl3945-firmware
    • iwl4965-firmware
    • iwl5000-firmware
    • iwl5150-firmware
    • iwl6000-firmware
    • iwl6000g2a-firmware
    • iwl6000g2b-firmware
    • iwl6050-firmware
    • iwl7260-firmware
    • iwl7265-firmware
    • kernel-tools
    • libsysfs
    • linux-firmware
    • microcode_ctl
    • postfix
    • rdma

So if you use --nobase in your kickstart file, you will get everything in Mandatory and Default lists. You can choose to exclude some of the defaults with minus (-) to slim up your install though, or use --nodefaults to skip them entirely.

To find these lists yourself, just ask yum! yum groupinfo core

Basics of Kickstart

If you need to install CentOS over and over, one useful thing is to create a kickstart file. This is a text config file that directs the install program and can make an install entirely unattended.

You can find a full reference of all options on Red Hat’s documentation site.23.3. Kickstart Syntax Reference.

Sources

So what do we need? And what are some nice addons?

First, we can say we want to install. This is optional, but encouraged. Followed by a source for the packages. This can be a local media like cdrom or harddisk, or a network share like nfs, or my favorite url.

The URL you specify here should be to the os folder on a mirror and have as a subfolder repodata. This URL will have all the packages needed to install CentOS. You can find a list of mirrors on the CentOS site, or just provide a mirrorlist URL instead. A mirrorlist URL will give YUM a place to fetch a list of mirrors to try and it will attempt to get the fastest one.

You can also specify additional repos for the installer to pull packages from as it sees fit or that you specify. I like to at least include the updates repo, so that we install the latest packages on the first try, and don’t have to do a yum update after the install. Here is our kickstart file so far.

# Do an install
install

# From this hard coded URL
# url --url=http://mirror.its.sfu.ca/mirror/CentOS/7/os/x86_64/
# Or better yet, from a mirrorlist with variables
url --mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
# Extra repos let us install the latest versions
repo --name="Updates" --mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
# Optional if you want packages from EPEL
repo --name="epel" --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-$releasever&arch=$basearch

Install Settings

We can select the kind of display we want from the installer. graphical, text, or cmdline. I choose cmdline as it provides the most helpful debug output. text mode gives you the classic ncurses display.

We can also set what the installer should do after finishing. I find reboot to be the most helpful, but you can also choose halt and poweroff.

Lastly, we’ll disable firstboot. That’s the “helper” you get on the first boot up asking you to make a user and such. Since we are trying to automate things, we don’t want to be bothered.

# Install mode
cmdline

# reboot when finished the install
reboot

# Disable firstboot
firstboot --disabled

System Config

Here we specify some settings for the system we are building. We’ll set the language, keyboard layout, timezone, and SELinux. We’ll also set the default password storage policy to the strongest available.

# System settings
lang en_US.UTF-8
keyboard us
authconfig --enableshadow --passalgo=sha512
selinux --disabled
timezone UTC

Network

Most places I use Linux there is Software Defined Networking (SDN) and it handles all the firewalling, so I just disable it in the system. We also want to turn off IPv6 as its just extra junk we don’t need. And we’ll stick to DHCP here.

network --bootproto dhcp --noipv6
firewall --disabled

Root Password

This just sets the root password. You can grab the hash for an existing user from /etc/shadow or just use a plaintext password.

#rootpw --plaintext mycoolpassword
rootpw --iscrypted $6$BHils6Q1$hTRN8PUTpmQG6y7bkeSPqWrWxCV9uja9EMhsmf5qk4rDhdnKHznYiz5CxBmFqiaO14I7utwu7ToH6y7gMwFeq/

Disk Space

Now we want to specify the disk layout. Do we want basic partitions or LVM? How big should stuff be? I usually go with a 1GB swap, 1GB /tmp and the rest as root disk. I add some safety options to /tmp to make sure evil things don’t try and exec from there.

# Set up the drive
bootloader --location=mbr
zerombr
clearpart --all --initlabel
part swap --asprimary --size=1024
part /tmp --fstype=ext4 --asprimary --size=1024 --fsoptions="defaults,nosuid,noexec"
part / --fstype=ext4 --grow --asprimary --size=100

Package Selection

I like to install the minimal possible and handle the rest with config management. The minimal install uses the @core group and not the @base group. @core includes a lot of packages by default that we probably don’t need. WiFi drivers, RAID card drivers, and junk like that. I’m usually building a image for VM use, so can exclude most of that by putting a minus (-) in front of the name. You can also use an asterisk (*) as a wildcard to match a bunch of packages. There are a few packages from @base I do like to include though, like acpid.

%packages --nobase
acpid
-aic94xx-firmware
-alsa-firmware
-bfa-firmware
-ivtv-firmware
-iwl*-firmware
-rdma
%end

Post Script

After the install is complete, you can run some shell scripts before the reboot to help get your system just right. I make some tweaks to grub and re-install it. Then I import all the RPM keys, so that when I run yum it doesn’t ask about importing them the first time.

%post
# Reduce timeout for faster boot
sed -i 's/GRUB_TIMEOUT=5/GRUB_TIMEOUT=1/' /etc/default/grub
# Set consoles for proper logging and vnc
# Be noisey to help debugging
sed -i 's/GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet"/GRUB_CMDLINE_LINUX="crashkernel=auto console=ttyS0,115200n8 console=tty0"/' /etc/default/grub
# Rebuild grub config
grub2-mkconfig -o /boot/grub2/grub.cfg
# Import all the keys
/bin/rpm --import /etc/pki/rpm-gpg/*
%end

And with all that, we are done a basic kickstart. Be sure to read the docs and customize as you see fit!

yum clean all – will miss stuff

So I learnt that yum clean all won’t delete folders from /var/cache/yum for repos that aren’t currently enabled. This can bloat VM images you are building. So it’s best to just rm -rf /var/cache/yum to make sure you get everything after a yum clean all.

Here is the full set of yum cleaning I do in my image build scripts to make them as small as possible.

yum history new
yum clean all
rm -f /var/lib/rpm/__db*
rm -rf /var/cache/yum
rm -rf /var/lib/yum

Parallel SSH

The knife command can use a lot of memory when doing a search. So knife ssh "name:web-*" can fail after eating all you RAMs and getting OOM killed.

parallel -j 10 -i \
  ssh -o StrictHostKeyChecking=no \
      -o UserKnownHostsFile=/dev/null \
      -i private.key \
      user@{}.example.com \
      'hostname; sudo chef-client;' \
  -- $( knife node list | grep '^web-' )

Different versions of parallel have different options, this was used on CentOS 7.

  • -j number of jobs to run in parallel
  • -i enable use of {} to insert the strings
  • — end of args, next is list of things