Who’s eating my swap?

Oh noes! Your monitoring just alerted you that a server is running low on swap! How can you find what is using the most swap space? Well top will tell you what is using the most memory, but not which process is using the most swap.

And with things like docker and kubernetes implementing cgroups to limit the RAM usage of things, you might have plenty of free memory and still run into swap issues. (protip: disable swap if using kubernetes). So how can you find out? Like this:

for file in /proc/*/status
do
  echo -n "$file "
  awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file
done | sort -k 3 -n -r | less

This will dig through all the processes on the system and sort them by VmSwap usage, making it easy to find the pid and process binary eating all your swap space.

Testing network performance: iperf3

Tools like ping and traceroute are good to see if a connection is up, and tcpdump can help diagnose packet issues, but how can you test the raw speed of a network?

Well, you could just dump /dev/zero over netcat and monitor the speed.

[root@localhost ~]# cat /dev/zero | pv -r | nc 192.168.0.120 5201
[ 223MiB/s]

But this is fairly limited in what it can test and doesn’t provide a lot of options. That’s where iperf comes to the rescue! Just install from epel on both sides and then start up one side as a server.

# # iperf3 -s 
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

Then go fire up a client with a couple threads and test the speed. (these numbers are between two Virtual Box VMs).

# iperf3 -c 192.168.0.120 -t 60 -P 2 -i0
Connecting to host 192.168.0.120, port 5201
[ 4] local 192.168.0.150 port 51074 connected to 192.168.0.120 port 5201
[ 6] local 192.168.0.150 port 51076 connected to 192.168.0.120 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-60.00 sec 6.60 GBytes 945 Mbits/sec 26338 188 KBytes 
[ 6] 0.00-60.00 sec 3.53 GBytes 506 Mbits/sec 34629 1.41 KBytes 
[SUM] 0.00-60.00 sec 10.1 GBytes 1.45 Gbits/sec 60967 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 6.60 GBytes 945 Mbits/sec 26338 sender
[ 4] 0.00-60.00 sec 6.60 GBytes 945 Mbits/sec receiver
[ 6] 0.00-60.00 sec 3.53 GBytes 506 Mbits/sec 34629 sender
[ 6] 0.00-60.00 sec 3.53 GBytes 506 Mbits/sec receiver
[SUM] 0.00-60.00 sec 10.1 GBytes 1.45 Gbits/sec 60967 sender
[SUM] 0.00-60.00 sec 10.1 GBytes 1.45 Gbits/sec receiver

iperf Done.

So what are all them thar options for?

-c run as a client
-t 60 run the test for 60 seconds
-P 2 run two parallel threads. Generally this should equal your CPU count or less
-i 0 don’t show per second data, I just want a summary

Bonus feature! iperf can be installed on all major OS platforms allowing testing between OSs. No more “its just slow because Linux” from them silly Windows guys!

There is also a good article on CertDepot on using iperf.

Stuck NFS mount

So I had an issue at work, and found an amazing answer on StackOverflow. This post is just to copy that info so I can find it later, and help you find it easier.

Sometimes networks flake out. This can cause Linux to half lose its NFS mount. It’s still mounted, but you can’t read or write to it. And calls to unmount it just timeout and fail.

This seems to be caused by stateful firewalls. After the TCP connection has gone stale, the firewall wont forward the packets anymore and may just be dropping them. The client however won’t time out the connection and start over because it still thinks it has a valid connection. So when the client makes queries it just keeps trying over and over and never getting a reply.

I’ve seen people simply revert to rebooting a system to clear the NFS mount and cleanly mount it again. But there is a better way!

Just add the remote IP to the local system temporarily as a secondary interface. This lets the NFS client get a negative response back to its queries quickly and fail properly.

# ifconfig eth0:fakenfs 192.0.2.55 netmask 255.255.255.255
# umount -f /path/to/nfs
# ifdown eth0:fakenfs

Where 192.0.2.55 is the IP of the NFS server you’ve lost connection to. Adding the IP locally lets the server get a reply from itself and safely unmount. You can then try mounting the NFS server again after removing the fake IP and if your network is happy again it should mount just fine.

Props to Daniel Papasian for the excellent answer.

Git Config

Here is my fancy Git setup to make things pretty and easy. First is the .gitconfig

[user]
	name = Steven Barre
	email = steven@stevenbarre.com
[core]
	editor = vim
[merge]
	tool = vimdiff
[color]
	ui = auto
[alias]
        br = branch
        ci = commit
        co = checkout
        dc = diff --cached
        di = diff
        last = log -1 HEAD
        lr = log --pretty=format:"%C(yellow)%h\\ %ad%Cred%d\\ %Creset%s%Cblue\\ [%cn]" --decorate --date=relative --graph
        ls = log --pretty=format:"%C(yellow)%h\\ %ad%Cred%d\\ %Creset%s%Cblue\\ [%cn]" --decorate --date=short --graph
        la = log --pretty=format:"%C(yellow)%h\\ %ad%Cred%d\\ %Creset%s%Cblue\\ [Committed\\ by:\\ %cn]%Cgreen\\ [Authored\\ by:\\ %an]" --decorate --date=short --graph
        search = grep --break --heading --line-number -P
        st = status
        unstage = reset HEAD --
[push]
        default = current

This gives you a bunch of nice aliases, colorizes the output, and sets VIM as the default editor and merger.

The default push action lets you just say “git push” on a new branch and have it create a matching repo on the remote and set it as the upstream. git-config docs for reference.

Next we’ll add some goodies to .bashrc to give us a pretty prompt.

First, we need to get the extra git shell functions loaded. We do this by linking the contrib file into /etc/profile.d

# sudo ln -s /usr/share/git-core/contrib/completion/git-prompt.sh /etc/profile.d/

Then put this into your .bashrc

# Show colors for branch name and indicators
export GIT_PS1_SHOWCOLORHINTS=1
# Show unstaged (*) and staged (+)
export GIT_PS1_SHOWDIRTYSTATE=1
# Show if something is stashed ($)
export GIT_PS1_SHOWSTASHSTATE=1
# Show if there are untracked files (%)
export GIT_PS1_SHOWUNTRACKEDFILES=1

# User@Host:pwd $
# User@host:pwd (branch) $
export PROMPT_COMMAND='__git_ps1 "\[\033[31;1m\]\u\[\033[0m\]@\[\033[34;1m\]\h\[\033[0m\]:\[\033[33m\]\w\[\033[0m\]" " \$ "'

Here is what it looks like.

Why .bashrc ? Because it’s loaded for all bash shells, not just ones created when logging in. This is important if you are using screen or a GUI terminal.

I’ve created a repo on GitHub to hold all my dotfiles to keep track of things like this.

SystemD and Networking

It turns out that network.target does not mean a fully functional network, just that the basic subsystem is up and running. For services that bind to the universal address 0.0.0.0 this isn’t an issue, but if it explicitly binds to an interface or IP it may fail during boot if the interface or IP isn’t available when it tries to start.

I recently had this problem with Samba. The following was in /etc/samba/smb.conf

[global]
interface = eth1
bind interfaces only = yes

Which caused an error on boot

open_sockets_smbd: No sockets available to bind to.

So what we actually need is network-online.target. This target doesn’t complete until after all configured interfaces have come up. This can make your boot slower by making DHCP a blocking step before some services, but in some cases like this one we need that extra time.

So based on my earlier post about overriding SystemD, we’ll create an override and make smb.service also depend on network-online.target.

# cat /etc/systemd/system/smb.service.d/override.conf
[Unit]
After=network-online.target

Now Samba will start up correctly on a reboot!

Further reading:

• https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
• https://bugzilla.redhat.com/show_bug.cgi?id=1119787

SystemD unit customization

What is the best, most correct way to customize SystemD unit files that come from packages?

Packages will leave their files in /usr/lib/systemd/system . I used to just copy those files into /etc/systemd/system and edit them as needed. But is this the best way?

Yes! You can create override files! If you create a directory like /etc/systemd/system/some-unit.service.d and create files named *.conf in there they will also be ready by SystemD (after a systemctl daemon-reload)

One trick is that to override some things like ExecStart= you need to first declare an empty one to reset it if the Type= isn’t oneshot.

[Service]
ExecStart=
ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND

You can also use systemctl edit some-unit.service to create and edit the file in one go. This will create override.conf for you, so that’s probably a best bet for naming any override files you might create.

I learned this from this excellent answer on AskUbuntu.

RPM Version Comparison

I was doing some research into how RPM compares versions, as it appeared to be more complex than simple semver comparisons. Turns out is super whacky. One of the Puppet authors wrote a blog post to much better explain what’s going on. I’m going to just copy it here so I have my own copy in case the blog ever goes away.

---

Package Naming and Parsing

RPM package names are made up of five parts; the package name, epoch, version, release, and architecture. This format is commonly referred to as the acronym NEVRA. The epoch is not always included; it is assumed to be zero (0) on any packages that lack it explicitly. The format for the whole string is n-e:v-r.a. For my purposes, I was only really concerned with comparing theEVR portion; Puppet knows about package names and the bug herein was with what Puppet calls the “version” (EVR in yum/rpm parlance). Parsing is pretty simple:

• If there is a : in the string, everything before it is the epoch. If not, the epoch is zero.
• If there is a - in the remaining string, everything before the first - is the version, and everything after it is the release. If there isn’t one, the release is considered null/nill/None/whatever.

How Yum Compares EVR

Once the package string is parsed into its EVR components, yum calls rpmUtils.miscutils.compareEVR(), which does some data type massaging for the inputs, and then calls out to rpm.labelCompare() (found in rpm.git/python/header-py.c).labelCompare() sets each epoch to “0” if it was null/Nonem, and then uses compare_values() to compare each EVR portion, which in turn falls through to a function called rpmvercmp() (see below). The algorithm for labelCompare() is as follows:

1. Set each epoch value to 0 if it’s null/None.
2. Compare the epoch values using compare_values(). If they’re not equal, return that result, else move on to the next portion (version). The logic within compare_values() is that if one is empty/null and the other is not, the non-empty one is greater, and that ends the comparison. If neither of them is empty/not present, compare them using rpmvercmp() and follow the same logic; if one is “greater” (newer) than the other, that’s the end result of the comparison. Otherwise, move on to the next component (version).
3. Compare the versions using the same logic.
4. Compare the releases using the same logic.
5. If all of the components are “equal”, the packages are the same.

The real magic, obviously, happens in rpmvercmp(), the rpm library function to compare two versions (or epochs, or releases). That’s also where the madness happens.

How RPM Compares Version Parts

RPM is written in C. Converting all of the buffer and pointer processing for these strings over to Ruby was quite a pain. That being said, I didn’t make this up, this is actually the algorithm that rpmvercmp() (lib/rpmvercmp.c) uses to compare version “parts” (epoch, version, release). This function returns 0 if the strings are equal, 1 if a (the first string argument) is newer than b (the second string argument), or -1 if a is older than b. Also keep in mind that this uses pointers in C, so it works by removing a sequence of 0 or more characters from the front of each string, comparing them, and then repeating for the remaining characters in each string until something is unequal, or a string reaches its end.

1. If the strings are binary equal (a == b), they’re equal, return 0.
2. Loop over the strings, left-to-right.
   1. Trim anything that’s not [A-Za-z0-9] or tilde (~) from the front of both strings.
   2. If both strings start with a tilde, discard it and move on to the next character.
   3. If string a starts with a tilde and string b does not, return -1 (string a is older); and the inverse if string b starts with a tilde and string a does not.
   4. End the loop if either string has reached zero length.
   5. If the first character of a is a digit, pop the leading chunk of continuous digits from each string (which may be ” for b if only one a starts with digits). If a begins with a letter, do the same for leading letters.
   6. If the segement from b had 0 length, return 1 if the segment from a was numeric, or -1 if it was alphabetic. The logical result of this is that if a begins with numbers and b does not, a is newer (return 1). If a begins with letters and b does not, then a is older (return -1). If the leading character(s) from a and b were both numbers or both letters, continue on.
   7. If the leading segments were both numeric, discard any leading zeros and whichever one is longer wins. If a is longer than b (without leading zeroes), return 1, and vice-versa. If they’re of the same length, continue on.
   8. Compare the leading segments with strcmp() (or <=> in Ruby). If that returns a non-zero value, then return that value. Else continue to the next iteration of the loop.
3. If the loop ended (nothing has been returned yet, either both strings are totally the same or they’re the same up to the end of one of them, like with “1.2.3” and “1.2.3b”), then the longest wins – if what’s left of a is longer than what’s left of b, return 1. Vice-versa for if what’s left of b is longer than what’s left of a. And finally, if what’s left of them is the same length, return 0.

---

I also found a GitHub repo for a pure Python implementation of this, instead of loading in the C library to python. Here is the main code (again, just copying to make sure I have my own copy)

#
# Copyright (c) SAS Institute Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

from __future__ import print_function
from __future__ import unicode_literals
import re


class Vercmp(object):
    R_NONALNUMTILDE = re.compile(br"^([^a-zA-Z0-9~]*)(.*)$")
    R_NUM = re.compile(br"^([\d]+)(.*)$")
    R_ALPHA = re.compile(br"^([a-zA-Z]+)(.*)$")

    @classmethod
    def compare(cls, first, second):
        first = first.encode("ascii", "ignore")
        second = second.encode("ascii", "ignore")
        while first or second:
            m1 = cls.R_NONALNUMTILDE.match(first)
            m2 = cls.R_NONALNUMTILDE.match(second)
            m1_head, first = m1.group(1), m1.group(2)
            m2_head, second = m2.group(1), m2.group(2)
            if m1_head or m2_head:
                # Ignore junk at the beginning
                continue

            # handle the tilde separator, it sorts before everything else
            if first.startswith(b'~'):
                if not second.startswith(b'~'):
                    return -1
                first, second = first[1:], second[1:]
                continue
            if second.startswith(b'~'):
                return 1

            # If we ran to the end of either, we are finished with the loop
            if not first or not second:
                break

            # grab first completely alpha or completely numeric segment
            m1 = cls.R_NUM.match(first)
            if m1:
                m2 = cls.R_NUM.match(second)
                if not m2:
                    # numeric segments are always newer than alpha segments
                    return 1
                isnum = True
            else:
                m1 = cls.R_ALPHA.match(first)
                m2 = cls.R_ALPHA.match(second)
                isnum = False

            if not m1:
                # this cannot happen, as we previously tested to make sure that
                # the first string has a non-null segment
                return -1  # arbitrary
            if not m2:
                return 1 if isnum else -1

            m1_head, first = m1.group(1), m1.group(2)
            m2_head, second = m2.group(1), m2.group(2)

            if isnum:
                # throw away any leading zeros - it's a number, right?
                m1_head = m1_head.lstrip(b'0')
                m2_head = m2_head.lstrip(b'0')

                # whichever number has more digits wins
                m1hlen = len(m1_head)
                m2hlen = len(m2_head)
                if m1hlen < m2hlen:                     return -1                 if m1hlen > m2hlen:
                    return 1

            # Same number of chars
            if m1_head < m2_head:                 return -1             if m1_head > m2_head:
                return 1
            # Both segments equal
            continue

        m1len = len(first)
        m2len = len(second)
        if m1len == m2len == 0:
            return 0
        if m1len != 0:
            return 1
        return -1


def vercmp(first, second):
    return Vercmp.compare(first, second)

Systemd dependencies

Do you want to make sure two separate services are connected in some way? That one service is up when the other is or even starts first? Thanks to systemd this is now super easy!

In the dark old days of SysV (/etc/init.d) services were started serially during boot and individually on demand. You could adjust the start and stop order by editing a special comment in the init file. For example

# chkconfig: - 85 15

would have the service start at priority 85 and stop at priority 15. Priorities are from 0 to 99 and are done in ascending order. So this would start pretty late in the boot and stop pretty early in the shutdown.

With directives like Requires, Wants, Before, and After you can give systemd more info on what you want. systemd can start things in parallel and will use these hints to build its dependency tree and start or stop everything in the right order.

Packages will install their systemd config files into /usr/lib/systemd/system/ . Don’t edit these files. If you want to make changes, copy them to /etc/systemd/system/ .

I’ll use mariadb and httpd as examples here. It makes sense that we want the database to be up and running before Apache so that we don’t serve broken pages. Lets explore our options and see how they each act under different circumstances.

Requires

This effectively makes one unit cascade the starting of another. If the one is enabled, the other is also effectively enabled. Both units will be started in parallel, but if one of the required units fails to start, the requiring unit will get stopped again.

We’ll add Requires=mariadb.service to the [Unit] section of /etc/systemd/system/httpd.service .

When we issue a start of httpd, mariadb is started in parallel.

When we issue a stop of httpd, mariadb keeps running.

If we break the mariadb config to prevent it from starting, then start httpd, it starts, but shows an error. I think despite what the documentation says, this is just a race condition since both are started in parallel. We’ll get into ordering in a bit.

If when both are running, we stop mariadb, httpd will stop automatically.

Wants

This is just a weaker version of Requires and is the recommended option. If the wanted service fails to start, the wanting service will still start.

When we issue a start of httpd, mariadb is started in parallel.

When we issue a stop of httpd, mariadb keeps running.

If we break the mariadb config to prevent it from starting, then start httpd, it starts with no error.

If when both are running, we stop mariadb, httpd will keep running.

After

Now we can get to some ordering! We’ll tell httpd to start After mariadb.

After=mariadb.service

On its own, it doesn’t enforce the starting of mariadb, just that if both happen to be starting at the same time, mariadb should be started first. And if you are stopping both at the same time, mariadb should be stopped last.

systemctl start httpd mariadb

will start them in order.

Before is just the inverse relationship. You could instead specify in the config for mariadb to start before httpd.

If we combine Requires and After we get what we want. httpd forcing mariadb to start first and fail to start if mariadb fails.

When we issue a start of httpd, mariadb is started first.

When we issue a stop of httpd, mariadb keeps running.

If we break the mariadb config to prevent it from starting, then start httpd, it will fail to start.

If when both are running, we stop mariadb, httpd will stop first.

More info:

Understanding Systemd Units and Unit Files

systemd.unit(5)

PXE Boot Kickstart

Normally to do a kickstart install, you need to first boot off an install disk (usually the NetBoot one) and then enter some extra params on the boot line. But what if you want a more automated process or don’t want to have to deal with disks? PXE boot to the rescue!

PXE boot is a process of having a computer fire up its network card, get an IP from DHCP, then pull down a bootable file.

TFTP

First we need to get a TFTP server set up, preferably on the same server as our installation source.

yum install tftp-server xinetd

Then we need to enable the TFTP server, as it comes disabled by default.

sed -i '/disable/s/yes/no/' /etc/xinetd.d/tftp

Allow TFTP through the firewall

firewall-cmd --add-service=tftp
firewall-cmd --permanent --add-service=tftp

tftp-server is a sub-service of the good old xinetd, so enable and start that.

systemctl start xinetd.service
systemctl enable xinetd.service

PXE Files

Next we need to get some PXE bootable image, preferably one with a menu system. Thankfully there already exists one! syslinux is a tool for installing a bootloader onto a FAT filesystem disk, but it comes with all the bits we need to boot from the network.

yum install syslinux

All the goodies we want are in /usr/share/syslinux/. We just need the pxelinux.0 boot image, and a menu displayer vesamenu.c32. Copy these to our TFTP server.

cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot/

Now we need to config the PXE menu. pxelinux.0 will look in pxelinux.cfg/ for a file called default. Let’s create that file.

mkdir /var/lib/tftpboot/pxelinux.cfg
vi /var/lib/tftpboot/pxelinux.cfg/default

Here is what we are going to put in it

# Use the ncurses menu
default menu.c32

# Show the prompt
prompt 0

# Don't timeout
timeout 0

menu title PXE Install CentOS

# Install CentOS 7
label centos7
 menu label Install CentOS 7
 # Use this as the default
 menu default
 kernel CentOS-7.2.1511/vmlinuz
 ipappend 2
 append initrd=CentOS-7.2.1511/initrd.img inst.ks=http://10.0.2.10/7.ks

menu separator

label rescue7
 menu label Rescure CentOS 7
 kernel CentOS-7.2.1511/vmlinuz
 ipappend 2
 append initrd=CentOS-7.2.1511/initrd.img inst.repo=http://10.0.2.10/CentOS-7.2.1511/ inst.lang=en_US.UTF-8 inst.keymap=us inst.rescue
# Boot the local disk
label local
 menu label Boot from local drive
 localboot 0xffff

Now we’ll copy in the kernel and init ramdisk from a install disk.

cp /media/CentOS-7.2.1511/images/pxeboot/vmlinuz /var/lib/tftpboot/CentOS-7.2.1511/
cp /media/CentOS-7.2.1511/images/pxeboot/initrd.img /var/lib/tftpboot/CentOS-7.2.1511/

DHCP Server

We need is a DHCP server to hand out an IP and pass along some pxeboot options. Your network might already have one, in which case you can simply configure it to point to your PXE server.

Let’s install and configure a DHCP server on our install server.

yum install  dhcp

Put something like the following into /etc/dhcp/dhcpd.conf

subnet 10.0.2.0 netmask 255.255.255.0 {
 option routers 10.0.2.1;
 range 10.0.2.100 10.0.2.200;
 next-server 10.0.2.10;
 filename "pxelinux.0";
}

Enable and start the DHCP server

systemctl enable dhcpd
systemctl start dhcpd

Showtime!

Now boot another server on the same network, and poke the appropriate BIOS button to get it to PXE boot. A sweet menu will show up and let you install CentOS with a single click!

References

• RHEL7 Manual – Network Install Server
• SysLinux Menus

VirtualBox Addon Installation in CentOS

OK, so you have a minimal install of CentOS in VirtualBox and you want to install the Guest Addons to help your VM keeps its clock correct, and share folders with your host OS. What are the dependencies? How do you get this to install?

These directions work for CentOS 5, 6 and 7.

Install some packages needed to build the kernel module.

yum install epel-release
yum install dkms gcc make bzip2 perl
yum install kernel-devel-$(uname -r)

We need EPEL to get dkms

We install dkms to ensure that the addons are rebuild when we upgrade the kernel in the future.

We do the fancy bit with the uname to ensure we get the right version of kernel-devel for the currently running kernel. If you’ve updated the kernel since booting you should reboot before installing kernel-devel and the addons or else you might have some problems building them.

Mount the Guest Addons CD and run the installer.

mkdir /mnt/cdrom
mount /dev/sr0 /mnt/cdrom/
/mnt/cdrom/VBoxLinuxAdditions.run

Answer yes a few times,  and you are done!

Virtualbox Guest Addons Manual

CentOS VirtualBox HowTo