Testing network performance: iperf3

Tools like ping and traceroute are good to see if a connection is up, and tcpdump can help diagnose packet issues, but how can you test the raw speed of a network?

Well, you could just dump /dev/zero over netcat and monitor the speed.

[root@localhost ~]# cat /dev/zero | pv -r | nc 5201
[ 223MiB/s]

But this is fairly limited in what it can test and doesn’t provide a lot of options. That’s where iperf comes to the rescue! Just install from epel on both sides and then start up one side as a server.

# # iperf3 -s 
Server listening on 5201

Then go fire up a client with a couple threads and test the speed. (these numbers are between two Virtual Box VMs).

# iperf3 -c -t 60 -P 2 -i0
Connecting to host, port 5201
[ 4] local port 51074 connected to port 5201
[ 6] local port 51076 connected to port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-60.00 sec 6.60 GBytes 945 Mbits/sec 26338 188 KBytes 
[ 6] 0.00-60.00 sec 3.53 GBytes 506 Mbits/sec 34629 1.41 KBytes 
[SUM] 0.00-60.00 sec 10.1 GBytes 1.45 Gbits/sec 60967 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 6.60 GBytes 945 Mbits/sec 26338 sender
[ 4] 0.00-60.00 sec 6.60 GBytes 945 Mbits/sec receiver
[ 6] 0.00-60.00 sec 3.53 GBytes 506 Mbits/sec 34629 sender
[ 6] 0.00-60.00 sec 3.53 GBytes 506 Mbits/sec receiver
[SUM] 0.00-60.00 sec 10.1 GBytes 1.45 Gbits/sec 60967 sender
[SUM] 0.00-60.00 sec 10.1 GBytes 1.45 Gbits/sec receiver

iperf Done.

So what are all them thar options for?

-c run as a client
-t 60 run the test for 60 seconds
-P 2 run two parallel threads. Generally this should equal your CPU count or less
-i 0 don’t show per second data, I just want a summary

Bonus feature! iperf can be installed on all major OS platforms allowing testing between OSs. No more “its just slow because Linux” from them silly Windows guys!

There is also a good article on CertDepot on using iperf.


Stuck NFS mount

So I had an issue at work, and found an amazing answer on StackOverflow. This post is just to copy that info so I can find it later, and help you find it easier.

Sometimes networks flake out. This can cause Linux to half lose its NFS mount. It’s still mounted, but you can’t read or write to it. And calls to unmount it just timeout and fail.

This seems to be caused by stateful firewalls. After the TCP connection has gone stale, the firewall wont forward the packets anymore and may just be dropping them. The client however won’t time out the connection and start over because it still thinks it has a valid connection. So when the client makes queries it just keeps trying over and over and never getting a reply.

I’ve seen people simply revert to rebooting a system to clear the NFS mount and cleanly mount it again. But there is a better way!

Just add the remote IP to the local system temporarily as a secondary interface. This lets the NFS client get a negative response back to its queries quickly and fail properly.

# ifconfig eth0:fakenfs netmask
# umount -f /path/to/nfs
# ifdown eth0:fakenfs

Where is the IP of the NFS server you’ve lost connection to. Adding the IP locally lets the server get a reply from itself and safely unmount. You can then try mounting the NFS server again after removing the fake IP and if your network is happy again it should mount just fine.

Props to Daniel Papasian for the excellent answer.

SystemD and Networking

It turns out that network.target does not mean a fully functional network, just that the basic subsystem is up and running. For services that bind to the universal address this isn’t an issue, but if it explicitly binds to an interface or IP it may fail during boot if the interface or IP isn’t available when it tries to start.

I recently had this problem with Samba. The following was in /etc/samba/smb.conf

interface = eth1
bind interfaces only = yes

Which caused an error on boot

open_sockets_smbd: No sockets available to bind to.

So what we actually need is network-online.target. This target doesn’t complete until after all configured interfaces have come up. This can make your boot slower by making DHCP a blocking step before some services, but in some cases like this one we need that extra time.

So based on my earlier post about overriding SystemD, we’ll create an override and make smb.service also depend on network-online.target.

# cat /etc/systemd/system/smb.service.d/override.conf

Now Samba will start up correctly on a reboot!

Further reading:

SystemD unit customization

What is the best, most correct way to customize SystemD unit files that come from packages?

Packages will leave their files in /usr/lib/systemd/system . I used to just copy those files into /etc/systemd/system and edit them as needed. But is this the best way?

Yes! You can create override files! If you create a directory like /etc/systemd/system/some-unit.service.d and create files named *.conf in there they will also be ready by SystemD (after a systemctl daemon-reload)

One trick is that to override some things like ExecStart= you need to first declare an empty one to reset it if the Type= isn’t oneshot.

ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND

You can also use systemctl edit some-unit.service to create and edit the file in one go. This will create override.conf for you, so that’s probably a best bet for naming any override files you might create.

I learned this from this excellent answer on AskUbuntu.

Systemd dependencies

Do you want to make sure two separate services are connected in some way? That one service is up when the other is or even starts first? Thanks to systemd this is now super easy!

In the dark old days of SysV (/etc/init.d) services were started serially during boot and individually on demand. You could adjust the start and stop order by editing a special comment in the init file. For example

# chkconfig: - 85 15

would have the service start at priority 85 and stop at priority 15. Priorities are from 0 to 99 and are done in ascending order. So this would start pretty late in the boot and stop pretty early in the shutdown.

With directives like Requires, Wants, Before, and After you can give systemd more info on what you want. systemd can start things in parallel and will use these hints to build its dependency tree and start or stop everything in the right order.

Packages will install their systemd config files into /usr/lib/systemd/system/ . Don’t edit these files. If you want to make changes, copy them to /etc/systemd/system/ .

I’ll use mariadb and httpd as examples here. It makes sense that we want the database to be up and running before Apache so that we don’t serve broken pages. Lets explore our options and see how they each act under different circumstances.


This effectively makes one unit cascade the starting of another. If the one is enabled, the other is also effectively enabled. Both units will be started in parallel, but if one of the required units fails to start, the requiring unit will get stopped again.

We’ll add Requires=mariadb.service to the [Unit] section of /etc/systemd/system/httpd.service .

When we issue a start of httpd, mariadb is started in parallel.

When we issue a stop of httpd, mariadb keeps running.

If we break the mariadb config to prevent it from starting, then start httpd, it starts, but shows an error. I think despite what the documentation says, this is just a race condition since both are started in parallel. We’ll get into ordering in a bit.

If when both are running, we stop mariadb, httpd will stop automatically.


This is just a weaker version of Requires and is the recommended option. If the wanted service fails to start, the wanting service will still start.

When we issue a start of httpd, mariadb is started in parallel.

When we issue a stop of httpd, mariadb keeps running.

If we break the mariadb config to prevent it from starting, then start httpd, it starts with no error.

If when both are running, we stop mariadb, httpd will keep running.


Now we can get to some ordering! We’ll tell httpd to start After mariadb.


On its own, it doesn’t enforce the starting of mariadb, just that if both happen to be starting at the same time, mariadb should be started first. And if you are stopping both at the same time, mariadb should be stopped last.

systemctl start httpd mariadb

will start them in order.

Before is just the inverse relationship. You could instead specify in the config for mariadb to start before httpd.

If we combine Requires and After we get what we want. httpd forcing mariadb to start first and fail to start if mariadb fails.

When we issue a start of httpd, mariadb is started first.

When we issue a stop of httpd, mariadb keeps running.

If we break the mariadb config to prevent it from starting, then start httpd, it will fail to start.

If when both are running, we stop mariadb, httpd will stop first.

More info:

Understanding Systemd Units and Unit Files


PXE Boot Kickstart

Normally to do a kickstart install, you need to first boot off an install disk (usually the NetBoot one) and then enter some extra params on the boot line. But what if you want a more automated process or don’t want to have to deal with disks? PXE boot to the rescue!

PXE boot is a process of having a computer fire up its network card, get an IP from DHCP, then pull down a bootable file.


First we need to get a TFTP server set up, preferably on the same server as our installation source.

yum install tftp-server xinetd

Then we need to enable the TFTP server, as it comes disabled by default.

sed -i '/disable/s/yes/no/' /etc/xinetd.d/tftp

Allow TFTP through the firewall

firewall-cmd --add-service=tftp
firewall-cmd --permanent --add-service=tftp

tftp-server is a sub-service of the good old xinetd, so enable and start that.

systemctl start xinetd.service
systemctl enable xinetd.service

PXE Files

Next we need to get some PXE bootable image, preferably one with a menu system. Thankfully there already exists one! syslinux is a tool for installing a bootloader onto a FAT filesystem disk, but it comes with all the bits we need to boot from the network.

yum install syslinux

All the goodies we want are in /usr/share/syslinux/. We just need the pxelinux.0 boot image, and a menu displayer vesamenu.c32. Copy these to our TFTP server.

cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot/

Now we need to config the PXE menu. pxelinux.0 will look in pxelinux.cfg/ for a file called default. Let’s create that file.

mkdir /var/lib/tftpboot/pxelinux.cfg
vi /var/lib/tftpboot/pxelinux.cfg/default

Here is what we are going to put in it

# Use the ncurses menu
default menu.c32

# Show the prompt
prompt 0

# Don't timeout
timeout 0

menu title PXE Install CentOS

# Install CentOS 7
label centos7
 menu label Install CentOS 7
 # Use this as the default
 menu default
 kernel CentOS-7.2.1511/vmlinuz
 ipappend 2
 append initrd=CentOS-7.2.1511/initrd.img inst.ks=

menu separator

label rescue7
 menu label Rescure CentOS 7
 kernel CentOS-7.2.1511/vmlinuz
 ipappend 2
 append initrd=CentOS-7.2.1511/initrd.img inst.repo= inst.lang=en_US.UTF-8 inst.keymap=us inst.rescue
# Boot the local disk
label local
 menu label Boot from local drive
 localboot 0xffff

Now we’ll copy in the kernel and init ramdisk from a install disk.

cp /media/CentOS-7.2.1511/images/pxeboot/vmlinuz /var/lib/tftpboot/CentOS-7.2.1511/
cp /media/CentOS-7.2.1511/images/pxeboot/initrd.img /var/lib/tftpboot/CentOS-7.2.1511/

DHCP Server

We need is a DHCP server to hand out an IP and pass along some pxeboot options. Your network might already have one, in which case you can simply configure it to point to your PXE server.

Let’s install and configure a DHCP server on our install server.

yum install  dhcp

Put something like the following into /etc/dhcp/dhcpd.conf

subnet netmask {
 option routers;
 filename "pxelinux.0";

Enable and start the DHCP server

systemctl enable dhcpd
systemctl start dhcpd


Now boot another server on the same network, and poke the appropriate BIOS button to get it to PXE boot. A sweet menu will show up and let you install CentOS with a single click!



VirtualBox Addon Installation in CentOS

OK, so you have a minimal install of CentOS in VirtualBox and you want to install the Guest Addons to help your VM keeps its clock correct, and share folders with your host OS. What are the dependencies? How do you get this to install?

These directions work for CentOS 5, 6 and 7.

Install some packages needed to build the kernel module.

yum install epel-release
yum install dkms gcc make bzip2 perl
yum install kernel-devel-$(uname -r)

We need EPEL to get dkms

We install dkms to ensure that the addons are rebuild when we upgrade the kernel in the future.

We do the fancy bit with the uname to ensure we get the right version of kernel-devel for the currently running kernel. If you’ve updated the kernel since booting you should reboot before installing kernel-devel and the addons or else you might have some problems building them.

Mount the Guest Addons CD and run the installer.

mkdir /mnt/cdrom
mount /dev/sr0 /mnt/cdrom/

Answer yes a few times,  and you are done!

Virtualbox Guest Addons Manual

CentOS VirtualBox HowTo

Mandatory Packages in Kickstart

When building a %packages section in a kickstart file you may want to remove some packages you don’t need. But they might get installed anyways. It turns out Anaconda requires some packages be installed. So, what are these mandatory packages?

Looking at the code for RHEL7 we can see that some basic packages from Storage, Realm, AuthConfig, and FirewallD need to be included.


    packages = storage.packages + ksdata.realm.packages
    packages += ksdata.authconfig.packages + ksdata.firewall.packages

    if not ksdata.bootloader.disabled:
        packages += storage.bootloader.packages

    if network.is_using_team_device:

storage.packages comes from Blivet and will include packages needed to configure your storage. Things like lvm2, or mdadm, or device-mapper-multipath. You can dig through the source to see which packages might be included depending on what storage you are choosing to configure.

If you use the kickstart command realm to join an Active Directory domain then the realm package will be installed.

authconfig is simply a required package to configure authentication on the system.

firewalld is also a required package.

The bootloader is usually grub2 and so will install the needed package. If you are using a non-standard bootloader check the source of bootloader.py for what packages you will get.

If you are using network teaming or bonding, then you will get teamd installed.

So be aware of these extra packages not listed in the @core group of packages.

What is a minimal install?

In my post Basics of Kickstart I talked about doing a minimal install. So what is included in the @core group for each major version of CentOS? I’ll document here for the current versions as it’s possible these values will change.

CentOS 5.11

  • Mandatory Packages
    • SysVinit
    • authconfig
    • basesystem
    • bash
    • centos-release
    • coreutils
    • cpio
    • dhclient
    • dhcpv6-client
    • e2fsprogs
    • ed
    • file
    • filesystem
    • glibc
    • hdparm
    • hmaccalc
    • initscripts
    • iproute
    • iputils
    • kbd
    • kudzu
    • libgcc
    • libhugetlbfs
    • libtermcap
    • mkinitrd
    • openssh-server
    • passwd
    • policycoreutils
    • prelink
    • procps
    • readline
    • redhat-logos
    • rootfiles
    • rpm
    • selinux-policy-targeted
    • setools
    • setserial
    • setup
    • shadow-utils
    • sysklogd
    • termcap
    • util-linux
    • vim-minimal
    • yum
  • Default Packages
    • Deployment_Guide-en-US
    • grub
    • sysfsutils
    • udftools

CentOS 6.8

  • Mandatory Packages
    • acl
    • attr
    • audit
    • authconfig
    • basesystem
    • bash
    • coreutils
    • cpio
    • cronie
    • dhclient
    • e2fsprogs
    • filesystem
    • glibc
    • initscripts
    • iproute
    • iptables
    • iptables-ipv6
    • iputils
    • kbd
    • ncurses
    • openssh-server
    • passwd
    • policycoreutils
    • procps
    • rootfiles
    • rpm
    • rsyslog
    • selinux-policy-targeted
    • setup
    • shadow-utils
    • sudo
    • system-config-firewall-base
    • util-linux-ng
    • vim-minimal
    • yum
  • Default Packages
    • aic94xx-firmware
    • atmel-firmware
    • b43-openfwwf
    • bfa-firmware
    • efibootmgr
    • grub
    • ipw2100-firmware
    • ipw2200-firmware
    • ivtv-firmware
    • iwl100-firmware
    • iwl1000-firmware
    • iwl3945-firmware
    • iwl4965-firmware
    • iwl5000-firmware
    • iwl5150-firmware
    • iwl6000-firmware
    • iwl6000g2a-firmware
    • iwl6050-firmware
    • kernel-firmware
    • kexec-tools
    • libertas-usb8388-firmware
    • netxen-firmware
    • postfix
    • ql2100-firmware
    • ql2200-firmware
    • ql23xx-firmware
    • ql2400-firmware
    • ql2500-firmware
    • rdma
    • rt61pci-firmware
    • rt73usb-firmware
    • xorg-x11-drv-ati-firmware
    • zd1211-firmware

CentOS 7.2.1511

  • Mandatory Packages
    • audit
    • basesystem
    • bash
    • biosdevname
    • btrfs-progs
    • coreutils
    • cronie
    • curl
    • dhclient
    • e2fsprogs
    • filesystem
    • firewalld
    • glibc
    • hostname
    • initscripts
    • iproute
    • iprutils
    • iptables
    • iputils
    • irqbalance
    • kbd
    • kexec-tools
    • less
    • man-db
    • ncurses
    • openssh-clients
    • openssh-server
    • parted
    • passwd
    • plymouth
    • policycoreutils
    • procps-ng
    • rootfiles
    • rpm
    • rsyslog
    • selinux-policy-targeted
    • setup
    • shadow-utils
    • sudo
    • systemd
    • tar
    • tuned
    • util-linux
    • vim-minimal
    • xfsprogs
    • yum
  • Default Packages
    • NetworkManager
    • NetworkManager-team
    • NetworkManager-tui
    • aic94xx-firmware
    • alsa-firmware
    • dracut-config-rescue
    • ivtv-firmware
    • iwl100-firmware
    • iwl1000-firmware
    • iwl105-firmware
    • iwl135-firmware
    • iwl2000-firmware
    • iwl2030-firmware
    • iwl3160-firmware
    • iwl3945-firmware
    • iwl4965-firmware
    • iwl5000-firmware
    • iwl5150-firmware
    • iwl6000-firmware
    • iwl6000g2a-firmware
    • iwl6000g2b-firmware
    • iwl6050-firmware
    • iwl7260-firmware
    • iwl7265-firmware
    • kernel-tools
    • libsysfs
    • linux-firmware
    • microcode_ctl
    • postfix
    • rdma

So if you use --nobase in your kickstart file, you will get everything in Mandatory and Default lists. You can choose to exclude some of the defaults with minus (-) to slim up your install though, or use --nodefaults to skip them entirely.

To find these lists yourself, just ask yum! yum groupinfo core

Basics of Kickstart

If you need to install CentOS over and over, one useful thing is to create a kickstart file. This is a text config file that directs the install program and can make an install entirely unattended.

You can find a full reference of all options on Red Hat’s documentation site.23.3. Kickstart Syntax Reference.


So what do we need? And what are some nice addons?

First, we can say we want to install. This is optional, but encouraged. Followed by a source for the packages. This can be a local media like cdrom or harddisk, or a network share like nfs, or my favorite url.

The URL you specify here should be to the os folder on a mirror and have as a subfolder repodata. This URL will have all the packages needed to install CentOS. You can find a list of mirrors on the CentOS site, or just provide a mirrorlist URL instead. A mirrorlist URL will give YUM a place to fetch a list of mirrors to try and it will attempt to get the fastest one.

You can also specify additional repos for the installer to pull packages from as it sees fit or that you specify. I like to at least include the updates repo, so that we install the latest packages on the first try, and don’t have to do a yum update after the install. Here is our kickstart file so far.

# Do an install

# From this hard coded URL
# url --url=http://mirror.its.sfu.ca/mirror/CentOS/7/os/x86_64/
# Or better yet, from a mirrorlist with variables
url --mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
# Extra repos let us install the latest versions
repo --name="Updates" --mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
# Optional if you want packages from EPEL
repo --name="epel" --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-$releasever&arch=$basearch

Install Settings

We can select the kind of display we want from the installer. graphical, text, or cmdline. I choose cmdline as it provides the most helpful debug output. text mode gives you the classic ncurses display.

We can also set what the installer should do after finishing. I find reboot to be the most helpful, but you can also choose halt and poweroff.

Lastly, we’ll disable firstboot. That’s the “helper” you get on the first boot up asking you to make a user and such. Since we are trying to automate things, we don’t want to be bothered.

# Install mode

# reboot when finished the install

# Disable firstboot
firstboot --disabled

System Config

Here we specify some settings for the system we are building. We’ll set the language, keyboard layout, timezone, and SELinux. We’ll also set the default password storage policy to the strongest available.

# System settings
lang en_US.UTF-8
keyboard us
authconfig --enableshadow --passalgo=sha512
selinux --disabled
timezone UTC


Most places I use Linux there is Software Defined Networking (SDN) and it handles all the firewalling, so I just disable it in the system. We also want to turn off IPv6 as its just extra junk we don’t need. And we’ll stick to DHCP here.

network --bootproto dhcp --noipv6
firewall --disabled

Root Password

This just sets the root password. You can grab the hash for an existing user from /etc/shadow or just use a plaintext password.

#rootpw --plaintext mycoolpassword
rootpw --iscrypted $6$BHils6Q1$hTRN8PUTpmQG6y7bkeSPqWrWxCV9uja9EMhsmf5qk4rDhdnKHznYiz5CxBmFqiaO14I7utwu7ToH6y7gMwFeq/

Disk Space

Now we want to specify the disk layout. Do we want basic partitions or LVM? How big should stuff be? I usually go with a 1GB swap, 1GB /tmp and the rest as root disk. I add some safety options to /tmp to make sure evil things don’t try and exec from there.

# Set up the drive
bootloader --location=mbr
clearpart --all --initlabel
part swap --asprimary --size=1024
part /tmp --fstype=ext4 --asprimary --size=1024 --fsoptions="defaults,nosuid,noexec"
part / --fstype=ext4 --grow --asprimary --size=100

Package Selection

I like to install the minimal possible and handle the rest with config management. The minimal install uses the @core group and not the @base group. @core includes a lot of packages by default that we probably don’t need. WiFi drivers, RAID card drivers, and junk like that. I’m usually building a image for VM use, so can exclude most of that by putting a minus (-) in front of the name. You can also use an asterisk (*) as a wildcard to match a bunch of packages. There are a few packages from @base I do like to include though, like acpid.

%packages --nobase

Post Script

After the install is complete, you can run some shell scripts before the reboot to help get your system just right. I make some tweaks to grub and re-install it. Then I import all the RPM keys, so that when I run yum it doesn’t ask about importing them the first time.

# Reduce timeout for faster boot
sed -i 's/GRUB_TIMEOUT=5/GRUB_TIMEOUT=1/' /etc/default/grub
# Set consoles for proper logging and vnc
# Be noisey to help debugging
sed -i 's/GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet"/GRUB_CMDLINE_LINUX="crashkernel=auto console=ttyS0,115200n8 console=tty0"/' /etc/default/grub
# Rebuild grub config
grub2-mkconfig -o /boot/grub2/grub.cfg
# Import all the keys
/bin/rpm --import /etc/pki/rpm-gpg/*

And with all that, we are done a basic kickstart. Be sure to read the docs and customize as you see fit!