Atlas
- CME Atlas Cluster
Setup
Head Node
The head node will need an OS, a running DHCP/DNS server, a running TFTP server, and a few NFS exports.
I've used Ubuntu Server 16.04.
Network
The server being used has four network ports, two embedded and two on a PCI expansion card. For this
cluseter, port 0 (enp32s0) will be used to connect to the UNR network, and port 1 (enp34s0) will be used to
connect to a local network switch. This can be accomplished by editing /etc/network/interfaces.
In this case, enp32s0 is set to dhcp to get an IP address from the UNR network, and enp34s0 is set to static with an IP of 10.0.0.1.
# /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). source /etc/network/interfaces.d/* # The loopback network interface auto lo iface lo inet loopback auto enp32s0 iface enp32s0 inet dhcp auto enp34s0 iface enp34s0 inet static address 10.0.0.1 netmask 255.255.255.0 network 10.0.0.0
DHCP/DNS
I've used dnsmasq for the DHCP/DNS server, and it is fairly straightforward to setup.
First, install the appropriate packages:
sudo apt update sudo apt install dnsmasq
Once installed, the config file for dnsmasq is located at /etc/dnsmasq.conf. Below is an example config file.
This config file specifies an interface for dnsmasq to run on, in this case enp34s0 (port 1), which ensures
dhcp is only run on the local network (10.0.0.0), and not the UNR network (134.197.0.0).
The dchp-option line tells dhcp clients which IP address to PXE boot from, and the
dhcp-boot lines tell dhcp clients which PXE files to boot with.
# /etc/dnsmasq.conf interface=enp34s0 dhcp-range=10.0.0.100,10.0.0.254,12h dhcp-option=3,10.0.0.1 dhcp-authoritative dhcp-boot=pxelinux.0 dhcp-boot=net:normalarch,pxelinux.0 #Optionally define MAC/IP for specific nodes #dhcp-host=xx:xx:xx:xx:xx:xx,10.0.0.101 #dhcp-host=xx:xx:xx:xx:xx:xx,10.0.0.102
Make the dnsmasq service start on boot, and restart it to ensure all changes are live.
sudo update-rc.d dnsmasq defaults sudo service dnsmasq restart
TFTP
For PXE booting clients to boot, they will need some files to boot with, provided by the head node. To accomplish this, a TFTP server must be configured, in this case tftpd-hpa was used. Install the appropriate packages:
sudo apt update sudo apt install tftpd-hpa
The configuration file for tftpd-hpa is located at /etc/default/tftdp-hpa. Below is an example config file. This config file specifies some options for the tftpd-hpa service, as well as specifying the root directorhy of the tftp server, in this case /tftp.
# /etc/default/tftpd-hpa TFTP_USERNAME="tftp" TFTP_DIRECTORY="/tftp" TFTP_ADDRESS=":69" TFTP_OPTIONS="--secure" RUN_DAEMON="yes" OPTIONS="-l -s /tftp"
Once configured, the /tftp directory will need to be created and populated with some files. In order for PXE clients to boot, the following files are needed to be in the /tftp directory:
boot/ images/ pxelinux.0 pxelinux.cfg/
Most of the files can be populated from these commands:
sudo mkdir /tftp sudo cp /usr/lib/PXELINUX/pxelinux.0 /tftp sudo mkdir -p /tftp/boot sudo cp -r /usr/lib/syslinux/modules/bios /tftp/boot/isolinux sudo mkdir -p /tftp/pxelinux.cfg sudo mkdir -p /tftp/images sudo touch /tftp/pxelinux.cfg/default
Make the tftpd-hpa service start on boot, and restart it to ensure all changes are live.
sudo update-rc.d tftpd-hpa defaults sudo service tftpd-hpa restart
PXE
The menu file for PXE is now located at /tftp/pxelinux.cfg/default. This can be configured to your liking, but here is a basic menu that will get the job done. The most important part to keep consistant if the menu is changed is the boot option for the NFSRoot label. This tells the PXE booting client to use the kernel located in the TFTP root, and to mount it's root filesystem from 10.0.0.1:/exports/xenial (which will be created later) as readonly.
# /tftp/pxelinux.cfg/default default menu.c32 prompt 0 timeout 30 ONTIMEOUT AtlasNFSRoot MENU TITLE PXE Boot Menu LABEL AtlasNFSRoot MENU LABEL Atlas NFS Root KERNEL /images/ubuntu-1604/linux APPEND root=/dev/nfs initrd=/images/ubuntu-1604/initrd.img nfsroot=10.0.0.1:/exports/xenial ip=dhcp ro
Exports
We will use /exports/ as our exporting directory, so it will need to be created.
sudo mkdir /exports
Now, add this export to /etc/exports, and sync the changes with sudo exportfs -arv.
# /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). /exports/xenial 10.0.0.0/24(ro,async,no_root_squash,no_subtree_check,insecure)
Creating the Filesystem
We will use debootstrap to create a filesystem for the booting nodes to mount, which can be installed via
sudo apt update sudo apt install debootstrap
Once created, use debootstrap to create a filesystem with a specified archetecture, distribution, and mirror, in our case amd64, xenial, and archive.ubuntu.com.
sudo debootstrap --arch amd64 xenial /exports/xenial http://archive.ubuntu.com/ubuntu
After debootstrap is finished, a few things will need to be configured within the created filesystem. You can use chroot to enter the filesystem and install packages and make configuration changes.
sudo chroot /exports/xenial/
Some essential packages to install within the filesystem are:
sudo apt install linux-firmware nano build-essential
For clients to boot from this nfsroot, some changes to fstab will need to be made. The nfs option tells fstab to mount a folder via nfs, and the tempfs option mounts a folder in memory. Within the chroot enviornment, replace /ect/fstab with this:
proc /proc proc defaults 0 0 /dev/nfs / nfs defaults,ro 1 1 none /tmp tmpfs defaults 0 0 none /var/tmp tmpfs defaults 0 0 none /var/log tmpfs defaults 0 0 none /var/lib/lightdm-data tmpfs defaults 0 0 none /var/lib/ubuntu-drivers-common tmpfs defaults 0 0 none /var/lib/pbis tmpfs defaults 0 0 none /var/lib/lightdm tmpfs defaults 0 0 none /home tmpfs defaults 0 0 none /usr/local/home/cse-admin tmpfs defaults 0 0 none /var/lib/dhcp tmpfs defaults 0 0
Updating network configuration
In order to ensure booting does not lag if interfaces are not up, enable hotplugging:
Edit /etc/network/interfaces and ensure allow-hotplug is set for the primary interface.
#/etc/network/interfaces source-directory /etc/network/interfaces.d allow-hotplug enp34s0 iface enp34s0 inet dhcp
Exit the chroot enviornment with exit when finished.
Generating initramfs
You will need to generate a new kernel and initramfs in order for it to support nfsroot arguments. This can be done with initramfs-tools
sudo apt update sudo apt install initramfs-tools
Edit /etc/initramfs-tools/initramfs.conf, and change the entries BOOT to nfs, MODULES to most, and NFSROOT to auto.
# initramfs.conf # Configuration file for mkinitramfs(8). See initramfs.conf(5). # # Note that configuration options from this file can be overridden # by config files in the /etc/initramfs-tools/conf.d directory. BOOT=nfs # # MODULES: [ most | netboot | dep | list ] # # most - Add most filesystem and all harddrive drivers. # # dep - Try and guess which modules to load. # # netboot - Add the base modules, network modules, but skip block devices. # # list - Only include modules from the 'additional modules' list # MODULES=most # # BUSYBOX: [ y | n | auto ] # # Use busybox shell and utilities. If set to n, klibc utilities will be used. # If set to auto (or unset), busybox will be used if installed and klibc will # be used otherwise. # BUSYBOX=auto # # COMPCACHE_SIZE: [ "x K" | "x M" | "x G" | "x %" ] # # Amount of RAM to use for RAM-based compressed swap space. # # An empty value - compcache isn't used, or added to the initramfs at all. # An integer and K (e.g. 65536 K) - use a number of kilobytes. # An integer and M (e.g. 256 M) - use a number of megabytes. # An integer and G (e.g. 1 G) - use a number of gigabytes. # An integer and % (e.g. 50 %) - use a percentage of the amount of RAM. # # You can optionally install the compcache package to configure this setting # via debconf and have userspace scripts to load and unload compcache. # COMPCACHE_SIZE="" # # COMPRESS: [ gzip | bzip2 | lzma | lzop | xz ] # COMPRESS=gzip # # NFS Section of the config. # # # DEVICE: ... # # Specify a specific network interface, like eth0 # Overridden by optional ip= bootarg # DEVICE= # # NFSROOT: [ auto | HOST:MOUNT ] # NFSROOT=auto
Now, generate the initramfs, and copy it and the kernel to the tftp directory:
Note:
sudo mkdir ~/tmp sudo mkinitramfs -o /tftp/images/ubuntu-1604/initrd.img sudo cp /boot/vmlinuz-$(uname -r) /tftp/images/ubuntu-1604/linux
Booting
At this point, you should have a bootable system. Add another node to the local network switch, turn it on, and enable PXE booting in the BIOS. The machine should come up with a PXE menu, and boot from the AtlasNFSRoot
- Last Author
- • abuettner
- Last Edited
- Jul 28 2017, 2:01 PM