Set up "Array" of two Cubieboard2's with MPI and HPL

Now that I have been able to get HPL working on the Cubieboard2 the next step would be to get it working on an array of boards. I was only able to get my hands on two boards so I am treating this as a proof of principle for later larger arrays.

If you do not have HPL set up on your board and would like a walk through please see my previous post: Installing HPL on Cubieboard2

Before we start. This is the setup I am using: Two Cubieboard2 running Ubuntu 13.10. Each board has one CPU and 2 cores with 1GB DDR3 RAM. In total we have 4 cores and 2GB RAM. I have called the boards cubiedev1 and cubiedev2 (Host names). OK lets get started.

MPI needs to be able to identify the nodes (actual machines or computers) so that it can execute the programs on each of the nodes cores. to do this we need to set up a hosts file.

Host names on Master Node

On the master node (Generally the node where you will issue the tests and store results) edit the host names file and add in the corresponding computers with their designated IP's. 

nano /etc/hosts

127.0.0.1 localhost
192.168.1.1 cubiedev1
192.168.1.2 cubiedev2

Note that you must not have the master node specified as localhost. I.E. You must not have 127.0.0.1 cubiedev1... Even if this is true for this board it will cause the other nodes to try connect to localhost when connecting to cubiedev1.

Using NFS for Ease of Testing

NFS allows you to mirror a hard drive over the network. This is extremely useful for us since to run a program such as HPL, the exact same version must be installed on all of the nodes. So instead of copying the program to all nodes we can mirror the drive and then do all our editing once and not have to worry about distributing the program around. 

To install run:

sudo apt-get install nfs-kernel-server

Now we need to share the folder we will work in... The sd card that the cubieboard has its OS on is only 8GB. I have an external HDD mounted in the directory /mnt/cub1/ if you want to mirror a folder on your sdcard its not a problem but the r/w speeds are generally not that great and you are limited by the size. So I created a directory called mpiuser on /mnt/cub1/ and I will run all my tests from this folder.

So now we have the directory /mnt/cub1/mpiuser and we must edit the folder exports and add the directory and restart the nfs service.

nano /etc/exports

/mnt/cub1/mpiuser *(rw,sync)
sudo service nfs-kernel-server restart

The folder mpiuser has now been shared but we need to mount this on the other nodes and link it to the master node. We can do this manually from the terminal each time we boot with the mount command or we can edit the fstab file so it mounts at boot.

nano /etc/fstab

cubiedev1:/mnt/cub1/mpiuser    /mnt/cub1/mpiuser    nfs

sudo mount -a
repeat on each node

Creating the user for all MPI programs

Creating one user with the same name and password on each board will allow us to easily access each node over ssh. We need to create the user and set the home directory to our shared folder mpiuser. We then also need to change the ownership of the folder to this user.

sudo adduser mpiuser --home /mnt/cub1/mpiuser  
sudo chown mpiuser /mnt/cub1/mpiuser 

Make sure that the password is the same on all boards.

Configure SSH to use keys and not passwords

Change to our new user:

su - mpiuser

Create the key using

ssh-keygen -t rsa

Use the default location as this is now a shared directory and will update to all nodes.
Now we need to add this key to the authorized keys:

cd .ssh  
cat id_rsa.pub >> authorized_keys

If you can ssh into the other nodes using their host names then you have set it up correctly. Test using:

ssh cubiedev2

MPI software

I have already installed the MPICH2 for my MPI program as I did this in the previous post mentioned before. You can use OpenMPI. It's up to you. 

We need to set up a machine file. This file will be a flag when running using the mpi command. It is a list of hosts with the specified number of nodes that you want to use. An example of the machines file that I have is:

cubiedev1:2 #The :2 represents the number of cores
cubiedev2:2

To test if this works we will use a simple test script which can be found on this blog. Save the content below to a file called mpi_hello.c

#include 
#include 

int main(int argc, char** argv) {
    int myrank, nprocs;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
    MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

    printf("Hello from processor %d of %d\n", myrank, nprocs);

    MPI_Finalize();
    return 0;
}

Compile it with

mpicc mpi_hello.c -o mpi_hello

Now run it with the correct number of specified processors (1 for each core)

mpirun -np 4 -f machines ./mpi_hello

The output I get is:

Hello from processor 0 of 4
Hello from processor 1 of 4
Hello from processor 2 of 4
Hello from processor 3 of 4

Cool... Now we know that all the processors are being "seen".

Set up the HPL files

Copy the HPL files that you have been using into the mpiuser directory on the shared hdd. Make sure the ower is set correctly via the chown hpl mpiuser command. If you are unsure of how to set up HPL please see Installing HPL on Cubieboard2

Set the HPL.dat file so that the product of P x Q = 4 (since we running it on both cubieboards) also make sure your problem size is large enough.

Now run HPL using:

mpirun -np 4 -f machines ./xhpl

Installing HPL on Cubieboard2 + Ubuntu 13.10

I am following almost exactly the same procedure as my previous post with Ubuntu 12.04. Here we are working with Ubuntu 13.10 Server on the Cubieboard2 which can be found here: http://www.cubieforums.com/index.php/topic,891.0.html

System Specs

• Cubieboard 2
•  Processor         - Allwinner A20
•  Cores               - Cortex-A7 Dual core
•  Graphics PU      - ARM® Mali400MP2
•  Memory           - 1GB DDR3

• Using Ubuntu 13.10 Server
• This version uses hardfp which is more suited for the arm and makes use of the VFP
• The GCC compiler for 13.10 is more updated than 12.04. We have 4.7

Prerequisites

HPL requires the availability of a Message Passing Interface (MPI) and either the Basic Linear Algebra Subprograms (BLAS) or Vector Signal Image Processing Library (VSIPL). In my case I have used MPICH2 and the ATLAS package both of which I got from the repository. Before you start thinking why I have not used an ATLAS tuned BLAS and that my results will be poor because of it I remind you that my main objective is to have HPL up and running first and foremost. There are too many things that can go wrong in the ATLAS tuned BLAS approach. I will however get to these topics in future posts.

Get the required packages

sudo apt-get install mpich2
sudo apt-get install libatlas3-base-dev

Then get the HPL source code from http://www.netlib.org/benchmark/hpl/hpl-2.1.tar.gz

And extract it to a folder in your home directory. We need to produce the generic make file and then edit this according to our system.

Now to install

tar -xvf hpl-2.1.tar.gz
cd hpl-2.1/setup
sh make_generic
cp Make.UNKNOWN ../Make.cubieboard

Now you must link your MPI libraries correctly in order for the build to incorporate multi core support. It took me a few hours of changing things around till I got it working. This is what I had to change in the end.

ARCH       = cubieboard
TOPdir     = $(HOME)/HDD/hpl-2.1
MPdir      = /usr/lib/mpich2
MPinc      = -I$(MPdir)/include
MPlib      = /usr/lib/libfmpich.a
LAdir      = /usr/lib/atlas-base/
LAlib      = $(LAdir)/libf77blas.so.3 $(LAdir)/libatlas.so.3
HPL_LIBS   = $(HPLlib) $(LAlib) $(MPlib) -lmpl -lcr
CCFLAGS    = $(HPL_DEFS) -mfpu=neon -mfloat-abi=hard -funsafe-math-optimizations -ffast-math -O3

Just make sure you use the correct TOPdir and if you have your libraries in different locations then change the above accordingly. I added the CCFLAGS as I wanted the best results (knowing I have standard BLAS libraries). Here is my entire make file if you would like to compare Make.cubieboard-U13.10 .

Now compile HPL

make arch=cubieboard

HPL has a large amount of input variables and an even large combination of them that can be very intimidating. I still have not wrapped my head around all of them. If you go into the HPL.dat file you will see what I mean. You can find it in the bin/cubieboard/ folder. You can find a full explanation of what the input variables do here. A very useful site I found gives you a standard HPL.dat file to start from. So lets start by going to the site and filling out the specs you need. Below is the HPL.dat file that I used.

HPLinpack benchmark input file
University of the Witwatersrand
HPL.out      output file name (if any)
8            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
8000         Ns
1            # of NBs
128           NBs
0            PMAP process mapping (0=Row-,1=Column-major)
1            # of process grids (P x Q)
1            Ps
2            Qs
16.0         threshold
1            # of panel fact
2            PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
4            NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
1            RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
1            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
1            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
64           swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
8            memory alignment in double (> 0)
##### This line (no. 32) is ignored (it serves as a separator). ######
0                               Number of additional problem sizes for PTRANS
1200 10000 30000                values of N
0                               number of additional blocking sizes for PTRANS
40 9 8 13 13 20 16 32 64        values of NB

Note that you must specify the number of cores that you want to run on. So in our case here the Cubieboard2 is a dual core hence we specify Ps X Qs = 1 X 2 = 2. If you wanted to run this on a single core then you would set Ps = Qs = 1. If you do not have the correct number of cores then you will get an error when running HPL. Note that if you run multiple process grids then you must start HPL with the maximum number of cores that are needed.

Now to start HPL on both cores I need to run the mpi command. This is done with

mpirun -np 2 ./xhpl

The -np determines the number of cores. This must be the same as the product Ps X Qs. The output is then piped to the file HPL.out

Next Up

This was largely successful as it proves that the HPL is working on both cores. The next steps will be to custom tune the BLAS libraries and also optimise the OS with better configured Kernels. This will be explain in a different post by Mitch.

Building a Cubieboard Kernel: Part 1

To date it seems that all of the pre-compiled kernels and toolchains online for the Cubieboard are using stock parameters which tend to be tuned for the Cortex-A9 or built using an older version of GCC which does not fully support the Cortex-A7 CPU!

For these reasons, and also because I would like to make a more 'Lean and Mean' kernel with less pointless drivers to waste memory, I have endeavoured to build my own. This post will describe the general process of building a kernel for the Cubieboard. I will note a few initial changes I made to the kernel config but there needs to be some testing before I can conclude whether my changes (and more to come, I'm sure) are worth it or not. I plan on making a 'Part 2' to confirm performance changes and my final kernel config.

Let's Get Started!

The first step is to ensure you have a working cross-compiler toolchain installed. If you do not, see my post here on setting up the latest Linaro toolchain. This post describes how to modify this toolchain to be more optimised for the Cortex-A7.

Besides the toolchain setup, above, please make sure you have u-boot-tools installed:

sudo apt-get install u-boot-tools

This package contains the mkimage command that is required to make the final image. You then need to get the source code. Kernel sources tend to be huge so I opted to get only the latest revision of code and no history. I think this at least halved the download size!

git clone --depth 1 https://github.com/linux-sunxi/linux-sunxi.git --branch sunxi-3.4

This was about a 400 MB download. Once it completes, there is a handy command to load an initial working config for the Cubieboard:

make ARCH=arm CROSS_COMPILE=${CC201310A7} sun7i_defconfig

Some Config Changes

If you would like to view or modify this default configuration then you can get to the normal menuconfig with:

make ARCH=arm CROSS_COMPILE=${CC201310A7} menuconfig

This will bring up the classic Linux kernel menuconfig. Here you can browse through, see some info on the various items with the help command, and change things! Be sure to save the config when you are done: there is a save option near the bottom of the main menu. Save the config as .config for it to be used by the make command.

As I mentioned earlier I chose to modify a few things in this initial run. I plan on comparing the performance between the kernels supplied by the community, a kernel that is a stock configuration but compiled for the Cortex-A7 with GCC 4.8 and also a kernel with my modifications to the config.

Initially, I chose to only turn of forced preemption, which should allow higher throughput by telling the kernel to not jump through tasks too quickly. The default was set to a real-time system which is great for desktop, but not great for processing tasks. Here's how you find the setting:

Kernel Features -> Preemption Model -> No Forced Preemption

Another issue I discovered was that by default, the ethernet drivers are not compiled into the kernel - they are build as a module. This means that to use the module we have to manually tell Linux to load it. I don't want this behaviour, so I specified to build the ethernet drivers into the kernel.

Note that to get ethernet to work after you first boot, later on in the process, you will probably have to tell the system to bring up the interface and add some stuff to the config files so that this happens on boot:

ifconfig eth0 up

echo auto eth0 >> /etc/network/interfaces

echo iface eth0 inet dhcp >> /etc/network/interfaces

The Build

Once you are happy with your changes you can build the kernel. Modify the -j3 to -j(number of CPU's + 1) to suite your build system for a faster build.

make ARCH=arm CROSS_COMPILE=${CC201310A7} uImage modules -j3

and then

make ARCH=arm CROSS_COMPILE=${CC201310A7} INSTALL_MOD_PATH=output modules_install

This will take a while... Once it's done you only need to copy the kernel uImage and modules onto your SD card! The commands below will do this for you. Note that I have mounted the boot partition of my SD card to /media/boot and the rootfs to /media/rootfs. If the uImage file is missing then the compile above failed at some point.

sudo cp -v arch/arm/boot/uImage /media/boot/

sudo rm -r /media/rootfs/lib/*

sudo cp -rv output/* /media/rootfs/lib/

Unmount the SD card, put it in your Cubieboard and hope for the best! ;)