32 Core - 8 Wandboard Array Rack Mounted

We thought it was time to make our Wandboard Array a little bit more formal. We purchased a small tray for our rack and started work on mounting the array next to its power supply.

Thanks to the Wits Physics workshop who did a great job with mounting the power supply and brackets.

Final Wandboard Array installed in rack.

Wandboard array fresh from the workshop.

Wandboard PCI-Express Adapter: Preliminary PCB 3D Images

I have been designing an adapter board to connect two Wandboards via their PCI-Express ports. The design will be finished by the end of this week, after which it will be sent in for manufacturing! This post is simply to show off two quick 3D renderings of the boards so far.

We will use this board for testing and benchmarking the potential throughput of the Freescale i.MX6 processor. The gigabit ethernet on the i.MX6 is limited to ~400 Mbps according to the datasheet, but we should be able to attain close to 5 Gbps with extremely low latency using the PCI-Express Gen 2 x1 port!

More details will follow when the board has been manufactured and tests are complete!

Complete Wandboard Array

Following the previous posts regarding the installation of HPL on the Wandboard and Cubieboard2 and the subsequent setup of the two Cubieboards connected and running HPL I am pleased to share that we have set up five Wandboards running Ubuntu 13.05 Server (Thanks to Martin Wild) and using MPICH2 as the MPI.

If you would like details on how to set up multiple boards please view my post of setting up the Cubieboard2 "array" here.

Getting things ready

The Wandboards arrived with out power adapters. We decided to build our own using a normal PC 300W power supply as this provides proper grounding and if there are static discharges against the boards they will be better protected.

Here is the first power connector we made. The green strip is a small two channel PCB board.

 The cable is standard two core cabling with a plug at the end which fits the Wandboard sockets. I used extra long spacer screws so that we could stack the boards on top of each other. They had to be spaced wide enough that the heat wouldn't be an issue and so that we could get fingers at each board in case we need to add hard drives.

Stacked array of the Wandboards

Now before connecting the power I had to write pre-made images to the sd card for each board Once I had one board up and running I then copied the SD card to the remaining four and set each IP address. There were a few issues with this which ill speak about at the end. Once the board were all up I connected everything together.

Completed Array

Wandboard array with power and Ethernet

 Now following similar procedures as my previous posts I set up HPL over a shared drive using NFS and I configured the HPL for neon and hardfp. I ran a quick test on the array using a small problem to test if all boards would indeed respond correctly. I was happy to see that all five boards showed xhpl in the processes (top) when I ran HPL.

Five terminals showing active processes when running HPL

Next up

I tried to compile ATLAS for the Cubieboard2 using neon-vfpv4 but the compiling got stuck at the L1 cache due to an infinity popping up somewhere. I will recompile that using just neon and do something similar for the Wandboard. This will improve performance quite a lot as I am using a standard ATLAS library at the moment. Once that is done I will be able to start tuning the HPL.dat file for the array.

Problems Encountered

An interesting problem came up when I copied the OS from one SD card to the other. During boot it would take exceptionally long and after finally starting up there would be no Ethernet. I checked for the adapters using ifconfig -a and they were named eth1 or eth2... not the default eth0. After doing some investigating it was quite obvious... When Linux boots up it searches for the devices and saves them in the following file:

/etc/udev/rules.d/70-persistent-net.rules

Since the hardware was changing it was appending the new hardware to the end of this list. Thus the system searched for the first one and then moved onto the next. Simply removing the content and restarting solved this issue.

Another issue was the locals. I am not sure why this one popped up but after some reading through some material I just generated the locale and reconfigured it. Using the following commands:

sudo locale-gen fi_FI.UTF-8
sudo dpkg-reconfigure locales