WandAdapter 2.0 Complete

I have finally got around to taking some photos of the "completed" WandAdapter 2.0. Check them out below! I have yet to test the performance of the boards Ethernet and PCI-Express - but as soon as I do I'll post some results.

I plan on releasing the designs under an open source license soon - stay tuned!

WandAdapter 2.0 with attached Wandboard

Wandboard SSD Stand

I decided that my development Wandboard needed a stand, as opposed to it lying on an anti-static mat at random angles. I 3D printed a simple plastic piece that also allowed me to attach a fast Corsair 60 GB SSD for SATA benchmarks and faster builds. A quick dd test gives me 113 MB/s write speed! Check the pictures below!

I plan to install Debian with a GUI, along with Eclipse, to do kernel development natively. This should make writing and debugging ARM-based drivers a much simpler exercise! The issue is that Eclipse only has a Debian ARM 'port' and does not have any Ubuntu or Arch variants. I will probably try to get it working on Linaro but only time will tell...

WandAdapter 2.0 Nearing Completion

I finally received the raw PCBs for what I call the WandAdapter 2.0! I'll be soldering over the next few days and hopefully it powers up and works! My concerns are, as always, the high speed signals for the PCI-Express (or PCIe), PCI-Express clock, USB and Ethernet. The board was simulated with Mentor Graphics HyperLynx so everything should work...

The idea behind this board is to connect a Wandboard to a standard PCIe x1 connector for true EndPoint operation. All of the important signals for PCIe have been routed to the Wandboard such as the external clock, wake, reset and of course the data! The board breaks out USB host and device (OTG) as well as gigabit Ethernet and debug UART.

One issue I have noticed so far is that the PCIe present pins (one short gold one is visible at the bottom) unfortunately has a track routed down which was used for the hard gold plating process. For future production runs, it's a good idea to route this signal to the edge of the board so that it doesn't have to be scratched off manually. What's the point of a short pin when it's as long as all the rest?

I'll be posting photos of the completed board soon for some eye candy!

32 Core - 8 Wandboard Array Rack Mounted

We thought it was time to make our Wandboard Array a little bit more formal. We purchased a small tray for our rack and started work on mounting the array next to its power supply.

Thanks to the Wits Physics workshop who did a great job with mounting the power supply and brackets.

Final Wandboard Array installed in rack.

Wandboard array fresh from the workshop.

Extra Steps for Building a Wandboard (i.MX6Q) Image

I have built several Linaro based images for the Wandboard or Freescale i.MX6Q SoC, and it's a seemingly simple process of building the kernel, partitioning and installing the Linaro rootfs to a SD card. One hit's a wall when anything 'fancy' needs to be done on this image... this post aims to document how to finish off the image so that it can be used for kernel development and other advanced tasks.

Please note that this is a fairly advanced howto. Most of the concepts here can be found in other places online. The methodology behind the i.MX6 libraries comes from 'reverse engineering' the LTIB install scripts from Freescale, yocto, and lots of reading. I'm sure there is a better way to do this - but I don't know it (I'm interested to hear though)!

Linux Source Code and Modules
The Linux kernel source code should be cleaned (make ARCH=arm clean) and copied onto the image (or SD card - I'll use the terms randomly) into the /usr/src/linux directory.

Before you clean the tree, build the modules and headers and install them to a known path to be copied onto your SD card root filesystem.

make modules_install INSTALL_MOD_PATH=/some/directory/

make headers_install INSTALL_HDR_PATH=/some/directory/

Copy the contents of /some/directory to into the SD card /lib directory. You should now have a new directory: /lib/modules/3.0.35Linux+ or something similar. The headers_install directory goes into /usr.

When you boot with the new SD card, you will need to modify some symlinks (build and source) that reference the wrong place (they will be linking to directories from your build machine, which are obviously not valid any more).

cd /lib/modules/3.0.35Linux+/
ls -l

You should see the wrong links. Now to remove them and re-add them, correctly:

rm ./build
rm ./source
ln -s /usr/src/linux ./build
ln -s /usr/src/linux ./source

These commands will have to be run with sudo or as root.

Something you may also have to do, in case you cleaned the kernel source tree too thoroughly, or you cross compiled (the kernel scripts directory is full of x86 binaries which wont work on ARM: modpost, etc.), is ensure your proper .config file is there and run:

make ARCH=arm oldconfig
make ARCH=arm prepare
make ARCH=arm modules_prepare

I've also found that doing a module build fixed up some errors!

make ARCH=arm modules

Download the i.MX6 Libraries
Download and extract the Freescale BSP somewhere onto your PC (not the Wandboard). There is a source directory with loads of .gz files for various applications. We are interested in several of these. This section will explain how to install the important ones.

A list of the files we will be working with is below. Copy them onto the SD card before booting, or SCP them across. From what I have seen, the 3.0.35-4.1.0 and 3.10.17-1.0.0 are basically the same. The 3.10.17 files can be acquired from a partial yocto installation - Google can probably help with finding the files otherwise!

imx-lib-*.tar.gz
imx-vpu-*.bin
firmware-imx-*.bin
imx-test-*.tar.gz
gpu-viv-bin-mx6q-*hfp.bin (the version from the Freescale BSP seems to be soft-fp... it won't work.)
gpu-viv-g2d-*.bin
fsl-gpu-sdk-*.bin

Some of these are bins, with things to agree to before they extract. Go ahead and extract everything in preparation for the installation steps.

Install the i.MX6 Libraries
First, a few exports to make our lives easier:

export KERNELDIR='/usr/src/linux'

export INCLUDES='-I$KERNELDIR/include -I$KERNELDIR/drivers/mxc/security/rng/include -I$KERNELDIR/drivers/mxc/security/sahara2/include'

As per the usual, make sure the KERNELDIR variable points to your specific kernel directory. Note that if you have installed the kernel headers, etc. properly in the steps above, you may not need to do these exports and you can leave the INCLUDE= part out of the make commands below.

firmware:

Simply copy the contents of the firmware-imx* directory into /lib so that you have new files in /lib/firmware/vpu, etc.

imx-lib:

From the imx-lib directory:

make -j1 PLATFORM="IMX6Q" INCLUDE="$INCLUDES"

sudo make PLATFORM="IMX6Q" install 

If all compiled and copied, you should now see a bunch of new libraries in /usr/lib! Congratulations!

imx-vpu:

Even if you don't want to use the VPU, this is a dependency for the imx-tests. From the imx-vpu directory:

make -j1 PLATFORM="IMX6Q" INCLUDE="$INCLUDES"

sudo make PLATFORM="IMX6Q" install 

If all compiled and copied, you should now see a bunch of new libraries in /usr/lib! Congratulations!

gpu-viv-bin and gpu-viv-g2d:

Copy the contents of the archives into your root. /opt and /usr will now contain new files.

imx-test:

From the imx-test directory:


make -j1 PLATFORM="IMX6Q" test

At this point you can run some of the unit tests that compiled successfully (not all will have) from the ./platform/IMX6Q/autorun*.sh files.

sudo make PLATFORM="IMX6Q" install 

If all compiled and copied, you should now see a bunch of new libraries in /usr/lib! Congratulations!

gpu-viv-bin:

The GPU drivers and binaries are closed source, so it's a matter of extracting the files into the correct place. Search for gpu-viv-*.gz and copy it onto your Wandboard.

Boot, extract it and cd into the new directory. You will see an 'opt' and a 'usr' directory. Run the following commands:

sudo cp -Rv ./opt/* /opt/
sudo cp -Rv ./usr/* /usr/

Make sure the files copied into the correct places.

Upgraded Wandboard Cluster

We have invested in a few more Wandboards to upgrade our mini-cluster. Since I repurposed two of the existing boards for my PCI-Express research, we now have 8 Wandboards in the cluster which makes 32 cores! A little more power for scientific applications on ARM...

Besides having more processing power available, I have made some enhancements to the power distribution board. The thin purple, pink and red wires in the background will be connected to one of the Wandboard I2C inputs to enable easy and accurate voltage, current and power measurements of the DC supply to the cluster. The excellent TI INA219 chip is used for this. It's possible to read these values from the Linux command-line so we can use a shell script to plot power values directly to a file!

Dual Wandboard PCI-Express Connector Complete

The Wandboard PCI-Express adpater is complete! I'm in the process of building a new kernel that supports the PCI-Express RC and Endpoint to start testing the Freescale i.MX6 Quad PCI-Express capabilities.

Here are some photos of the completed board with two Wandboards attached! In the last photo you can see our (now slightly diminished) Wandboard cluster. We are waiting for another five Wandboards to make it 40 cores but no suppliers seem to have stock at the moment...

Wandboard PCI-Express Connector PCB Photos

Today I took delivery of the first (and hopefully last if it's bug-free) version of the dual Wandboard PCI-Express adapter. I plan on having it soldered and ready for testing by early next week. Check out the photos!

Wandboard PCI-Express Adapter: Update

It's been a while since I last posted about the Wandboard PCI-Express adapter I have been working on... I decided to redesign the PCB to be more compact. This saves manufacturing costs and it looks better, in my opinion.

The PCB has been sent for manufacture so hopefully in a week or two I can post some photos! Shortly after that - assuming everything goes according to plan - I'll post some results for the PCI-Express performance of the Freescale i.MX6 SoC. I don't think the PCI-Express interface to the Wandboard has been tested by anyone, so hopefully it works...

High Speed PCB Routing

I have been quite busy designing high-speed PCB's for my PCI-Express research. I found this video on YouTube from Texas Instruments which provides an excellent overview on high speed routing and I recommend it to anyone interested.

To reiterate the issues one usually faces when designing high speed circuit boards (from the video above):

• Timing: the lengths of the tracks must be similar enough for the electrical signals to arrive at the receiver at the same time. ~0.6 times the speed of light is too slow!
• Signal Integrity: The shape of the signal needs to be right when it arrives at the receiver.
• Noise: There can be a lot of crosstalk and noise on a high speed PCB and this noise can adversely affect signals.

To 'solve' these concerns:

• Maintain the correct impedance from the transmitter to the receiver. This is not always trivial and so this is usually the biggest problem!
• Matched lengths minimise signal skew.
• Leave space around the traces to minimise noise. More space makes an exponential difference.

Wandboard PCI-Express Adapter: Preliminary PCB 3D Images

I have been designing an adapter board to connect two Wandboards via their PCI-Express ports. The design will be finished by the end of this week, after which it will be sent in for manufacturing! This post is simply to show off two quick 3D renderings of the boards so far.

We will use this board for testing and benchmarking the potential throughput of the Freescale i.MX6 processor. The gigabit ethernet on the i.MX6 is limited to ~400 Mbps according to the datasheet, but we should be able to attain close to 5 Gbps with extremely low latency using the PCI-Express Gen 2 x1 port!

More details will follow when the board has been manufactured and tests are complete!

Complete Wandboard Array

Following the previous posts regarding the installation of HPL on the Wandboard and Cubieboard2 and the subsequent setup of the two Cubieboards connected and running HPL I am pleased to share that we have set up five Wandboards running Ubuntu 13.05 Server (Thanks to Martin Wild) and using MPICH2 as the MPI.

If you would like details on how to set up multiple boards please view my post of setting up the Cubieboard2 "array" here.

Getting things ready

The Wandboards arrived with out power adapters. We decided to build our own using a normal PC 300W power supply as this provides proper grounding and if there are static discharges against the boards they will be better protected.

Here is the first power connector we made. The green strip is a small two channel PCB board.

 The cable is standard two core cabling with a plug at the end which fits the Wandboard sockets. I used extra long spacer screws so that we could stack the boards on top of each other. They had to be spaced wide enough that the heat wouldn't be an issue and so that we could get fingers at each board in case we need to add hard drives.

Stacked array of the Wandboards

Now before connecting the power I had to write pre-made images to the sd card for each board Once I had one board up and running I then copied the SD card to the remaining four and set each IP address. There were a few issues with this which ill speak about at the end. Once the board were all up I connected everything together.

Completed Array

Wandboard array with power and Ethernet

 Now following similar procedures as my previous posts I set up HPL over a shared drive using NFS and I configured the HPL for neon and hardfp. I ran a quick test on the array using a small problem to test if all boards would indeed respond correctly. I was happy to see that all five boards showed xhpl in the processes (top) when I ran HPL.

Five terminals showing active processes when running HPL

Next up

I tried to compile ATLAS for the Cubieboard2 using neon-vfpv4 but the compiling got stuck at the L1 cache due to an infinity popping up somewhere. I will recompile that using just neon and do something similar for the Wandboard. This will improve performance quite a lot as I am using a standard ATLAS library at the moment. Once that is done I will be able to start tuning the HPL.dat file for the array.

Problems Encountered

An interesting problem came up when I copied the OS from one SD card to the other. During boot it would take exceptionally long and after finally starting up there would be no Ethernet. I checked for the adapters using ifconfig -a and they were named eth1 or eth2... not the default eth0. After doing some investigating it was quite obvious... When Linux boots up it searches for the devices and saves them in the following file:

/etc/udev/rules.d/70-persistent-net.rules

Since the hardware was changing it was appending the new hardware to the end of this list. Thus the system searched for the first one and then moved onto the next. Simply removing the content and restarting solved this issue.

Another issue was the locals. I am not sure why this one popped up but after some reading through some material I just generated the locale and reconfigured it. Using the following commands:

sudo locale-gen fi_FI.UTF-8
sudo dpkg-reconfigure locales

EDM Connector Altium Library

The Wandboard uses something called an EDM Connector to connect the mezzanine board to the base board. This connector is actually a standard MXM3.0 connector used by graphics cards in laptops. Some people over at http://www.edm-standard.org/ have begun work on developing a standard that uses this connector for multimedia and other general signals - which is exactly why the Wandboard implements this standard! Download the PDF from their web site and check it out!

I was unable to find any schematic libraries on the internet for this connector, including in the Altium libraries! Thus, I was forced to make one. I'm making it available for free use - but if you improve it please share your improvements with me, and others. The footprint is based off both the Foxconn and JAE datasheets which can be found at Digi-Key and Future Electronics.

Edit:
Note that the JAE connector will not work for the EDM standard as some of the E3 and E4 signals are not present, which means the ethernet will be unimplemented. The connector is mostly the same though. You can see the notch to the left of the picture above - this should not be there.

The JAE connector should work. The missing signals from the notch to the left of the connector are as follows: E1-10, E2-10, E3-1, E4-1. These are VCC and GND signals which are elsewhere on the connector. We can assume that the board connected is using a ground and power plane so this should not adversely affect operation.

Here's my link on Google Drive: MXM3.IntLib. You should be able to download it without already having Google Drive yourself. Google might try open it in Google Docs - I'm not sure why - but click on File -> Download and you will get the actual file!

Installing HPL on Wandboard + Ubuntu 12.04

With the ultimate goal of benchmarking an array of Wandboards a good starting point is to install High Performance LINPACK on one machine and then begin expanding the number of boards. This post will discuss how to configure all the required packages and the HPL itself to get it up and running. Note: This is not a discussion on tuning or optimizing for high flop counts but just to get a working benchmark. I will discuss tuning in a later post.

System Specs

• Wandboard Quad
•  Processor         - Freescale i.MX6 Quad
•  Cores               - Cortex-A9 Quad core
•  Graphic engine  - Vivante GC 2000 + Vivante GC 355 + Vivante GC 320
•  Memory           - 2GB DDR3

• Using Ubuntu 12.04 LTS
• This version uses softfp which is not ideal and hence why I am not going to tune the HPL
• The GCC compiler for 12.04 is outdated when it comes to ARM and does not support hardfp
• Ubuntu 13.10 is available and has the much needed updates. I will move across to this soon when I have a decent set of results to compare the 12.04 and 13.10 versions. This will be a nice comparison primarily between the hardfp vs softvp effect.

Prerequisites

HPL requires the availability of a Message Passing Interface (MPI) and either the Basic Linear Algebra Subprograms (BLAS) or Vector Signal Image Processing Library (VSIPL). In my case I have used MPICH2 and the ATLAS package both of which I got from the repository. Before you start thinking why I have not used an ATLAS tuned BLAS and that my results will be poor because of it I remind you that my main objective is to have HPL up and running first and foremost. There are too many things that can go wrong in the ATLAS tuned BLAS approach. I will however get to these topics in future posts. I assume you have the standard compilers that have come with the Ubuntu 12.04 image. If you are wondering why I used MPICH2 rather than OpenMPI is that MPICH2 worked first :)

Get the required packages

sudo apt-get install mpich2
sudo apt-get install libatlas-base-dev

Then get the HPL source code from http://www.netlib.org/benchmark/hpl/hpl-2.1.tar.gz

And extract it to a folder in your home directory. We need to produce the generic make file and then edit this according to our system.

Now to install

tar -xvf hpl-2.1.tar.gz
cd hpl-2.1/setup
sh make_generic
cp Make.UNKNOWN ../Make.wandboard

Now you must link your MPI libraries correctly in order for the build to incorporate multi core support. It took me a few hours of changing things around till I got it working. This is what I had to change in the end.

ARCH       = wandboard
TOPdir     = $(HOME)/HDD/hpl-2.1
MPdir      = /usr/lib/mpich2
MPinc      = -I$(MPdir)/include
MPlib      = /usr/lib/libfmpich.a
LAdir      = /usr/lib/atlas-base/
LAlib      = $(LAdir)/libf77blas.a $(LAdir)/libatlas.a
HPL_LIBS   = $(HPLlib) $(LAlib) $(MPlib) -lmpl -lcr
CCFLAGS    = $(HPL_DEFS) -mfpu=neon -mfloat-abi=softfp -funsafe-math-optimizations -ffast-math -O3

Just make sure you use the correct TOPdir and if you have your libraries in different locations then change the above accordingly. I added the CCFLAGS as I wanted the best results (knowing I have standard BLAS libraries). Here is my entire make file if you would like to compare Make.wandboard-U12.04.

Now compile HPL

make arch=wandboard

HPL has a large amount of input variables and an even large combination of them that can be very intimidating. I still have not wrapped my head around all of them. If you go into the HPL.dat file you will see what I mean. You can find it in the bin/wandboard/ folder. You can find a full explanation of what the input variables do here. A very useful site I found gives you a standard HPL.dat file to start from. So lets start by going to the site and filling out the specs you need. Below is the HPL.dat file that I used.

HPLinpack benchmark input file
University of the Witwatersrand
HPL.out      output file name (if any)
8            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
10240         Ns
1            # of NBs
128           NBs
0            PMAP process mapping (0=Row-,1=Column-major)
1            # of process grids (P x Q)
2            Ps
2            Qs
16.0         threshold
1            # of panel fact
2            PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
4            NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
1            RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
1            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
1            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
64           swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
8            memory alignment in double (> 0)
##### This line (no. 32) is ignored (it serves as a separator). ######
0                               Number of additional problem sizes for PTRANS
1200 10000 30000                values of N
0                               number of additional blocking sizes for PTRANS
40 9 8 13 13 20 16 32 64        values of NB

Note that you must specify the number of cores that you want to run on. So in our case here the Wandboard is a quad core hence we specify Ps X Qs = 2 X 2 = 4. If you wanted to run this on a single core then you would set Ps = Qs = 1. If you do not have the correct number of cores then you will get an error when running HPL. Note that if you run multiple process grids then you must start HPL with the maximum number of cores that are needed.

Now to start HPL on all four cores I need to run the mpi command. This is done with

mpirun -np 4 ./xhpl

The -np determines the number of cores. This must be the same as the product Ps X Qs. The output is then piped to the file HPL.out

Next Up

This was largely successful as it proves that the HPL is working on 4 cores. I quite like the idea of having results before any optimizations as we can quantitatively see how much improvement we get from the tuned BLAS and new hardfp we will have in the next set of testing on the Ubuntu 13.10 system.