8GHz Tektronix Oscilloscope Delivered

We were extremely fortunate to be granted a major funding application from the university last year for a very high end, Tektronix 8 GHz, 25 GSa/s oscilloscope. We also received an 8 GHz differential probe for the oscilloscope enabling us to get the most out of it.

We plan to use this oscilloscope for measuring PCI-Express and other high speed signals.

We don't have any photos of signal measurements yet, but the scope works perfectly so far! Check out the photos below of Oscar just after installation, and Marc when he realized that the oscilloscope runs Windows 7...

Hopefully other staff and post-grads at the university will find this oscilloscope useful to their research as well. We hope to get many years of good service from this awesome piece of equipment.

Interview with the SABC

This week our HEP group had a great visit from the SABC. Their crew were filming a piece for the show “Teenagers On a Mission” (Tomz). They were very interested in the 3D printing development that we have been doing for the Mini PC project. We were able to outline the whole 3D printing process and give them quick demonstration using our 3D scanning and printing setup in the lab.

Thomas Wrigley giving his take on 3D printing in South Africa

“Tomz, part factual, part fictional with some drama, comedy and science fiction will be broadcast in a fair spread of Xhosa, Sotho, Sepedi, English and Zulu with full subtitles in English”

 Airs every week: Monday 4:00-4:30pm, Tuesday 4:00-4:30pm on SABC 1

Lectures by Dr Peter Braam

Dr Peter Braam, the creator of the massively parallel Lustre file-system is in South Africa for a few days to lecture on topics related to software engineering, high performance computing and of course how these topics are relevant to the Square Kilometer Array project.

The series of lectures, organised by the Center for High Performance Computing (CHPC) was originally intended for just the Cape Town region. Due to a high level of interest as well as the new high throughput electronics lab at The University of the Witwatersrand, he spent the day in Johannesburg too.

A great lunch was served and attendance was good, with over 30 people attending the afternoon event. Dr Braam lectured on software engineering practices and also on the state of high performance computing storage requirements which rang several bells in line with the groups research.

I would have loved to post a photo of the event but unfortunately my phone corrupted the image. These things happen...

New NVIDIA Jetson-TK1 Cluster

We recently finished setting up a new cluster, but this time we wanted to go for maximum processing power compared to our existing Wandboard Cluster! We opted for the new NVIDIA Jetson-TK1 development boards which hosts the NVIDIA Tegra K1 System on Chip...

The Tegra K1 is a beast: it has a quad-core ARM Cortex-A15 CPU which runs at up to 2.3 GHz with a fifth low power core which is also a Cortex-A15 except it's clock is limited to a few hundred MHz. There is also a 192 core CUDA (Kepler) GPU on the SoC which, according to the spec sheets, can attain about 350 GFLOPS of performance. Preliminary benchmarks of the CPU indicate that High Performance Linpack scores over 20 GFLOPS in single precision! 

We have built a cluster with 11 boards, which equates to 44 ARM Cortex-A15 cores (~220 GFLOPS), 22 GB RAM, Gigabit Ethernet and around 3850 GFLOPS worth of GPGPU processing power! The entire cluster should consume less than 200 W of electricity under load.

We have started running benchmarks on the cluster and will report the results soon...

Adventures with ARM GCC Auto-Vectorization

I have been experimenting with how well gcc (4.8.2) works with compiling a simple dot product of two vectors with 8 elements in each. I have discovered some interesting things (whether they are bugs or quirks I don't know) and I wanted to share my findings...

The first thing to know is how to generate assembly with gcc. Simply add -S to the gcc or g++ command. I am using Eclipse CDT and I have created several build configurations (some of which generate assembly for inspection).

I have been experimenting with the Eigen Library since it aparently supports ARM NEON. What I found is that it works extremely well with x86 SIMD instructions but not so well with ARM NEON. It seems that well coded C++ is better for NEON that Eigen...

x86 SSE2 with Eigen

movaps      32(%rdi), %xmm0

movaps      16(%rdi), %xmm1

mulps       32(%rsi), %xmm0

mulps       16(%rsi), %xmm1

addps       %xmm0, %xmm1

movaps      %xmm1, %xmm0

movhlps     %xmm1, %xmm0

addps       %xmm1, %xmm0

movaps      %xmm0, %xmm1

shufps      $1, %xmm0, %xmm1

addss       %xmm1, %xmm0

x86 SSE2 without Eigen

movq      (%rdi), %rdx

movq      (%rsi), %rax

movss     (%rdx), %xmm1

mulss     (%rax), %xmm1

movss     4(%rdx), %xmm0

mulss     4(%rax), %xmm0

addss     .LC0(%rip), %xmm1

addss     %xmm0, %xmm1

movss     8(%rdx), %xmm0

mulss     8(%rax), %xmm0

addss     %xmm0, %xmm1

movss     12(%rdx), %xmm0

mulss     12(%rax), %xmm0

addss     %xmm0, %xmm1

movss     16(%rdx), %xmm0

mulss     16(%rax), %xmm0

addss     %xmm0, %xmm1

movss     20(%rdx), %xmm0

mulss     20(%rax), %xmm0

addss     %xmm0, %xmm1

ARM NEON-VFPv4 with Eigen

flds        s0, .L2

ldr         r3, [r1]

ldr         r2, [r0]

flds        s15, [r3]

flds        s14, [r2]

vfma.f32    s0, s14, s15

flds  s6,   [r2, #4]

flds  s7,   [r3, #4]

flds  s8,   [r2, #8]

flds  s9,   [r3, #8]

flds  s10,  [r2, #12]

flds  s11,  [r3, #12]

flds  s12,  [r2, #16]

flds  s13,  [r3, #16]

flds  s14,  [r2, #20]

flds  s15,  [r3, #20]

vfma.f32    s0, s6, s7

vfma.f32    s0, s8, s9

vfma.f32    s0, s10, s11

vfma.f32    s0, s12, s13

vfma.f32    s0, s14, s15

ARM NEON-VFPv4 without Eigen

ldr     r3, [r0]

vmov.f32    q8, #0.0  @ v4sf

ldr     r2, [r1]

vld1.32     {q9}, [r3]!

vld1.32     {q10}, [r2]!

vmul.f32    q10, q10, q9

vst1.64     {d20-d21}, [sp:64]

vld1.32     {q9}, [r3]

vld1.32     {q11}, [r2]

vmov     q12, q10  @ v4sf

vfma.f32    q12, q11, q9

vadd.f32    d18, d24, d25

vpadd.f32   d16, d18, d18

vmov.32     r3, d16[0]

It took quite some effort to get the non-Eigen ARM code to be better then the Eigen code. The "naive" version with a simple dot-product for-loop (shown below) was similar to what Eigen produced. The a and b variables have been __restrict__ed and are pointers to aligned memory.

for (int i = 0; i < 8; i++)

{

    out += a[i] * b[i];

}

The results are not what I would expect. I decided to split these two operations into two separate loops and I got the NEON version shown above! 

float prods[8];

for (int i = 0; i < 8; i++)

{

    prods[i] = a[i] * b[i];

}

for (int i = 0; i < 8; i++)

{

    out += prods[i];

}

I should also add that without -funsafe-math-optimizations, the auto-vectorization doesn't work. I'm going to keep working on it to see if I can shed a few more instructions, but so far so good!

ACAT 2014 Workshop

Today was the conclusion of the 16th international workshop for Advanced Computing and Analysis Techniques (ACAT) in physics research. I am extremely glad to have attended this excellent workshop. It was very well organised and sported a wide variety of topics from new computing hardware and techniques to algorithms and even some physics theory, making a well rounded program. The networking and friend-making opportunities were also great.

My talk was an overview of the research the group at Wits has performed so far, as well as details on my progress and plans for the PCI-Express SoC interconnect. There was also a poster from the University of Cape Town, but unfortunately Josh (the author) was unable to attend the conference. I presented the poster in his stead to much interest from the community.

There were two other talks and a plenary by the Barcelona Supercomputing Center on ARM System on Chips. Several other talks also made passing mention of ARMs potential suitability for scientific computing. All of these presentations were met with positive comments and questions from the audience.

I felt honored several times as the organizers and ACAT founders repeatedly mentioned that they had a South African participant (the first apparently)! Our project also received several mentions in the summaries - enhanced by the South African label.

I look forward to attending the workshop again in 18 months time and I hope next time more South Africans will be fortunate enough to attend. Perhaps with the Square Kilometer Array project surging ahead, we can attract the ACAT workshop to South Africa in a few years?

Optimal Compiler Flags for ARM

ARM GCC FLAGS

To get a good approximation for the ideal set of gcc flags to use on ARM, COREMARK was compiled and run several times with different flag combinations. I used 16 different combinations with what I assumed were the most important generic flags (Please leave a comment if you know of any other flags I should include!). Below is an example for the Cortex-A7. A similar set of flags was used for the A9 and A15.

Example of flag combinations for the Cortex-A7

I chose coremark for four reasons. One, it is supported by ARM and suggested as the embedded benchmark of choice. Two, it is easily compiled and allows easy configuration. Three, it does not use external libraries so the flags have an affect. Four, the benchmark generates random data during the test so no work is optimised away by the compiler,

Results for different flag combinations for A7, A9 and A15.

Listed below are the flags that were found to be the best. (Generally between 6-10% improvement).

A7    : -O3 -mfpu=neon-vfpv4 -march=armv7-a -mtune=cortex-a7
A9    : -O3 -mfpu=neon         -march=armv7-a
A15  : -O3 -mfpu=neon-vfpv4 -march=armv7-a

A sanity check on my runs found that the iterations I was using in CoreMark was fair for each board. This can be seen by the plateau that is reached by each board. 

Iteration count vs performance. 10s run time was adequate to reach a plateau.

WandAdapter 2.0 Complete

I have finally got around to taking some photos of the "completed" WandAdapter 2.0. Check them out below! I have yet to test the performance of the boards Ethernet and PCI-Express - but as soon as I do I'll post some results.

I plan on releasing the designs under an open source license soon - stay tuned!

WandAdapter 2.0 with attached Wandboard

Linux /usr/include Proper Permissions

I recently broke the permissions in my /usr/include folder - probably whilst copying new include files into the directory. None of my compiles seemed to be working out as they should have been...

I did a little research and found out what they should be. I've written the commands to execute below for my own future reference and in the hope that someone else finds use in them!

sudo su -
cd /usr/include
find . -type d -exec chmod 755 {} \;
find . -type f -exec chmod 644 {} \;
chown -R root:root /usr/include/*

NVIDIA Tegra K1 Development Board Arrived

Today our first sample NVIDIA Jetson TK1 development board arrived! This was a pleasant surprise since I was unaware that one had finally been shipped! Thanks NVIDIA!


The board is larger than I expected and has many great features. With a powerful ARM Cortex-A15 quad core CPU, 2 GB RAM and a 192 core Kepler GPU (that apparently can perform at ~350 GFLOPS single precision), this SoC is going to be great fun to benchmark and hopefully use in later research... Watch this space!

Wandboard SSD Stand

I decided that my development Wandboard needed a stand, as opposed to it lying on an anti-static mat at random angles. I 3D printed a simple plastic piece that also allowed me to attach a fast Corsair 60 GB SSD for SATA benchmarks and faster builds. A quick dd test gives me 113 MB/s write speed! Check the pictures below!

I plan to install Debian with a GUI, along with Eclipse, to do kernel development natively. This should make writing and debugging ARM-based drivers a much simpler exercise! The issue is that Eclipse only has a Debian ARM 'port' and does not have any Ubuntu or Arch variants. I will probably try to get it working on Linaro but only time will tell...

WandAdapter 2.0 Nearing Completion

I finally received the raw PCBs for what I call the WandAdapter 2.0! I'll be soldering over the next few days and hopefully it powers up and works! My concerns are, as always, the high speed signals for the PCI-Express (or PCIe), PCI-Express clock, USB and Ethernet. The board was simulated with Mentor Graphics HyperLynx so everything should work...

The idea behind this board is to connect a Wandboard to a standard PCIe x1 connector for true EndPoint operation. All of the important signals for PCIe have been routed to the Wandboard such as the external clock, wake, reset and of course the data! The board breaks out USB host and device (OTG) as well as gigabit Ethernet and debug UART.

One issue I have noticed so far is that the PCIe present pins (one short gold one is visible at the bottom) unfortunately has a track routed down which was used for the hard gold plating process. For future production runs, it's a good idea to route this signal to the edge of the board so that it doesn't have to be scratched off manually. What's the point of a short pin when it's as long as all the rest?

I'll be posting photos of the completed board soon for some eye candy!

INA219 on ARM Linux (Wandboard)

I mentioned in a prior post that I integrated an INA219 power measurement IC onto the power distribution board of our Wandboard cluster. It took quite a bit of scratching around to get it to work though! It's actually very simple so here are the steps in case they save someone else a few hours.

The first step is to compile the kernel with TI INA* support. This can be as a module or built-in. I chose the module route so that I can check that the module loaded with the lsmod command. Here's a screenshot of the Linux menuconfig showing the option you need to enable:

Loading the Module and Setup

Once your system is back up and running, it's a simple matter of loading the module and doing some setup. You could put this setup stuff in a script that runs on startup.

modprobe ina2xx
echo 'ina219 0x40' > /sys/bus/i2c/devices/i2c-2/new_device

I connected the I2C from the INA219 IC to the I2C3 (i2c-2, above) port of the Wandboard. I also powered the INA219 with 3.3 V, taken from the Wandboard external expansion headers.

Reading Power Values

Simply type:

cat /sys/bus/i2c/devices/2-0040/power1_input 

And you will see a large number. Divide this number by 1000000 to get the power in Watts. You can also run sensors if you have lmsensors installed. Awesome!

Freescale P4080 PCIE Development Board

Yesterday we received a Freescale P4080PCIE development board! This board features a Gen2 PCI-Express x4 link, as well as 2 SFP+ connectors for some high data throughput (~20 Gb/s) experimentation. The chip itself is an eight core PowerPC (or something pretty similar) with loads of I/O connectivity and the ability to attach loads of RAM! This is the chip you can expect to find in very high-end routers...

32 Core - 8 Wandboard Array Rack Mounted

We thought it was time to make our Wandboard Array a little bit more formal. We purchased a small tray for our rack and started work on mounting the array next to its power supply.

Thanks to the Wits Physics workshop who did a great job with mounting the power supply and brackets.

Final Wandboard Array installed in rack.

Wandboard array fresh from the workshop.

Extra Steps for Building a Wandboard (i.MX6Q) Image

I have built several Linaro based images for the Wandboard or Freescale i.MX6Q SoC, and it's a seemingly simple process of building the kernel, partitioning and installing the Linaro rootfs to a SD card. One hit's a wall when anything 'fancy' needs to be done on this image... this post aims to document how to finish off the image so that it can be used for kernel development and other advanced tasks.

Please note that this is a fairly advanced howto. Most of the concepts here can be found in other places online. The methodology behind the i.MX6 libraries comes from 'reverse engineering' the LTIB install scripts from Freescale, yocto, and lots of reading. I'm sure there is a better way to do this - but I don't know it (I'm interested to hear though)!

Linux Source Code and Modules
The Linux kernel source code should be cleaned (make ARCH=arm clean) and copied onto the image (or SD card - I'll use the terms randomly) into the /usr/src/linux directory.

Before you clean the tree, build the modules and headers and install them to a known path to be copied onto your SD card root filesystem.

make modules_install INSTALL_MOD_PATH=/some/directory/

make headers_install INSTALL_HDR_PATH=/some/directory/

Copy the contents of /some/directory to into the SD card /lib directory. You should now have a new directory: /lib/modules/3.0.35Linux+ or something similar. The headers_install directory goes into /usr.

When you boot with the new SD card, you will need to modify some symlinks (build and source) that reference the wrong place (they will be linking to directories from your build machine, which are obviously not valid any more).

cd /lib/modules/3.0.35Linux+/
ls -l

You should see the wrong links. Now to remove them and re-add them, correctly:

rm ./build
rm ./source
ln -s /usr/src/linux ./build
ln -s /usr/src/linux ./source

These commands will have to be run with sudo or as root.

Something you may also have to do, in case you cleaned the kernel source tree too thoroughly, or you cross compiled (the kernel scripts directory is full of x86 binaries which wont work on ARM: modpost, etc.), is ensure your proper .config file is there and run:

make ARCH=arm oldconfig
make ARCH=arm prepare
make ARCH=arm modules_prepare

I've also found that doing a module build fixed up some errors!

make ARCH=arm modules

Download the i.MX6 Libraries
Download and extract the Freescale BSP somewhere onto your PC (not the Wandboard). There is a source directory with loads of .gz files for various applications. We are interested in several of these. This section will explain how to install the important ones.

A list of the files we will be working with is below. Copy them onto the SD card before booting, or SCP them across. From what I have seen, the 3.0.35-4.1.0 and 3.10.17-1.0.0 are basically the same. The 3.10.17 files can be acquired from a partial yocto installation - Google can probably help with finding the files otherwise!

imx-lib-*.tar.gz
imx-vpu-*.bin
firmware-imx-*.bin
imx-test-*.tar.gz
gpu-viv-bin-mx6q-*hfp.bin (the version from the Freescale BSP seems to be soft-fp... it won't work.)
gpu-viv-g2d-*.bin
fsl-gpu-sdk-*.bin

Some of these are bins, with things to agree to before they extract. Go ahead and extract everything in preparation for the installation steps.

Install the i.MX6 Libraries
First, a few exports to make our lives easier:

export KERNELDIR='/usr/src/linux'

export INCLUDES='-I$KERNELDIR/include -I$KERNELDIR/drivers/mxc/security/rng/include -I$KERNELDIR/drivers/mxc/security/sahara2/include'

As per the usual, make sure the KERNELDIR variable points to your specific kernel directory. Note that if you have installed the kernel headers, etc. properly in the steps above, you may not need to do these exports and you can leave the INCLUDE= part out of the make commands below.

firmware:

Simply copy the contents of the firmware-imx* directory into /lib so that you have new files in /lib/firmware/vpu, etc.

imx-lib:

From the imx-lib directory:

make -j1 PLATFORM="IMX6Q" INCLUDE="$INCLUDES"

sudo make PLATFORM="IMX6Q" install 

If all compiled and copied, you should now see a bunch of new libraries in /usr/lib! Congratulations!

imx-vpu:

Even if you don't want to use the VPU, this is a dependency for the imx-tests. From the imx-vpu directory:

make -j1 PLATFORM="IMX6Q" INCLUDE="$INCLUDES"

sudo make PLATFORM="IMX6Q" install 

If all compiled and copied, you should now see a bunch of new libraries in /usr/lib! Congratulations!

gpu-viv-bin and gpu-viv-g2d:

Copy the contents of the archives into your root. /opt and /usr will now contain new files.

imx-test:

From the imx-test directory:


make -j1 PLATFORM="IMX6Q" test

At this point you can run some of the unit tests that compiled successfully (not all will have) from the ./platform/IMX6Q/autorun*.sh files.

sudo make PLATFORM="IMX6Q" install 

If all compiled and copied, you should now see a bunch of new libraries in /usr/lib! Congratulations!

gpu-viv-bin:

The GPU drivers and binaries are closed source, so it's a matter of extracting the files into the correct place. Search for gpu-viv-*.gz and copy it onto your Wandboard.

Boot, extract it and cd into the new directory. You will see an 'opt' and a 'usr' directory. Run the following commands:

sudo cp -Rv ./opt/* /opt/
sudo cp -Rv ./usr/* /usr/

Make sure the files copied into the correct places.

Upgraded Wandboard Cluster

We have invested in a few more Wandboards to upgrade our mini-cluster. Since I repurposed two of the existing boards for my PCI-Express research, we now have 8 Wandboards in the cluster which makes 32 cores! A little more power for scientific applications on ARM...

Besides having more processing power available, I have made some enhancements to the power distribution board. The thin purple, pink and red wires in the background will be connected to one of the Wandboard I2C inputs to enable easy and accurate voltage, current and power measurements of the DC supply to the cluster. The excellent TI INA219 chip is used for this. It's possible to read these values from the Linux command-line so we can use a shell script to plot power values directly to a file!

Dual Wandboard PCI-Express Connector Complete

The Wandboard PCI-Express adpater is complete! I'm in the process of building a new kernel that supports the PCI-Express RC and Endpoint to start testing the Freescale i.MX6 Quad PCI-Express capabilities.

Here are some photos of the completed board with two Wandboards attached! In the last photo you can see our (now slightly diminished) Wandboard cluster. We are waiting for another five Wandboards to make it 40 cores but no suppliers seem to have stock at the moment...

Wandboard PCI-Express Connector PCB Photos

Today I took delivery of the first (and hopefully last if it's bug-free) version of the dual Wandboard PCI-Express adapter. I plan on having it soldered and ready for testing by early next week. Check out the photos!

Wandboard PCI-Express Adapter: Update

It's been a while since I last posted about the Wandboard PCI-Express adapter I have been working on... I decided to redesign the PCB to be more compact. This saves manufacturing costs and it looks better, in my opinion.

The PCB has been sent for manufacture so hopefully in a week or two I can post some photos! Shortly after that - assuming everything goes according to plan - I'll post some results for the PCI-Express performance of the Freescale i.MX6 SoC. I don't think the PCI-Express interface to the Wandboard has been tested by anyone, so hopefully it works...