Erna van Wyk has written a nice report of the opening day of the workshop:
http://www.wits.ac.za/newsroom/newsitems/201401/22739/news_item_22739.html
http://www.wits.ac.za/newsroom/newsitems/201401/22739/news_item_22739.html
Wednesday, 29 January 2014
Tuesday, 28 January 2014
Photo of Peter Jenni with the Wits High Energy Physics group
Below is a picture of Peter Jenni with member of the Wits High Energy Physics group. Prof. John Carter, the Head of the School of Physics, is also in the photo. Missing in the picture are Oana, Itumeleng, Pablo, Rudolph, Trevor, Kieran and Reto.
The High-performance Signal and Data Processing workshop took off on Monday January 27th. The workshop puts together astronomy, astro- and particle physics to address common issues pertaining to the Big Data problem. Attendance has been very nice with over 135 people registered, out of which 60 are students from different parts of the country. Below is the workshop picture:
Tomorrow start the hands-on sessions with FPGA-based electronics.
Tomorrow start the hands-on sessions with FPGA-based electronics.
Monday, 20 January 2014
Tuesday, 14 January 2014
LHeC workshop to take place January 20-21st in Chavannes-de-Bogis, Switzerland
The 2014 LHeC workshop follows the publication of its conceptual design report (CDR) and the discovery of the Higgs boson in 2012. Recent LHeC progress concerns the study of the increase of its luminosity to 10^34cm-2s-1, the choice of the frequency of 802 MHz as well as the design towards an energy recovery test facility with up to 1 GeV of electron beam energy. The workshop will discuss progress on the LHeC physics (Higgs, top, heavy ions etc) in conjunction with the evolution and simulation of the detector design. It will consider maximising the ep luminosity in synchronous ep and pp operation at the LHC, including a heavy ion programme, and progress in the design of the ERL test facility at CERN. Very recent developments towards a future multi-TeV proton accelerator at CERN open prospects for ep/eA experimentation at yet enlarged energy. The workshop is open for contributions on the various aspects of the LHeC development and its relation to the LHC. It is prepared by an organising committee in collaboration with the working group convenors and an international advisory committee.
http://indico.cern.ch/conferenceDisplay.py?confId=278903
Thursday, 19 December 2013
FFTW Benchmarks on Cortex-A7
The FFT algorithm has many scientific uses. The most obvious uses are in radio astronomy, for the frequency analysis of signals and is vital to Software Defined Radio (SDR) which is used extensively in the Square Kilometer Array (SKA). In line with the goals of the MAC Project, I am curious about how well an ARM processor (specifically the Cortex-A7) can do FFT - which leads to these benchmarks.
I discovered some existing benchmarks of FFTW done on the Cortex-A8 and A9 by Vesperix here. I used their modified FFTW 3.2.2 for ARM NEON and also ran benchmarks using the latest official version of FFTW: 3.3.3. Both sets of results are presented below with a short discussion afterwards.
I was unable to get the FFTW 3.3.3 NEON version working. I was repeatedly hit by a segmentation fault which I think is due to different memory alignment in the newer NEON and VPPv4 FPU's. I will post these specific benchmarks when the error is resolved.
I first tested the Vesperix FFTW 3.2.2 and then the FFTW 3.3.3. In all cases I used the following configure flags for single precision and the only available timer on the ARM processor:
--enable-single --with-slow-timer
I ran non-SIMD (no NEON) test without any extra flags, and NEON SIMD tests with the flag below:
--enable-neon
I also tried out the fused multiply-add flag since the Cortex-A7 has this instruction in the VFPv4 FPU but I found that this flag actually caused performance to decrease! A short description of why this is can be found in the FFTW 3.3.3 tests section.
--enable-fma
In all cases I modified the configure script to optimise for the CPU with the '-mcpu=cortex-a7' flag. I also modified the configure script to try out different GCC FPU options where appropriate, but in general I am only presenting the fastest results in this post. The options I tried are listed below for reference:
-mfpu=neon
-mfpu=neon-vfpv4
-mfpu=vfpv4-d16
-mfpu=vfpv3-d16
I repeated the tests with NEON on a threaded version of FFTW to see at which point multiple threads (on multiple cores) makes a difference and by how much. To enable the threaded version, FFTW must be recompiled with the flag below. Note that FFTW can be compiled once with this flag and used in both a threaded or unthreaded way.
--enable-threads
I plan on running an MPI version with more threads (4 to 16) on our Cubieboard and Wandboard clusters at a later stage.
I used the script provided by Vesperix to automate the benchmarks. For the threaded tests I modified the script to contain the '-onthreads=2'. The number can be adjusted to suit the number of cores available on the system. The modified script is shown below.
#!/bin/sh
for TYPE in 'c'; do
for PLACE in 'i' 'o'; do
echo "$TYPE $PLACE 1-D powers of two (2 threads)"
for SIZE in '2' '4' '8' '16' '32' '64' '128' '256' '512' '1024' '2048' '4096' \
'8192' '16384' '32768' '65536' '131072' '262144' '524288' '1048576' '2097152'; do
./bench -onthreads=2 $OPTS ${PLACE}${TYPE}${SIZE}
done
done
for PLACE in 'i' 'o'; do
echo "$TYPE $PLACE 1-D powers of two (1 thread)"
for SIZE in '2' '4' '8' '16' '32' '64' '128' '256' '512' '1024' '2048' '4096' \
'8192' '16384' '32768' '65536' '131072' '262144' '524288' '1048576' '2097152'; do
./bench $OPTS ${PLACE}${TYPE}${SIZE}
done
done
done
Although not necessarily totally accurate in the classic FLOPS sense, it is calculated the same way in all cases, so it works as a way to compare between runs. For comparison to other algorithms, I would rather use the actual time the algorithm takes to run on a specific FFT size (N).
I ran a test of the threaded version of FFTW 3.2.2 and the results are promising for a scaled-up system. The Cubieboard2 is only a dual-core system but I plan on running MPI tests with more cores at a future date.
Note how the FMA versions have slightly lower performance. In the Benchmark Methodology section I mentioned that the --enable-fma flag actually causes performance to decrease. The reason for this is not intuitive as one would think that a Fused Multiply Add (FMA) instruction would save cycles as it replaces separate Multiply and Add instructions. In the computation of an FFT, two of the common operations are:
t0 = a + b * c
t1 = a - b * c
The way that the NEON FMA instruction works, however, is not conducive solving this. This is what happens when you use the NEON FMA:
t0 = a
t0 += b * c
t1 = a
t1 -= b * c
Notice that we have to use up two move instructions for initially setting t0 and t1. It turns out that in this specific case it's faster to just use Multiplies and Adds:
t = b * c
t0 = a + t
t1 = a - t
All in all, the FMA version does 2 Moves, 2 FMA's. The optimal version does 1 Multiply and 2 Adds. It's a small difference, one which the compiler may or may not take note of and optimise, but when done a significant number of times it makes a difference.
I discovered some existing benchmarks of FFTW done on the Cortex-A8 and A9 by Vesperix here. I used their modified FFTW 3.2.2 for ARM NEON and also ran benchmarks using the latest official version of FFTW: 3.3.3. Both sets of results are presented below with a short discussion afterwards.
I was unable to get the FFTW 3.3.3 NEON version working. I was repeatedly hit by a segmentation fault which I think is due to different memory alignment in the newer NEON and VPPv4 FPU's. I will post these specific benchmarks when the error is resolved.
System Specifications
The tests were run on a Cubieboard2 with the following specifications:
- Allwinner A20 Dual-Core Cortex-A7 SoC @ ~1GHz
- VFPv4 and NEONv2 FPU
- 256 kB L2 Cache
- 1 GB DDRIII RAM
- 8GB Class 10 MicroSD Card
- sunxi kernel 3.4.67+
- Linaro 13.04 (with GCC 4.7.3)
Benchmark Methodology
I am only presenting the results for a complex 1D FFT with powers of two and non-powers of two. These are the types of FFTs that are most useful to radio astronomy since signal phase and amplitude are represented as a complex number. I ran several sets of benchmarks with various optimisations for comparison, each of which I will describe below.I first tested the Vesperix FFTW 3.2.2 and then the FFTW 3.3.3. In all cases I used the following configure flags for single precision and the only available timer on the ARM processor:
--enable-single --with-slow-timer
I ran non-SIMD (no NEON) test without any extra flags, and NEON SIMD tests with the flag below:
--enable-neon
I also tried out the fused multiply-add flag since the Cortex-A7 has this instruction in the VFPv4 FPU but I found that this flag actually caused performance to decrease! A short description of why this is can be found in the FFTW 3.3.3 tests section.
--enable-fma
In all cases I modified the configure script to optimise for the CPU with the '-mcpu=cortex-a7' flag. I also modified the configure script to try out different GCC FPU options where appropriate, but in general I am only presenting the fastest results in this post. The options I tried are listed below for reference:
-mfpu=neon
-mfpu=neon-vfpv4
-mfpu=vfpv4-d16
-mfpu=vfpv3-d16
I repeated the tests with NEON on a threaded version of FFTW to see at which point multiple threads (on multiple cores) makes a difference and by how much. To enable the threaded version, FFTW must be recompiled with the flag below. Note that FFTW can be compiled once with this flag and used in both a threaded or unthreaded way.
--enable-threads
I plan on running an MPI version with more threads (4 to 16) on our Cubieboard and Wandboard clusters at a later stage.
I used the script provided by Vesperix to automate the benchmarks. For the threaded tests I modified the script to contain the '-onthreads=2'. The number can be adjusted to suit the number of cores available on the system. The modified script is shown below.
#!/bin/sh
for TYPE in 'c'; do
for PLACE in 'i' 'o'; do
echo "$TYPE $PLACE 1-D powers of two (2 threads)"
for SIZE in '2' '4' '8' '16' '32' '64' '128' '256' '512' '1024' '2048' '4096' \
'8192' '16384' '32768' '65536' '131072' '262144' '524288' '1048576' '2097152'; do
./bench -onthreads=2 $OPTS ${PLACE}${TYPE}${SIZE}
done
done
for PLACE in 'i' 'o'; do
echo "$TYPE $PLACE 1-D powers of two (1 thread)"
for SIZE in '2' '4' '8' '16' '32' '64' '128' '256' '512' '1024' '2048' '4096' \
'8192' '16384' '32768' '65536' '131072' '262144' '524288' '1048576' '2097152'; do
./bench $OPTS ${PLACE}${TYPE}${SIZE}
done
done
done
MFLOPS Result Interpretation:
The result provided by FFTW is 'MFLOPS': this is not true MFLOPS. It is estimated by FFTW based on an assumption of algorithmic complexity for the standard Cooley-Tukey FFT algorithm:
FFTW 3.2.2 (Vesperix):
Please examine the various graphs below. Clearly, NEON makes quite a large difference and is a 'no-brainer' for any application. I am showing one set of non-power of two benchmarks to illustrate why they should not be used.
I ran a test of the threaded version of FFTW 3.2.2 and the results are promising for a scaled-up system. The Cubieboard2 is only a dual-core system but I plan on running MPI tests with more cores at a future date.
FFTW 3.3.3 (Official):
I was unable to get the NEON version of FFTW 3.3.3 working. I was able to run benchmarks of the scalar version of the code which shows a performance improvement over the 3.2.2 scalar results. I compiled one graph comparing all the different scalar versions, with FMA instructions and without.
Note how the FMA versions have slightly lower performance. In the Benchmark Methodology section I mentioned that the --enable-fma flag actually causes performance to decrease. The reason for this is not intuitive as one would think that a Fused Multiply Add (FMA) instruction would save cycles as it replaces separate Multiply and Add instructions. In the computation of an FFT, two of the common operations are:
t0 = a + b * c
t1 = a - b * c
The way that the NEON FMA instruction works, however, is not conducive solving this. This is what happens when you use the NEON FMA:
t0 = a
t0 += b * c
t1 = a
t1 -= b * c
Notice that we have to use up two move instructions for initially setting t0 and t1. It turns out that in this specific case it's faster to just use Multiplies and Adds:
t = b * c
t0 = a + t
t1 = a - t
All in all, the FMA version does 2 Moves, 2 FMA's. The optimal version does 1 Multiply and 2 Adds. It's a small difference, one which the compiler may or may not take note of and optimise, but when done a significant number of times it makes a difference.
Conclusion
The results from this set of benchmarks are very similar to those attained by Vesperix on Cortex-A9 boards. The multi-threaded version is also significantly better for larger FFT sizes. The results at different FFT sizes are very dependant on the processor implementation details such as cache sizes and memory access times. With smaller FFT's the overhead associated with calculating the FFT is a large factor and this is clearly visible up to sizes of 128.
The scalar results for FFTW 3.3.3 are better than those from 3.2.2 so it is logical to assume that the newer version's NEON performance will be better as well.
Since FFTW works by creating a 'plan' before actually calculating the FFT, it chooses to not use more than one thread in the multi-threaded version before a certain FFT size. This is clearly visible as it chooses to use multi-threading at greater than size 128. The overhead associated with doing this causes the result to be poor at size 256 and based on the results, multi threading should only be enabled for sizes over 1024.
The power usage of the Cortex-A7 processor is lower than that of the Cortex-A9 and so if a large cluster of these devices is used for a computational task such as radio astronomy, one could speculate that it may be worthwhile to use more Cortex-A7's over fewer Cortex-A9's since the performance is similar.
Friday, 6 December 2013
Current Measurement Board
Since we are interested in power measurements for the different ARM platforms, I decided to quickly design and build a simple current measurement board that we can connect an oscilloscope to to plot current (and hence calculate power with a corresponding voltage measurement on the second channel).
The concept is based on Ohms law: the voltage across a resistor is equal to the resistance multiplied by the current through it. The board design in the schematic caters for a known 0.01 Ohm resistor with a 1% tolerance and a gain of 100 with a 1500 Hz low pass filter. The gain of 100 results in a voltage output that is proportional to the current: 1 A current gives 1 V output. This is so that we can use the oscilloscopes built in multiplication to see power in real-time. The low pass filter is there so that we don't get too much noise on the current measurement, but still enough response to see a spike when we start benchmarks.
I have posted images of the schematic and photos of the finished board. If you would like the Cadsoft Eagle design files, I'm happy to share them - just put a request in the comments below. Something I should also mention is the op-amp is pretty high end, unnecessarily! To be honest, it is a free sample that I gratefully received from Maxim, so I used it...
Specifications:
Input Current: ~50 mA - 5 A
Output Voltage: ~50 mV - 5 V (dependant on how close to the negative / ground rail the op-amp can go)
Frequency Response: -3 dB @ 1591 Hz
Supply Voltage: 5 V (dependant on the op-amp)
Power Loss in 0.01 Ohm Resistor @ 5 A: 0.25 W (one can use a spreadsheet to compensate for this error)
The concept is based on Ohms law: the voltage across a resistor is equal to the resistance multiplied by the current through it. The board design in the schematic caters for a known 0.01 Ohm resistor with a 1% tolerance and a gain of 100 with a 1500 Hz low pass filter. The gain of 100 results in a voltage output that is proportional to the current: 1 A current gives 1 V output. This is so that we can use the oscilloscopes built in multiplication to see power in real-time. The low pass filter is there so that we don't get too much noise on the current measurement, but still enough response to see a spike when we start benchmarks.
I have posted images of the schematic and photos of the finished board. If you would like the Cadsoft Eagle design files, I'm happy to share them - just put a request in the comments below. Something I should also mention is the op-amp is pretty high end, unnecessarily! To be honest, it is a free sample that I gratefully received from Maxim, so I used it...
Specifications:
Input Current: ~50 mA - 5 A
Output Voltage: ~50 mV - 5 V (dependant on how close to the negative / ground rail the op-amp can go)
Frequency Response: -3 dB @ 1591 Hz
Supply Voltage: 5 V (dependant on the op-amp)
Power Loss in 0.01 Ohm Resistor @ 5 A: 0.25 W (one can use a spreadsheet to compensate for this error)
Wednesday, 4 December 2013
Wandboard PCI-Express Adapter: Preliminary PCB 3D Images
I have been designing an adapter board to connect two Wandboards via their PCI-Express ports. The design will be finished by the end of this week, after which it will be sent in for manufacturing! This post is simply to show off two quick 3D renderings of the boards so far.
We will use this board for testing and benchmarking the potential throughput of the Freescale i.MX6 processor. The gigabit ethernet on the i.MX6 is limited to ~400 Mbps according to the datasheet, but we should be able to attain close to 5 Gbps with extremely low latency using the PCI-Express Gen 2 x1 port!
More details will follow when the board has been manufactured and tests are complete!
We will use this board for testing and benchmarking the potential throughput of the Freescale i.MX6 processor. The gigabit ethernet on the i.MX6 is limited to ~400 Mbps according to the datasheet, but we should be able to attain close to 5 Gbps with extremely low latency using the PCI-Express Gen 2 x1 port!
More details will follow when the board has been manufactured and tests are complete!
Saturday, 30 November 2013
Complete Wandboard Array
Following the previous posts regarding the installation of HPL on the Wandboard and Cubieboard2 and the subsequent setup of the two Cubieboards connected and running HPL I am pleased to share that we have set up five Wandboards running Ubuntu 13.05 Server (Thanks to Martin Wild) and using MPICH2 as the MPI.
If you would like details on how to set up multiple boards please view my post of setting up the Cubieboard2 "array" here.
The Wandboards arrived with out power adapters. We decided to build our own using a normal PC 300W power supply as this provides proper grounding and if there are static discharges against the boards they will be better protected.
The cable is standard two core cabling with a plug at the end which fits the Wandboard sockets. I used extra long spacer screws so that we could stack the boards on top of each other. They had to be spaced wide enough that the heat wouldn't be an issue and so that we could get fingers at each board in case we need to add hard drives.
Now before connecting the power I had to write pre-made images to the sd card for each board Once I had one board up and running I then copied the SD card to the remaining four and set each IP address. There were a few issues with this which ill speak about at the end. Once the board were all up I connected everything together.
Now following similar procedures as my previous posts I set up HPL over a shared drive using NFS and I configured the HPL for neon and hardfp. I ran a quick test on the array using a small problem to test if all boards would indeed respond correctly. I was happy to see that all five boards showed xhpl in the processes (top) when I ran HPL.
I tried to compile ATLAS for the Cubieboard2 using neon-vfpv4 but the compiling got stuck at the L1 cache due to an infinity popping up somewhere. I will recompile that using just neon and do something similar for the Wandboard. This will improve performance quite a lot as I am using a standard ATLAS library at the moment. Once that is done I will be able to start tuning the HPL.dat file for the array.
An interesting problem came up when I copied the OS from one SD card to the other. During boot it would take exceptionally long and after finally starting up there would be no Ethernet. I checked for the adapters using ifconfig -a and they were named eth1 or eth2... not the default eth0. After doing some investigating it was quite obvious... When Linux boots up it searches for the devices and saves them in the following file:
/etc/udev/rules.d/70-persistent-net.rules
Since the hardware was changing it was appending the new hardware to the end of this list. Thus the system searched for the first one and then moved onto the next. Simply removing the content and restarting solved this issue.
Another issue was the locals. I am not sure why this one popped up but after some reading through some material I just generated the locale and reconfigured it. Using the following commands:
sudo locale-gen fi_FI.UTF-8
sudo dpkg-reconfigure locales
If you would like details on how to set up multiple boards please view my post of setting up the Cubieboard2 "array" here.
Getting things ready
The Wandboards arrived with out power adapters. We decided to build our own using a normal PC 300W power supply as this provides proper grounding and if there are static discharges against the boards they will be better protected.
 |
| Here is the first power connector we made. The green strip is a small two channel PCB board. |
 |
| Stacked array of the Wandboards |
Now before connecting the power I had to write pre-made images to the sd card for each board Once I had one board up and running I then copied the SD card to the remaining four and set each IP address. There were a few issues with this which ill speak about at the end. Once the board were all up I connected everything together.
Completed Array
 |
| Wandboard array with power and Ethernet |
 |
| Five terminals showing active processes when running HPL |
Next up
I tried to compile ATLAS for the Cubieboard2 using neon-vfpv4 but the compiling got stuck at the L1 cache due to an infinity popping up somewhere. I will recompile that using just neon and do something similar for the Wandboard. This will improve performance quite a lot as I am using a standard ATLAS library at the moment. Once that is done I will be able to start tuning the HPL.dat file for the array.
Problems Encountered
An interesting problem came up when I copied the OS from one SD card to the other. During boot it would take exceptionally long and after finally starting up there would be no Ethernet. I checked for the adapters using ifconfig -a and they were named eth1 or eth2... not the default eth0. After doing some investigating it was quite obvious... When Linux boots up it searches for the devices and saves them in the following file:
/etc/udev/rules.d/70-persistent-net.rules
Since the hardware was changing it was appending the new hardware to the end of this list. Thus the system searched for the first one and then moved onto the next. Simply removing the content and restarting solved this issue.
Another issue was the locals. I am not sure why this one popped up but after some reading through some material I just generated the locale and reconfigured it. Using the following commands:
sudo locale-gen fi_FI.UTF-8
sudo dpkg-reconfigure locales
Subscribe to:
Posts (Atom)