Building ATLAS 3.11 on ARM with gcc 4.9

I am in the process of re-running our HPL benchmarks on the ARM boards. The latest version of ATLAS, 3.11.32 has several optimisations for ARM and specifically ARMv8 which I would like to compare to the older and more stable ATLAS 3.10. I also discovered that gcc 4.9.2 makes quite a big difference in the CoreMark benchmark on all of the platforms we have available.

This post is just some quick documentation for myself as well as others as to how to get ATLAS to compile on the Wandboard, Cubieboard A20 and the Applied Micro X-Gene (Mustang).

In all cases I found that the recipe for success is to manually specify the compilers. Of course, setting the CPU frequency governor to performance is also required.

The Tegra-K1 compile doesn't seem to be working yet. I'll post as soon as I'm successful!

On all of the boards the following commands work to set the CPU governors:
sudo su
echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

and on the Tegra-K1 it's required to turn off the power saver core:
echo 0 > /sys/devices/system/cpu/cpu0/cpuquiet/...

When you are done setting up the CPU governors, find out the running frequency and exit the su:
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
exit

Go ahead and create a directory where you would like to download and build ATLAS to. Inside that directory, fetch the latest unstable version of atlas and if required, rename it.
wget http://sourceforge.net/projects/math-atlas/files/Developer%20%28unstable%29/3.11.32/atlas3.11.32.tar.bz2/download

Extract it and head to a build directory:
tar xf ./atlas3.11.32.tar.bz2
cd ATLAS
mkdir build
cd build

Now the magic happens... The configure command has loads of options but I find that in the unstable (3.11) range, the defaults work well for ARM. It just needs a little help. We have to manually specify the CPU frequency and since I am using gcc-4.9 and gfortran-4.9 I force them. I found that even if you tell the system that gcc and gfortran are the respective -4.9 versions, it doesnt always work.

Go ahead and type the following command to do the configure:
../configure -m <fpu_freq> -C acg gcc-4.9 -C if gfortran-4.9 -D c -DWALL

Everything should work out with no errors. If there are, then you need to delete the contents of the build directory and start again. Try different configure flags.

When the configure is done with no errors simply run

make build

and give it an hour or two!

Interview with the SABC

This week our HEP group had a great visit from the SABC. Their crew were filming a piece for the show “Teenagers On a Mission” (Tomz). They were very interested in the 3D printing development that we have been doing for the Mini PC project. We were able to outline the whole 3D printing process and give them quick demonstration using our 3D scanning and printing setup in the lab.

Thomas Wrigley giving his take on 3D printing in South Africa

“Tomz, part factual, part fictional with some drama, comedy and science fiction will be broadcast in a fair spread of Xhosa, Sotho, Sepedi, English and Zulu with full subtitles in English”

 Airs every week: Monday 4:00-4:30pm, Tuesday 4:00-4:30pm on SABC 1

Lectures by Dr Peter Braam

Dr Peter Braam, the creator of the massively parallel Lustre file-system is in South Africa for a few days to lecture on topics related to software engineering, high performance computing and of course how these topics are relevant to the Square Kilometer Array project.

The series of lectures, organised by the Center for High Performance Computing (CHPC) was originally intended for just the Cape Town region. Due to a high level of interest as well as the new high throughput electronics lab at The University of the Witwatersrand, he spent the day in Johannesburg too.

A great lunch was served and attendance was good, with over 30 people attending the afternoon event. Dr Braam lectured on software engineering practices and also on the state of high performance computing storage requirements which rang several bells in line with the groups research.

I would have loved to post a photo of the event but unfortunately my phone corrupted the image. These things happen...

Cooperation with the Centre for High-performance Computing

Prof. Ekow Otoo and myself were invited to attend a meeting with employees of the Centre for High-performance Computing (CHPC) and the Department of Science and Technology (DST) on January 29th. The aim of the meeting was to discuss the strategic objectives for the next three years. The CHPC plays a central role in providing and facilitating the use of large computing resources to a wide range of disciplines.

More details on the meeting are available at the CHPC website:

http://www.chpc.ac.za/index.php/news2/189-planning-for-the-next-three-years

The CHPC hosts 1200 cores and about 800 TB of disk the SA-ALICE and SA-ATLAS clusters. It is intended to sign a Memorandum of Understanding with CERN on the creation of a Tier2 centre that complies with the corresponding requirements. We are all very excited at the prospects.

Below is a picture of the CHPC Director, Dr Happy Sithole, standing next to the above mentioned resources that have been earmarked for the SA-ALICE and SA-ATLAS clusters.

Below few pictures of the CHPC's computing facilities:

Below is the certificate of the CHPC been given a TOP500 certificate in 2010.

Dr. Happy Sithole and myself have discussed areas of cooperation between the CHPC and our High-throughput electronics laboratory (HTEL). The Advanced Computing  Engineering Lab (ACE) deals with new technologies a the CHPC. It was felt that ACE and the HTEL could work on high-throughput  Below are pictures of a the front and back-plane of a 100 Mbps system that has been donated to the CHPC and that the HTEL could also profit from.

The Director of the Centre for High Performance Computing visits Wits electronics lab

The Director of the Centre for High Performance Computing (CHPC), Dr Happy Sithole and the Wits Deputy Vice-Chancellor for research, Prof. Zeblon Vilakazi have visited today the High-throughput Electronics Lab (HTEL) at the School of Physics. Some of our group members currently abroad connected via CERN's vidyo.

Dr. Sithole has shown great interest in the research activities of the students and young researchers of our team. The CHPC is interested in collaborating in the area of high-throughput computing with ARM processors and GPUs. One of the challenges that High-performance computing displays is the ability to move the high volumes of data to performant processors with low latency. The HTEL is devising solutions to this problem based on local designs. The involvement of the HTEL in the upgrade activities of the Tile Calorimeter of the ATLAS detector is a critical driver for these activities.

The function started by introducing Dr. Sithole to the group members and with a presentation about the main activities of the HTEL: FGPA-based electronics, high-throughput general purpose electronics, GPGPUs, ATCA systems, development of low cost computers and electronics radiation, high-throughput data analysis.

Below is a group picture taken at the HTEL. First row, centre, one can see Dr. Sithole, the Head of the School of Physics, Prof. John Carter and Prof. Vilakazi. Missing in the picture are Dr. Deepak Kar, Pablo Moreno, Thokozani Khumalo, Dr. Xifeng Ruan, Dr. Luis March, Shell-may Liao, Guillermo Hamity, Kehinde Tomiwa, the technical support staff and last, but far from least, Prof. Elias Sideras-Haddad.

Prof. Vilakazi addresses the students, referring to the HTEL as a "success story":

The rest of us are happy to listen:

Attended the "Big Switch On" event, the Boitumelong secondary school, the first school in School in South Africa going digital

Prof. Bea Lacquet, the Wits Deputy Vice Chancellor of Information, Knowledge and Infrastructure Management and me attended the "Big Switch On" event. The Boitumelong secondary school, in Tembisa, about 50 km north-east of Johannesburg:

http://www.boitumelong.co.za/

has become the first public school in South Africa to go digital. This was a high-profile event organised by the Member of the Executive Council (MEC) for Education of the Gauteng Province, Panyaza Lesufi and the Government of the Gauteng province. The Deputy President of South Africa, few Ministers and deputy Ministers, the  Premier of the Gauteng Government, city Majors and other personalities attended the event.

Managed to take a peak at some of the classes. One can see that each student is assigned a tablet and each class has digital boards. The school is connected to the backbone of Gauteng with a 1 Gb/s network, which is quite a bandwidth, taking into account the location of the school. This school, together with other 6 in the area will serve as models to roll over the implementation of connectivity and computers in the classroom. Gauteng has about 2100 state schools.

The tent was packed. Many enthusiastic students. Tons of fun, indeed.

MEC Lesufi opens the main event after plaque unveilings in different parts of the school and officials from central Government have taken their seats

There was even a bit of music

and some dancing

The Minister of Basic Education, Angelina Motshekga, addressing the audience and introducing the Deputy President of South Africa

The Deputy President of South Africa, Cyril Ramaphosa, addresses the audience.

Managed to grab the MEC's attention

Higgs (N)NLO MC and Tools Workshop for LHC RUN-2 at CERN

The  (N)NLO MC and Tools Workshop for LHC RUN-2 at CERN started on Wednesday, December 17th and it will run till Friday December 19th.

https://indico.cern.ch/event/345455/other-view?view=standard

The progress in MC development in the past year is impressive. Looking back 10 years ago progress is simply amazing.

Announcement of the Workshop on High Energy Particle Physics, iThemba LABS - Gauteng, February 11th - 13th, 2015.

Dear Colleagues

We are happy to announce that the Workshop on High Energy Particle Physics will take place at iThemba LABS - Gauteng, February 11th - 13th, 2015.

The goal of the workshop is to provide an opportunity for students and young researchers to give presentations and to write proceedings. A morning plenary session, followed by 15+10 minute presentations, shall be the format across our three days. Topics to be covered will be high-energy theory and phenomenology (heavy ions, pp, ep, ee, collisions), ALICE and ATLAS physics. We envision providing Honours and undergraduate students with the chance to give poster presentations.

The deadline for the submission of abstracts is January 15th, 2015. The deadline for submission of proceedings is February 28th, 2015.

The event will be catered. Funding is available for students outside the Gauteng Province for travel and accommodation arrangements. The deadline to apply for financial support is January 15th 2014.

More details are available at

http://hep.wits.ac.za/HEPPW2015.php

Best regards

The Local Organising Committee

Zinhle Buthelezi (iThemba LABS)
Alan Cornell (NITheP/Wits, Co-chair)
Andrew Hamilton (UCT)
Deepak Kar (Wits)
Bruce Mellado (Wits, Co-chair)
Elias Sideras-Haddad (Wits)
W. A. Horowitz (UCT)
Sahal Yacoob (UKZN)

Visiting iThemba Labs in Cape Town: Investigating capabilities of Single Even Upset studies with protons and neutrons

Milos Lokajicek, Ivo Polak, from the Institute of Physics of the Czech Academy of Sciences, and myself visited the beam-lines and halls of iThemba Labs in Cape Town. Siegfried Fortsch organised the visit. We met with nuclear physicists involved in single-event upset (SEU)studies for aerospace-related projects. We also met with the head of the accelerator division, Lowry Conradie. The aim of the visit is to learn about the capabilities and conditions for SEUs of electronics. In short:

Protons can be provided with energies up to 200 MeV and currents ranging between 0.5 nA and multiples of micro-A (which is far more than we need). Beams with 66 MeV can be provided on parasitic mode when ion implantation and medical physics do not use beams. Beams can be defocused to 1cm x 1cm with reasonable homogeneity.  The A-line is preferred for proton beams (see pictures below).

Monochromatic neutrons can be provided in the range between 40 and 200 MeV with fluxes that ranges between 4.7 10^4 cm^{-2} s^{-1} (at 66 MeV) and 4.6 10^{3} cm^{-2} s^{-1} (at 200 MeV). The beam size is about 10 cm x 10 cm. Dose measurements are provided by the lab. The D-line is preferred for neutron beams (see pictures below). It is important to note that the energy range provided is unique and the metrology community comes regularly to iThemba to perform measurements.

Below is a picture of the head of the accelerator division, Lowry Conradie. To the right, on the board is the schematic of the different facilities in the lab.

A close up of the facilities at iThemba - Cape. The "star" corresponds to the cyclotron.

Below is a picture of the run control

Siegfried Fortsch points to the current monitor. At the time of the picture over 200 micro-A of protons were being provided to the ion implantation folks.

Below is a picture of the A-line (preferred for proton studies). In blue is the vacuum chamber.

Quadrupole magnets in the A-line, upstream of the vacuum chamber

Below is a picture of the feedthroughs of the vacuum chamber on the A-line.

Opened the vacuum chamber in the A-line. Below is a future of the inside of the chamber. To the left is the orifice of the promo beam. At the centre is a possible location of the target. After discussion with experts placing the samples holders on one of the arms seems the preferred option.

A close-up of one of the arms where we will place the the sample holders. Siegie will pass the drawings for the workshop at Wits to design the holder.

Inside the hall of the D-line (neutron beam). Neutron flux measurements are provided by the lab.

Close-up of the table where the samples will be located on the D-line.

Below is a picture of the data acquisition system (analog) from the different halls and lines.

Many thanks to Siegie for setting this up!

Wits attending the Tile Calorimeter Upgrade week in Valencia, hands-on session

Oscar Kureba, Pablo Moreno, Robert Reed, Xifeng Ruan and myself are attending the the Tile Calorimeter Upgrade week in Valencia. Matthew Spoor had to take off back to SA due to visa restrictions and can't attend. Yesterday Oscar, Pablo, Robert and Xifeng gave a bunch of tutorials to the community, see the snapshot of the agenda. The hands-on session took place at the lab in Valencia and was attended by a number of Tile Calorimeter experts. This session was quite important to get useful feedback for future development.