Thursday, 30 April 2015

UCT Workshop: High Performance Computing with CUDA 2015

Last week I attended the third annual GPGPU conference at the University of Cape Town.  Speakers included two international NVIDIA fellows, John Stone and Manuel Ujaldón from the Universities of Malaga and Illinois, as well as local speaker Bruce Merry who is working on GPU parallel processing with the Square Kilometer Array. Relatively new to the world of GPGPU myself, I was exposed to a privileged amount of expert knowledge. John, Miguel and Bruce represent a well of insight into the field of parallel processing - both technical aspects as well as a familiarity with the politics of GPUs and the microprocessor industry as a whole. 

There was a theme throughout the lectures of achieving processor performance excellence through a close relationship between programmer and hardware. Learning to harness the power of potentially millions of independent processing units requires attention to memory management and inter-processor relationships. GPUs have several types of memory, varying by size and speed of access and how the many processing units inside share all that memory must be carefully planned. To get the best possible results, one needs to ensure that as many processing units as possible are active at any given time. However, as John pointed out more than once, one cannot measure performance in GFLOPs only, as this metric does not necessarily represent 'scientific throughput' in a simple way. Perhaps the most valuable aspect of the workshop was being privy to John and Bruce's code and coding thought processes. John has done some interesting work on taking quantum-chemical graphic simulation generation from the realm of static image generation to real-time interactive animations. John and Manuel have been working with GPUs since GPGPU's early days, and as such talked us through both the future and the past of GPGPU. Parallel processing does not follow the CPU pattern of ever faster clock rate and more advanced control architecture, but rather seeks a balance between parallel abstractions, memory management, and speed. A crucial constraint in all high-throughput computing environments, like ATLAS, is I/O throughput. Future architectures, transfer protocols (like NVLink),and processor cluster configurations aim to improve this parameter - they themselves are a function of technology as well as the cooperation of big businesses who do not necessarily benefit from cooperating with competitors. 

This information is being put to work in my research into the role GPGPU will play in the high-throughput TileCal detector. The GPU(s) I will use will be working in tandem with ARM processors, which amongst other benefits, have an open-source architecture allowing for more versatile and customised designs. If you're interested in GPGPU techniques and their role in ATLAS, keep an eye on this blog for more parallel processing posts.

Wednesday, 29 April 2015

Higgs to di-photon workshop hosted by ATLAS

CERN has recently hosted an HGamma workshop at the Business Center in St-Genis. This 2.5 day workshop (from Monday April 27th to Wednesday April 29th) was intended to assess the readiness of HGamma analyses for run2. HGamma is the name given to this particular decay channel being studied. A walkthrough for early analyses was given followed by a review of the status of combined performance and common tools. This was followed by a look at all longer term analyses which require more data.

The Wits members involved in the HGamma group include:
  1. Luis March 
  2. Mthokozisi Masuku 
  3. Bruce Mellado 
  4. Stefan von Buddenbrock 
  5. Deepak Kar 
  6. Oscar Kureba 
  7. Shell-May Liao 
  8. Robert Graham Reed 
  9. Xifeng Ruan 
  10. Kehinde Gbenga Tomiwa

The University of the Witwatersrand had a total of five presentations. The presenters and topic overviews were:

  • Luis March - This talk discussed the new Missing Transverse Momentum (ETmiss) definition that will combat the issues associated with the increase in collision energy. The presentation showed the status and development of the ETmiss for Run-II, comparing the performance of two ET miss definitions by using Monte Carlo datasets. In addition, the status of an ETmiss tool which will be used by the missing energy groups was shown.
  • Robert Reed - An overview talk about the status of the analysis framework for the Higgs to di-photon and di-bjet decay channel. This is a sub analysis of the HGamma channel. The talk included a short tutorial on how to run the analysis and the selection criteria involved. Further studies were presented regarding jet energy resolutions as well as simulation type comparisons.

  • Xifeng Ruan - Presented three talks:
  1. This talk presented the inauguration of a new study involving the Higgs and missing transverse energy (ETmiss). This is a primary candidate in the study for beyond the Standard Model. The talk involved the understanding of the missing transverse energy from the 2011, 2012 and 2015 data to look for new physics. 
  2. A summary of the recent simulation samples was shown. The simulation samples are the theoretical estimates we should observe during data taking. The ETmiss group has requested new samples for study for the new data period in 2015. 
  3. A detailed look at the future prospects of the ETmiss study was discussed. This included the intermediate and high ETmiss/BSM groups. This also included all the ongoing efforts in the group.

Tuesday, 14 April 2015

Radio Interview with the Wits-ATLAS group

Thomas Wrigley and myself were interviewed by POWERFM98.7 Radio station in Johannesburg. Great host, Lawrence Tlhabane and a nice experience. The recording of the interview is available at:

More about the radio station at.

Friday, 10 April 2015

New ADC daughter board for the ATLAS TileCal PROMETEO test-bench

An improved version of the ADC daughter board used in the PROMETEO test-bench has been in development at WITS. PROMETEO is the new test-bench system that will be used for the full certification of the upgraded front-end electronics of the ATLAS TileCal.

The board was developed to filter and digitise analog trigger signals coming from the front-end electronics of the TileCal. It makes use of two ADC571 chips to sample 16 differential analog signals at 40 MHz using a 12-bit resolution.  A Xilinx VC707 dev-board with a Virtex-7 FPGA is used for the control and data read-out of the ADC board. The 16 data channels to the FPGA have a combined data rate of 7680 Mbps.

This week we received the newly manufactured PCBs which have since been sent off for assembly. Once final testing is completed the new board will be fully integrated into the ATLAS system.

Wednesday, 8 April 2015

Press release by Wits regarding the re-start of the LHC and the role of Wits scientists

New discoveries await scientists

8 April 2015


As the world’s most powerful particle accelerator, the Large Hadron Collider (LHC) at the European Organization of Nuclear Research (CERN) in Geneva, Switzerland, re-started over the weekend physicists from the University of the Witwatersrand in Johannesburg are ready to further explore the frontiers of science that led to the discovery of the Higgs boson particle in 2012.
The circulation beams rebooted on Sunday morning, 5 April 2015 – a major milestone towards resuming proton-to-proton collisions and delivering protons at a new energy frontier before the summer in the northern hemisphere. "The return of beams to the LHC rewards a lot of intense, hard work from many teams of people," said Head of CERN’s Beam Department, Paul Collier. "It is very satisfying for our operators to be back in the driver’s seat, with what is effectively a new accelerator to bring on-stream, carefully, step-by-step.”
"Operating accelerators for the benefit of the physics community is what CERN is here for,” said CERN Director General, Professor Rolf-Dieter Heuer. 
The team reacts as the Large Hadron Collider restarts. Image: CERNThe historic observation three years ago of the Higgs boson, a puzzle piece in the Standard Model of Particle Physics, opened up possibilities of more exotic particles waiting to be discovered.
“There are many questions left unanswered in the Universe, some of which we hope to answer when the Large Hadron Collider (LHC) is once again fully operational,” says Professor Bruce Mellado from the Wits School of Physics.
An expert on the Higgs boson, Mellado – who leads the University’s involvement in the Atlas experiment at CERN - was recently appointed co-convener in the Higgs Cross-section Group at CERN that puts together the work of more than 100 theorists and experimentalists, and oversees seven subgroups that are setting up standards for the study of the properties of the Higgs boson with the data that the Large Hadron Collider will provide. Mellado has also been made a member of the Large Hadron Electron Collider (LHeC) Coordination Group – a new proposed R12-billion facility to be constructed.

Wits and the ATLAS experiment
Mellado and colleagues in the High Energy Physics (HEP) group in the Wits School of Physics are investigating peculiarities seen in the Higgs boson results made publicly available by the ATLAS experiment.
The possible explanations to these peculiarities range from a set of random statistical fluctuations, which may disappear with more data, to a much more interesting possibility of the observation in laboratory conditions of dark matter particles. This would signify a finding as important as that of the Higgs boson.
HEP will be quite active in this investigation and crucial support from the Department of Science and Technology, through the SA-CERN programme and bursaries, as well as University contributions will play a critical role in this endeavour. Professor Alan Cornell, a high-energy particle physics theorist from the Mandelstam Institute for Theoretical Physics at Wits, collaborates with experimentalists of the HEP group to devise new scenarios unexplored before. Another new member to the HEP group, Dr Deepak Kar from the University of Glasgow, will bring a complimentary dimension to the group through his knowledge and experience.
Since the start of 2013, the Wits-ATLAS group has submitted and/or published over 60 proceedings and research papers. Students Harshna Jivan, Mitchell Cox and Robert Reed have received five prizes in the same period of time. The group has been honoured with the First Time Inventor Award (2014) and the Most Cited Researcher Award (2015) by the University.

Wits played an active role during shutdown
The HEP group was very active during the shutdown of the LHC since 2013 – in order to prepare the collider for almost double the previous collision energies. Members have designed and locally produced High Voltage, LED and ADC electronic boards that are used in a mobile integrity-checking system that validates and consolidates the electronics on the ATLAS detector at the LHC. These boards along with custom software, also developed by the group, used a new design, called Prometeo, which is intended for the upgrade of the LHC expected in the year 2022.

Dealing with data explosion and development of low-cost computers for South Africa
Storing the large amount of data – in the order of a few Petabytes per second – that the LHC is expected to generate in the near future, is a major concern to physicists. In response, the HEP group also initiated the development of software tools that integrate the new hardware into the ATLAS detector control system, and the group is developing a novel way to deal with the data explosion by processing the data, in real time, using energy efficient and cost effective processors found in mobile phone devices.
These high-end electronics are being designed and developed in the new High-throughput Electronics Laboratory (HTEL) in the Wits School of Physics. The lab was inaugurated in November last year by the Wits Vice-chancellor and Principal, Professor Adam Habib, the Dean of the Faculty of Science, Professor Helder Marques, and Head of the School of Physics, Professor John Carter.
 As a spinoff of this high-end electronics research, the HTEL also houses the design of low-cost computers for schools in Gauteng. The Technology Innovation Agency has awarded the HTEL with a grant to launch a pilot project at selected schools to study and develop these devices to best suit the education environment

Wits’ efforts with ALICE
The re-start of the LHC at the new energy frontier and with greater intensity also brings reinvigorated excitement for the ALICE experiment at the LHC. Professor Zeblon Vilakazi, Deputy Vice-Chancellor: Research, leads Wits’ efforts at the ALICE experiment. The ALICE experiment seeks to investigate the state of matter that excited a millionth of a second after the Big Bang - known as the quark gluon plasma — a soup of matter particles that cooled down to form atomic nuclei that we observe today.

Synergies with nuclear physics and material sciences
Professor Elias Sideras-Haddad leads synergistic activities between high-energy physics and the HTEL with nuclear physics and material sciences. At the forefront of these investigations lies a unique set of South African accelerator-based facilities that allow scientists to reproduce the challenging conditions of radiation at the LHC. In particular, radiation damage of ATLAS detector components is studied using several techniques developed for the material sciences.