Titan (supercomputer)


Titan or OLCF-3 was a supercomputer built by Cray at Oak Ridge National Laboratory for use in a variety of science projects. Titan was an upgrade of Jaguar, a previous supercomputer at Oak Ridge, that uses graphics processing units in addition to conventional central processing units. Titan was the first such hybrid to perform over 10 petaFLOPS. The upgrade began in October 2011, commenced stability testing in October 2012 and it became available to researchers in early 2013. The initial cost of the upgrade was US$60 million, funded primarily by the United States Department of Energy.
Titan was eclipsed at Oak Ridge by Summit in 2019, which was built by IBM and features fewer nodes with much greater GPU capability per node as well as local per-node non-volatile caching of file data from the system's parallel file system.
Titan employed AMD Opteron CPUs in conjunction with Nvidia Tesla GPUs to improve energy efficiency while providing an order of magnitude increase in computational power over Jaguar. It used 18,688 CPUs paired with an equal number of GPUs to perform at a theoretical peak of 27 petaFLOPS; in the LINPACK benchmark used to rank supercomputers' speed, it performed at 17.59 petaFLOPS. This was enough to take first place in the November 2012 list by the TOP500 organization, but Tianhe-2 overtook it on the June 2013 list.
Titan was available for any scientific purpose; access depends on the importance of the project and its potential to exploit the hybrid architecture. Any selected programs must also be executable on other supercomputers to avoid sole dependence on Titan. Six vanguard programs were the first selected. They dealt mostly with molecular scale physics or climate models, while 25 others were queued behind them. The inclusion of GPUs compelled authors to alter their programs. The modifications typically increased the degree of parallelism, given that GPUs offer many more simultaneous threads than CPUs. The changes often yield greater performance even on CPU-only machines.

History

Plans to create a supercomputer capable of 20 petaFLOPS at the Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory originated as far back as 2005, when Jaguar was built. Titan will itself be replaced by an approximately 200 petaFLOPS system in 2016 as part of ORNL's plan to operate an exascale machine by 2020. The initial plan to build a new 15,000 square meter building for Titan, was discarded in favor of using Jaguar's existing infrastructure. The precise system architecture was not finalized until 2010, although a deal with Nvidia to supply the GPUs was signed in 2009. Titan was first announced at the private ACM/IEEE Supercomputing Conference on November 16, 2010, and was publicly announced on October 11, 2011, as the first phase of the Titan upgrade began.
Jaguar had received various upgrades since its creation. It began with the Cray XT3 platform that yielded 25 teraFLOPS. By 2008, Jaguar had been expanded with more cabinets and upgraded to the XT4 platform, reaching 263 teraFLOPS. In 2009, it was upgraded to the XT5 platform, hitting 1.4 petaFLOPS. Its final upgrades brought Jaguar to 1.76 petaFLOPS.
Titan was funded primarily by the US Department of Energy through ORNL. Funding was sufficient to purchase the CPUs but not all of the GPUs so the National Oceanic and Atmospheric Administration agreed to fund the remaining nodes in return for computing time. ORNL scientific computing chief Jeff Nichols noted that Titan cost approximately $60 million upfront, of which the NOAA contribution was less than $10 million, but precise figures were covered by non-disclosure agreements. The full term of the contract with Cray included $97 million, excluding potential upgrades.
The yearlong conversion began October 9, 2011. Between October and December, 96 of Jaguar's 200 cabinets, each containing 24 XT5 blades, were upgraded to XK7 blade while the remainder of the machine remained in use. In December, computation was moved to the 96 XK7 cabinets while the remaining 104 cabinets were upgraded to XK7 blades. ORNL's external ESnet connection was upgraded from 10 Gbit/s to 100 Gbit/s and the system interconnect was updated. The Seastar design used in Jaguar was upgraded to the Gemini interconnect used in Titan which connects the nodes into a direct 3D torus interconnect network. Gemini uses wormhole flow control internally. The system memory was doubled to 584 TiB. 960 of the XK7 nodes were fitted with a Fermi based GPU as Kepler GPUs were not then available; these 960 nodes were referred to as TitanDev and used to test code. This first phase of the upgrade increased the peak performance of Jaguar to 3.3 petaFLOPS. Beginning on September 13, 2012, Nvidia K20X GPUs were fitted to all of Jaguar's XK7 compute blades, including the 960 TitanDev nodes. In October, the task was completed and the computer was finally renamed Titan.
In March 2013, Nvidia launched the GTX Titan, a consumer graphics card that uses the same GPU die as the K20X GPUs in Titan. Titan underwent acceptance testing in early 2013 but only completed 92% of the tests, short of the required 95%. The problem was discovered to be excess gold in the female edge connectors of the motherboards' PCIe slots causing cracks in the motherboards' solder. The cost of repair was borne by Cray and between 12 and 16 cabinets were repaired each week. Throughout the repairs users were given access to the available CPUs. On March 11, they gained access to 8,972 GPUs. ORNL announced on April 8 that the repairs were complete and acceptance test completion was announced on June 11, 2013.
Titan's hardware has a theoretical peak performance of 27 petaFLOPS with "perfect" software. On November 12, 2012, the TOP500 organization that ranks the world's supercomputers by LINPACK performance, ranked Titan first at 17.59 petaFLOPS, displacing IBM Sequoia. Titan also ranked third on the Green500, the same 500 supercomputers ranked in terms of energy efficiency. In the June 2013 TOP500 ranking, Titan fell to second place behind Tianhe-2 and to twenty-ninth on the Green500 list. Titan did not re-test for the June 2013 ranking, because it would still have ranked second, at 27 petaFLOPS.

Hardware

Titan uses Jaguar's 200 cabinets, covering 404 square meters, with replaced internals and upgraded networking. Reusing Jaguar's power and cooling systems saved approximately $20 million. Power is provided to each cabinet at three-phase 480 V. This requires thinner cables than the US standard 208 V, saving $1 million in copper. At its peak, Titan draws 8.2 MW, 1.2 MW more than Jaguar, but runs almost ten times as fast in terms of floating point calculations. In the event of a power failure, carbon fiber flywheel power storage can keep the networking and storage infrastructure running for up to 16 seconds. After 2 seconds without power, diesel generators fire up, taking approximately 7 seconds to reach full power. They can provide power indefinitely. The generators are designed only to keep the networking and storage components powered so that a reboot is much quicker; the generators are not capable of powering the processing infrastructure.
Titan has 18,688 nodes, each containing a 16-core AMD Opteron 6274 CPU with 32 GB of DDR3 ECC memory and an Nvidia Tesla K20X GPU with 6 GB GDDR5 ECC memory. There are a total of 299,008 processor cores, and a total of 693.6 TiB of CPU and GPU RAM.
Initially, Titan used Jaguar's 10 PB of Lustre storage with a transfer speed of 240 GB/s, but in April 2013, the storage was upgraded to 40 PB with a transfer rate of 1.4 TB/s. GPUs were selected for their vastly higher parallel processing efficiency over CPUs. Although the GPUs have a slower clock speed than the CPUs, each GPU contains 2,688 CUDA cores at 732 MHz, resulting in a faster overall system. Consequently, the CPUs' cores are used to allocate tasks to the GPUs rather than directly processing the data as in conventional supercomputers.
Titan runs the Cray Linux Environment, a full version of Linux on the login nodes that users directly access, but a smaller, more efficient version on the compute nodes.
Titan's components are air-cooled by heat sinks, but the air is chilled before being pumped through the cabinets. Fan noise is so loud that hearing protection is required for people spending more than 15 minutes in the machine room. The system has a cooling capacity of 23.2 MW and works by chilling water to 5.5 °C, which in turn cools recirculated air.
Researchers also have access to EVEREST to better understand the data that Titan outputs. EVEREST is a visualization room with a 10 by 3 meter screen and a smaller, secondary screen. The screens are 37 and 33 megapixels respectively with stereoscopic 3D capability.

Projects

In 2009, the Oak Ridge Leadership Computing Facility that manages Titan narrowed the fifty applications for first use of the supercomputer down to six "vanguard" codes chosen for the importance of the research and for their ability to fully utilize the system. The six vanguard projects to use Titan were:
In 2013 thirty-one codes were planned to run on Titan, typically four or five at any one time.

Code modifications

The code of many projects has to be modified to suit the GPU processing of Titan, but each code is required to be executable on CPU-based systems so that projects do not become solely dependent on Titan. OLCF formed the Center for Accelerated Application Readiness to aid with the adaptation process. It holds developer workshops at Nvidia headquarters to educate users about the architecture, compilers and applications on Titan. CAAR has been working on compilers with Nvidia and code vendors to integrate directives for GPUs into their programming languages. Researchers can thus express parallelism in their code with their existing programming language, typically Fortran, C or C++, and the compiler can express it to the GPUs. Dr. Bronson Messer, a computational astrophysicist, said of the task: "an application using Titan to the utmost must also find a way to keep the GPU busy, remembering all the while that the GPU is fast, but less flexible than the CPU." Moab Cluster Suite is used to prioritize jobs to nodes to keep utilization high; it improved efficiency from 70% to approximately 95% in the tested software. Some projects found that the changes increased efficiency of their code on non-GPU machines; the performance of Denovo doubled on CPU-based machines.
The amount of code alteration required to run on the GPUs varies by project. According to Dr. Messer of NRDF, only a small percentage of his code runs on GPUs because the calculations are relatively simple but processed repeatedly and in parallel. NRDF is written in CUDA Fortran, a version of Fortran with CUDA extensions for the GPUs. Chimera's third "head" was the first to run on the GPUs as the nuclear burning could most easily be simulated by GPU architecture. Other aspects of the code were planned to be modified in time. On Jaguar, the project modeled 14 or 15 nuclear species but Messer anticipated simulating up to 200 species, allowing far greater precision when comparing the simulation to empirical observation.