The average building in business, industry, or academia simply won’t support the unusual requirements of the latest generation of supercomputers. Lawrence Livermore’s new Terascale Simulation Facility not only supports those requirements today, but is designed to take weird science 35 years down the road.

In this issue’s ExtremeTech I: Building a Better World with Terascale and Petascale Supercomputers?, we looked at terascale and petascale computing and touched upon related issues of global competitiveness.

In terms of speed, BlueGene/L recently beat is own record set only six months ago (92 teraflops)—183.5 teraflops. BG/L’s hardware will soon double in size and is expected to reach 367 teraflops later this year. As of last month, Purple has surpassed 100 trillion floating-point operations per second.

Putting things into a different perspective, let’s take a look just at the ‘slower’ Purple. It has over 12,000 Power5 CPUs, 50 TB of memory, more than 8,000 hard disk drives, weighs more than 300 tons, and is interconnected with 200 miles of cable. When finally assembled (Will it ever be ‘final’?), Purple will require 4.8 megawatts of power—enough to feed 4,800 average homes—and is expected to generate more than 16 million BTUs per hour in heat.

Imagine the construction and architectural considerations of such a project. Well, Lawrence Livermore National Laboratories already has, with its new $91M Terascale Simulation Facility (TSF) which houses both BlueGene/L and Purple computers.

As design and construction manager for the facility, Anna Maria Bailey, an electrical engineer, found herself in a unique position of clearly placing function over form. “Most architects want to dictate the design. The architectural features had to take a back seat,” however, due to electrical and mechanical requirements of the behemoth supercomputers.

On the second floor of the four-story, 23,504 square-foot TSF, Purple will reside on the west room while BG/L is setup in the east. A wall separating the two rooms can be removed to create one large room if such a need arises. The second floor also includes offices, lab space, and areas for vendors and computer engineers to maintain and service the computers. On the first floor, a mechanical utility room has 28 air-handling units which push 80,000 cfm of 13 deg C air to the second floor through grated and perforated floor tiles. After cooling the computers, the air continues upward, forced into large plenums for recirculation.

The computer room and office tower cooling systems call upon four 1,200-ton chillers, one 400-ton chiller, and cooling towers capable of handling 8,176 liters of water per minute. Large water pipes are suspended between the first and second floors, just in case it’s necessary to shift from air- to liquid-cooling in the future.

Clear-span construction techniques of the supercomputer rooms eliminate columns and other potential obstructions. A single obstruction might require increased cable length—which isn’t possible since all hardware and communication cables must be of the same length (25 meters) since signals must travel consistently and predictably. When the distance between components is less than 25 meters, excess cable can be routed through the also-cooled, 1.2-meter raised subfloor.

TSF’s designed power capacity is 25 megawatts—enough to power a city of 16,000 to 20,000 people. Almost 5 kilometers of trenches were dug to accommodate power lines between the TSF and two substations, one of which is a backup.

Two theaters—one for classified and the other for non-classified use—help scientists, engineers, and others visualize the enormous quantities of data (including 3D simulations) generated by the supercomputers.

TSF has been designed for a 35-year lifespan.

Now that’s computing infrastructure.

Terascale Simulation Facility: Built for Flexibility
http://www.llnl.gov/str/JanFeb05/Atkinson.html

OTHER SOURCES

BlueGene/L Retains Its Title as World’s Fastest Supercomputer
http://www.llnl.gov/pao/news/news_releases/2005/NR-05-06-12.html

IBM’s BlueGene Hits Warp Speed
http://www.businessweek.com/technology/content/mar2005/tc20050325_0261_tc024.htm

IBM’s Purple Supercomputer Tops Teraflop Mark
http://www.internetnews.com/ent-news/article.php/3522321

IBM Purple Surpasses 100 Trillion Floating-Point Operations per Second
http://www.enterprisenetworksandservers.com/monthly/art.php/1622

Top 500 Supercomputer Sites
http://www.top500.org/lists/plists.php?TB=1&M=06&Y=2005