Cryptocurrency Miners Turn to Exotic Cooling Systems as Competition Heats Up

Amid Singapore’s humid equatorial heat, a group of researchers and faculty at the island’s main technical education institute are sweating over an unconventional cooling solution – for litecoin miners.

The institute, known more for strait-laced vocational courses, has partnered with litecoin startup CloudMining.sg to develop a liquid immersion cooling system for the firm’s miners, in conjunction with materials science giant 3M.

Neal Blackburn, who runs business development for the startup, said:

“[Other universities] we approached weren’t comfortable enough to take this on as a project. With the Institute of Technical Education, they want commercialised research, we were signing the paperwork in 48 hours.”

No wonder the institute jumped at the project. Its goal is to build a “beach cooler” sized module containing an exotic cooling fluid from 3M that quickly dissipates heat. Circuit boards for litecoin mining are submerged directly into the liquid, resulting in energy cost savings of 96%, said Blackburn.

Exotic cooling provides an edge

Miners in the cryptocurrency world are locked in a zero-sum game to discover new blocks. As competition intensifies, miners must find an increasingly elusive edge to beat their rivals. To do this, they’re now turning to expensive and specialised cooling systems that were once the exclusive preserve of cutting-edge research laboratories and military installations.

According to Michael Bedford Taylor, director of the Center for Dark Silicon at the University of California, San Diego, liquid immersion cooling is currently “extremely rare” in general-purpose data centres, but it could become a common sight in bitcoin mining installations.

“I can’t name any current machines that do immersion cooling […] I think it’s a great possibility [for the bitcoin mining industry],” he said.

Firms supplying liquid immersion cooling technology and services have reported a spike in enquiries from bitcoin miners.

3M, for example, produces a liquid called Novec that is used to cool circuit boards directly. This means the boards are submerged in the liquid, typically in an ‘open bath’ system, where they lie uncovered, for easy access.

Michael Garceau, who sells Novec to data centres on behalf of 3M, told CoinDesk:

“Enquiries have grown exponentially, and sales have grown as well. We’re seeing a transition [of bitcoin mining] to an industrial scale now.”

ASICMiner in Hong Kong uses a famously documented Novec system, while CoinTerra has said that it’s considering an immersion-based system too.

Different immersion systems

The 3M liquid works by changing from liquid to vapour at a very low boiling point, typically 49 degrees Celsius. As the submerged electronics generate heat, the liquid heats up and evaporates. The vapour rises into a condenser, cools down and then returns to the tank as liquid.

The chief competitor to Novec-based systems is liquid cooling using mineral oil.

Green Revolution Cooling (GRC), a firm based in Austin, Texas, sets up oil-based systems and counts the United States Air Force, the Department of Defense and the Tokyo Institute of Technology among its customers. It calls its immersion-cooling technology the ‘CarnotJet system’ (see image at top).

While similar in principle, oil immersion cooling is slightly more complex than Novec-filled systems.

The circuit boards are submerged in an oil-based coolant, as with Novec, but the oil has to be circulated with a pump to remove heat. The oil travels to a heat exchanger, where heat is transferred to water-filled pipes that ultimately connect to a cooling tower, located somewhere outside the building. The cooling tower then dispels the heat into the air.

Moore’s Law and miners

Bedford Taylor says chips used in bitcoin mining are now so advanced that they are hitting the limits of semiconductor fabrication, as set out by the famous dictum, Moore’s Law.

Moore’s Law states that the number of transistors on an integrated circuit will double every two years. Transistor counts on a chip are determined by the chip fabrication process. As process technology improves, it shrinks the size of the elements on a chip, allowing more transistors to be arranged on one chip.

The size of transistors on a chip is measured in nanometres. According to the UCSD’s Bedford Taylor, chips used in miners are now using the smallest transistors that can be fabricated.

He said:

“Bitcoin mining chips are now in the most advanced technology nodes of Moore’s Law: 20nm and 20nm. The first chips were in 130nm, a relatively old process.”

Bespoke chips and dark silicon

Bedford Taylor, M, ‘Bitcoin and the Age of Bespoke Silicon’, 2013

In a paper studying the progress of chips used in bitcoin mining, Bedford Taylor concludes that technological advancement in the area has been “amazingly fast” in the face of increasing difficulty. He describes four generations of bitcoin mining chips, starting at regular CPUs and ending with the ASICs (application-specific integrated circuits) that miners currently use.

The current generation of mining chips represent the first examples of a new era of “bespoke silicon” – custom-made chips produced in relatively small batches for a specific purpose – Bedford Taylor argues in a 2013 paper.

“The latest round of hardware – dedicated ASICs – have been financed, developed and deployed by bitcoin users, which is perhaps an unprecedented event in recent history. One question is whether this model can scale to other application areas and usher in a new era of bespoke silicon,” he wrote.

Bedford Taylor was one of the authors of a paper that described a phenomenon known as ‘dark silicon’.

The dark silicon problem says that, as transistor counts on chips increase, the portion of the chip that is actually utilised, given the the chip’s power constraints, drops exponentially. As a result, a portion of the chip remains without power – the dark silicon – and is not utilised.

The researcher said that some bitcoin miners have been advancing so rapidly that dark silicon is already an issue and performance has been reduced as a result.

When immersion cooling makes sense

So if liquid immersion is so effective at cooling these chips that are being pushed to the limits of current technology, why aren’t all miners using it? It turns out that immersion cooling systems only make sense when miners face certain constraints.

As Brandon Moore of GRC notes, three factors drive adoption of immersion cooling systems: expensive electricity, expensive real estate and the absence of a legacy cooling system.

“If they’ve got expensive power, are space constrained and don’t have a cooling infrastructure in place already, then I can come in. Hook up a cooling tower and we’re done,” he said.

Garceau of 3M agrees with Moore’s assessment, although he notes that the bitcoin miners he has spoken to are increasingly interested in optimising their installations even without physical or monetary constraints.

He said:

“If you have real-estate constraints, if you have high cost of energy, then the economics get better. But bitcoin miners are looking for the best economics they can get all round.”

Costs and benefits

Immersion cooling can slash running costs. Allied Control – a Hong Kong-based company that set up ASICMiner’s much talked about mining installation in that city, – claims that its Novec-based cooling system saved the miner more than 90% of its electricity consumption. Green Revolution Cooling, meanwhile, claims to cut running costs in half with its oil-based cooling system.

Even with the potential savings, immersion cooling remains expensive. For a conventional data centre, 3M estimates that the cost of using its Novec product can rise to 25% of the cost of the hardware that it’s cooling. The firm claims that bitcoin miners, however, operate such high density servers that fluid costs of 2%–5% of hardware are possible.

Research firm IHS finds that special cooling fluids are about five times more expensive than a traditional chilled-water system.

Is the future containerised?

Allied Control’s DataTank container

As immersion cooling gains traction, mining farms may take on even more outlandish forms. Both GRC and Allied Control are offering ‘containerised’ solutions for miners who might want to take advantage of, say, a plum location near a hydroelectric dam.

Green Revolution Cooling can build its CarnoJet system inside a 40-foot shipping container, which it promises is ‘plug and play’. Allied Control, meanwhile, touts its DataTank modules, which offers remote, high-definition security cameras as an option.

GRC’s Moore said:

“Not everyone can mine in the Arctic, so if you don’t have that luxury, a solution like ours makes sense. You can be efficient anywhere in the world, whether you’re in the Amazon or Canada.”

Air cooling remains dominant

Bitcoin mining enterprises may be buying cooling fluids by the litre now, but they remain an oddity to the vast majority of data centres. Almost all existing data centres rely on air-cooling systems, as they remain cheaper to run, less complex to manage and more widely available.

According to research firm IHS, liquid cooling accounts for less than 1% of the data centre cooling market. However, the firm also found that demand for liquid-cooling systems is growing, driven by rising energy costs and the emergence of high-performance data centres. For the likes of 3M and GRC, that’s a trend to keep an eye on.

As GRC’s Moore says:

“Bitcoin miners are almost like kids with a magnifying glass over an ant mound: they’re burning up hardware and pushing it to the limit as fast as they can. When you’re cooling your equipment on shelves with box-fans, there’s only so much you can do.”