You may have seen the recent article on Ars Technica by Dan Goodin about KoreLogic. We (Rick Redman and Dale Corpron, KoreLogic consultants) dipped a computer in oil, and left it there, running, 24x7.

Although this idea isn't really all that new (Cray did it in 1985!), our use of it is relatively rare. We dipped a GPU powered password cracking system in the oil. Thanks to Midas Green Tech's help, it was really easy to do. Our hardware wasn't new or even custom, but it's running, right now, in mineral oil.

So, why did we do it?
Well, we built an additional hardened password cracking system out of some extra AMD Radeon 6990s that were sitting around the office. Next, we placed the system in an air-cooled COLO space, and, guess what? It overheated. So we added more fans, and it overheated again. We removed the side of the case, re-attached the heat sinks with better thermal paste, and it still overheated. Granted, we could have moved the 4U twin 6990 system to another COLO which chills their server rooms to 60 degrees, but the cost would have been almost twice as much. Granted, that would have come with higher bandwidth too, but a GPU password cracking system doesn't really need that much bandwidth (unless it's downloading/updating wordlists). So we had an overheating GPU system, with possibly malfunctioning Radeon cards, and the box was air-cooled at a COLO that already does oil-based cooling. It just seemed logical.

Midas assured us it was safe (they've been doing it for a while now). They had multiple other systems already submerged and knew how to do it. The vendor who supplied the hardware (grcooling.com) assured us the GPUs were safe to submerge. So, we decided to go for it.

What it takes:
Remove all fans from your system (except the power supply fan which is "required" by the power supplies). Remove all heat sinks from the CPUs and GPUs. Clean off all the thermal paste that was under the heat sinks, and replace with thermal tape (Just Google for " Thermal Adhesive Tape") and then re-attach just the heat sink (no fans). The idea of running Radeon 6990 cards without fans sounds crazy. They would overheat in 2 seconds in the air.

The next step is tricky... Take your computer, drop it into a tank of cooled mineral oil, and turn it on.

Here is video of us doing just that.

We have to say, it was nerve-racking. It goes against everything you believe in. Computers don't run in liquid. But they do! They run cheaper at the same speed (if not faster) than air cooling. The liquid oil at Midas is pumped out, put through a cooler/radiator using "cold" water, and returns into our rack at approximately 80 degrees (F). Sure, we heat that oil up, but it's very quickly sucked out of the rack, and cooled. The temperature of the oil could be lowered to much colder temps, but why? That would require more electricity.

You can see the oil rising out of the GPU cards in the following video. Note, there are no fans on those cards, the oil you see moving is just the hot oil rising to the top of the rack. Notice the bubbles on the surface of the oil, see how they are moving? The blue light is the power supply and (still intact) internal fan. (It's a 1200 watt power supply, if you are curious). At the end of the video you can see we pan over to the temporary X11 interface and see that we are running oclHashcat-plus. (Don't trust the temps listed in the video, that was a bug that was fixed later.)

In conclusion, this method is cheaper for us as a COLO customer, uses less electricity, and prevents our systems from overheating. Why NOT do it? Whats our attraction in inefficient air cooling? We will update the blog on details on how the system handles the oil long-term, but it's been weeks now, and everything is working great. Currently, the 4 GPU cores and running at "full blast" cracking a few hundred thousand NTLMs and are running at 51-68 degrees Celsius. The oil is approx 21.6 degrees Celsius. oclHashcat-plus will "abort" a GPU cracking job at 90 degrees Celsius.

Here are more pictures of oil cooled systems.
(KoreLogic has no relationship with Green Revolution Cooling, we didn't coordinate with them on this write-up, etc.)

Our answers to some questions:

• Yes, you can do this at home with an acrylic fish tank. Notice "our" tank is made of steel. I've heard that the eventually heat will melt the sealant used on non-commercial tanks, so high-heat uses are not encouraged. Details here. We are not affiliated with, nor sell/resell their systems. It's just a link.


• We are aware this isn't "ground breaking". But why aren't you doing it? Why isn't everyone doing it? Why, as an industry, are we cooling our systems using inefficient methods such as air conditioning? Why is this the only COLO (we could find) that provides this service?


• "Does it really save electricity?" - Well, We aren't running the COLO, so we aren't the best people to ask. But, it costs less (as a customer) to COLO in oil vs. air cooled. So, we assume someone has done the math.


• "What about water cooling and/or water "blocks" for the GPUs?" - This is a decent solution for some people, and we have worked with that for some systems. It still requires lots of extra hardware, and tons of noisy fans. Plus, this is "ready to go" method for us, at a COLO. Just drop it in the oil, and start cracking!


• "Why are you doing this?" - We are security professionals and a big part of what we do is performing security assessments and penetration tests. We often encounter tens of thousands of passwords that we need to crack, in a short period of time, and we'd like to think we've become pretty good at it. We've always believed that it is important to give back to the community, so we've published HOWTOs, given talks, published password cracking rules, forensics tools, and various widgets and utilities. We had some spare hardware lying around and we wanted to see how it would perform in this environment. We thought some people might be interested in this too.


• "You don't really crack 90%." - Sure we do. That was not just some made-up value. Password cracking technology has come a long way. Consider the password cracking contest we run at DEFCON (Crack Me If You Can). We release over 120,000 password hashes and the teams that participate have only 48 hours to crack as many as they can. The hashes are in many different formats, some extremely difficult and computationally expensive to crack. The contest tries to simulate real world environments--but the crackers are so good now, that we are forced to include 12-16 and longer character passwords. The teams that participate crack most of them in the 48 hour window.
  
  So yeah, we often crack over 90% of the passwords we encounter on pentests. This is a big part of what we do. Rick Redman has spoken at conferences (BSides, DEFCON, AHA, DerbyCon, etc.) on advanced password cracking techniques. One of our specialties is developing patterns/rules/wordlists specifically designed to crack corporation/enterprise password sets. Granted, many of the lists are NTLMs (from Active Directories) and those are easier to crack. Thanks Microsoft ;)


• "How is the oil cooled?" - Look in the pictures on grcooling.com's website. There is a large cooling tank in the back of the room. We do not have the specifics on how it works, but it works. This is a professional/industrial solution. The only need for air conditioning in the room, is for the humans.


• "Noise?" - In the YouTube videos, we are whispering. There are approx 40 or so 1U computers running (in oil) in that same room. The "beep" you hear is a laptop speaker about 5 feet away. So it's pretty quiet.


• "Cost?" - It's based on power/bandwidth usage. This system pulls approximately 6 AMPs of power. We will not disclose the actual cost to KoreLogic out of respect for Midas.


• "Can I do it?" - YES! Midas Green Tech is a normal COLO and would gladly oil-dip your machines for you. We have no relationship with them, they are just a group of friendly geeks doing cool stuff. We really like that we are supporting a local, small business by doing this as well.


• "Contamination of oil by dust?" - We do not know the details of this. Maybe there is an oil filter? We're willing to bet that even dust-filled oil is still better at cooling GPUs, than air is. We're not willing to test that theory though.


• "Why not overclock?" - Radeon 6990s are beasts as-is. And we obviously could overclock them, but to extend their life-span we currently aren't planning to overclock.


• "What about your devices when you pull them out of the oil?" - According to Midas, "spraying electronics cleaner" will safely remove the oil and they are "safe" to run in the air afterwards. This has been tested by Midas (we have not validated this test ourselves).


• "Warranty?" - According to Midas, "Several large server component manufacturers like SuperMicro warranty their products in the oil." In this case we were experimenting with spare hardware so it was not a big concern.