The most essential part for cooling an overclocked computer system is a good radiator. Because the machine needs to run in a data center we had to look hard for a suitable solution. In the end we bought a ERM-3K3UC cooling system from Koolance.

This system is rack-mountable and features a total of nine fans with a diameter of 12cm. At an ambient temperature of 25C, approximately 2000-2500W of heat can be dissipated. Besides the integrated water pump, the cooling system also features flow measurement and temperature sensors that turn off the Tor machine if something goes wrong.

The price for the base ERM-3K3UC system was around 1300 Euro. Koolance also offers a cheaper model (ERM-3K3UA) with an aluminum core, but we decided to spend the extra money for the system with a copper core. Aluminum is more sensitive to biological contaminants in the water and corrosion.

The next picture shows the control port unit that needs to be installed along with the DB9 cable that connects the ERM-3K3UC.

Running an overclocked CPU makes it necessary to monitor the cooling system. The flow meter (INS-FM18D) monitors the amount of water flowing through the cooling loop. Should the cooling system loose water or if someone opens the cooling loop, this unit initiates an immediate system shutdown. Two additional temperature sensors that are connected to the control unit can be setup to protect the system from overheating. While the temperature sensors are included with the cooling system, the flow meter must be bought separately for about 40 Euro.

The ERM-3K3UC was installed at the bottom of the rack to sit right in the cold air stream from the data center cooling system. The installation itself was pretty painless. The mounting brackets for the racks were included in the package and worked better at keeping the cooling system in place horizontally than we expected.

We had to add two modifications to adapt the system to our needs. In its default configuration the fans suck air in to cool the radiator. Unfortunately in the horizontal position the fans would be on the top, pushing the hot air down into the cooler AC air from the data center. We decided to turn all of the fans to suck the air out of the unit instead, dissipating the heat more quickly.

The second minor modification involves the water valve of the unit. In the normal position of the unit the water valve is at the top, so when installed into the rack it is on the left side instead. To really make sure that no water drips out, we secured the valve screw with a bit of hot glue.

Better safe than sorry ... after all, it is running in a data center and who knows how many setups similar to this exist.

To be able to disconnect the radiator from the machine chassis, tube coupling connectors are needed. While there are many connectors on the market, it was important to us to be able to open the cooling loop without any water spilling out. Below you see the QD4 series connectors, also manufactured by Koolance.

Of course we tested how well the no spill feature works when the cooling loop is active. As it turned out it works really well. Only one or two drops of water were present on the connectors and the instant system shutdown worked as advertised.

Unfortunately for us the 3U server case did not offer enough room to install quick connect couplers on the CPU cold plate. Instead we took the normal connectors there which worked fine too.

We cut off a few 20-25cm long pieces of the tube that were then used to install the connectors. Simply push them over the connector and gently turn until it goes no further, then carefully pull off the tube while maintaining the torque direction.

The installation of the cooling loop tubes was a two person job. One to hold and turn the tube and the second person needed to pull the clamps over the tube to seal it properly. Once installed you have to be really careful to prevent torque on the tubes that could easily loosen the connectors leading to water spill.

The only drawbacks we experienced with those connectors were the installation process and handling of the tubes after the installation. If you have enough room we recommend installing the no-spill quick connectors on the CPU cold plate.

When it comes to water tubes there are a lot of options you can choose from. Since we don't care if the tubes glow under UV light we went for tubes from the PrimoFlex Advanced LRT series.

They work well in small spaces because of they can be bent in a tight radius without the tube collapsing. The tubes are also polished on the inside to minimize the surface area biological contaminants can stick to. Another plus point is that they are manufactured without hazardous chemicals inside.

One important thing we had to do was to wash out the tubes with hot clear water before connecting them to the cooling loop. There was quite some dust inside because the ends of the tubes were not sealed wherever the supplier had stored them.

When cutting the tubes make sure you have a really really sharp knife with a long blade. You want to cut the whole tube in one go. It also helps to squeeze the tube just a tiny weeny bit to even out the down pressure from the knife to make sure you get a clean cut. If done right you get a straight cut that is not curved in or slanted at the end.

Metal clamps are used to fix the tubes on metal connectors and keep the water from escaping. Those clamps are a really tight fit, but only four of them were needed. A pair of channel locks was used to loosen the clamps and pull them over the tubes. Since the space on the CPU cooler is very narrow, you have to be extra careful.

Of course some kind of liquid coolant needs to go into the cooling loop. The choice of the radiator kind of locked us into the available options offered by Koolance. You can put in other cooling liquids, but we went with the recommendation from Koolance and got five bottles of colorless cooling liquid.

This is no ordinary water from the tap and features a low conductivity as well as corrosion protection and biological inhibitors. All in all, we had to use about 4 bottles of the cooling liquid. Since the ERM-3K3UC was installed horizontally, we had to add a bit more (one bottle) than the required level of fluid for vertical operations.

The price per bottle was about 20 Euro.

The next hardware part on our list is the actual CPU cooler or cold plate. After comparing various models we imported a Cuplex kryos (beware German only website) solid copper cooler from aquacomputer for the CPU. The price for this piece of hardware was 80 Euro.

A more efficient version (+10% heat transfer), the HF .925 silver edition, is available that incorporates a silver bottom plate, but this model costs 240 Euro. Our budget was not big enough to cover the extra costs for this particular optimization.

The CPU itself is a regular off the shelf Intel i7-4790k Devils Canyon with four 4 GHz Haswell cores. This CPU is interesting because it has unlocked multipliers which means you can overclock it as much as you want as long as you can get it to run stable. Of interest when using this CPU model is that Intel integrated the voltage regulator modules directly into the CPU.

We were able to clock the CPU at 5GHz and compile kernels, run OpenSSL benchmarks or other synthetic workloads. Once we hit higher loads with Tor we had to scale down the CPU frequency to 4.9GHz. It would run at higher speeds but would throw machine check exceptions every couple of days which would reboot the machine. At 4.9GHz it runs more or less stable for a bit over one month now.

As it turns out, the new Haswell CPUs have a much lower yield when it comes to overclocking stability. While we would love to clock the CPU at 5GHz when running Tor, we prefer stability over speed. If you want a 5GHz Tor exit node feel free to send us i7-4790k CPUs that we can use or send us some BTC to 1Q3mjKbZwZFEigC8edUZ8ywX4QD7kxFzNC. :)

We were using an i7-4770k CPU before (shown in the picture), which we could only clock up to 4.5GHz. In July Intel released the Devils Canyon update, that we were eagerly anticipating due to the overclocking woes with the i7-4770k. Go for the i7-4790k if you consider replicating the setup.

The Cuplex kryos includes some heat-conductive paste of the type Prolimatech PK-1. Since an upgrade to the Prolimatech PK-3 was about 20 Euro, we bought that instead. The heat-conductive paste compensates for micro scratches in the attached surfaces and transfers heat between those.

The Cuplex CPU cooler mounted on the mainboard. To make the installation a bit easier we marked the screws to be able to count the number of turns on each of them.

Since server mainboard manufacturers are not that much into overclocking, we had to go with a mainboard that offers dedicated overclocking support. We decided to use the MAXIMUS VI EXTREME Rev. 1 Republic of Gamers edition mainboard from ASUS.

Of course this piece of hardware comes with a ton of features that we don't really need to build the Tor server like the SATA ports, the mini-PCIe slot, audio S/PDIF port, or 4-way SLI support. Some features we explicitly were looking for are the good default cooling options for the chipset (no extra cooling needed), DDR3 support up to 3100MHz, Intel Z87 chipset, backup BIOS support, saved overclocking profiles in the BIOS, and the external display of the CPUs temperature and POST status via the OC panel.

One feature all of the desktop boards are missing is support for a proper IPMI module to remotely manage the machine. We worked around that issue by installing a Raspberry PI for remote power control and a Lantronix Spider for KVM-over-IP.

Once all the hardware was assembled we used an overclocking guide from ASUS to do the basic tuning of the system. The work flow was to find the most basic stable settings and store them as an OC profile in the BIOS. Then change whatever settings we thought would help us to gain more performance followed by a battery of tests. Make small steps when changing the default values for power control, bus speeds, RAM timings and test, test, test.

To make sure the machine runs stable there are basically 2 options available. Option 1 is to install Windows and use the available benchmark suites to stress test the system. Option 2 is to define your own set of synthetic benchmarks. Since we are not so much into Windows we decided to go with option 2 and prepare our own set of benchmarks / stress tests to make sure the system runs stable.

In the end we had a bunch of shell scripts that executed various CPU, RAM and IO benchmarks in a loop. Since most of those tests are quite synthetic and only stress a limited number of resources in the system we complemented our own little test setup with tasks like compiling the Linux kernel in a loop or generating Diffie Hellman keys. With those tests we were able to run the system stable at 4.9GHz. After having passed a 2 week burn in test we decided to move the machine to the data center.

One nifty feature of the Maximus VI Extreme Republic of Gamers edition is the overclocking panel. Although we do not use it to overclock the machine when needed it is a pretty useful quick diagnostic tool in the data center. By default it displays the CPU core temperature and speed, the machine state and the output of the POST (saves us from opening the machine). Fortunately for us the 3U server case has two 5 1/4 inch slots available to install the OC panel.

The next item on our hardware list is the machine RAM. From our past experiences with Tor and having bigger than usual defaults for buffers, we decided to put 32GB of RAM into the machine. While the board supports DDR3 memory with speeds up to 3100MHz it is quite expensive to buy such fast memory. The 32GB of ADATA 16GB DDR3-2800MHz XPG V2 RAM came in 4 modules with a whopping price tag of 1300 Euro. We would have taken other modules, but those were the only ones we were able to get for a 32GB configuration at that time.

The installation of the RAM itself was pretty painless. We simply plugged it in and let the board run its RAM detection feature. After that was done we added a little bit of fine tuning according to the available OC guides. Since that is a very specific task, depending on the actual combination of the mainboard and RAM, we wont go into further details here.

The RAM in the picture below is not the ADATA one but some cheaper RAM we used until the ADATA modules were available.

One particular problem was that sometimes other people were not too happy about the Tor exit node and decided to DDOS the poor thing out of the Internet. So we decided to gift our new shiny Tor machine with a 10G Ethernet port upgrade. With a 10G port it is much easier to eat most of the DDOS coming in while maintaining operations. In the past the Tor servers usually had Intel network cards, because we experienced some issues with them we decided to go with Myricom this time. The 10G network card we bought was a 10G-PCIE2-8C2-2S for about 550 Euro.

The installation of the card and drivers was pretty much RTFM. The only thing we noted was that this NIC needs a proper airflow to keep it cool. When the card was installed in the test setup it caused a few system freezes until we had the cooling correctly set up. To make sure we stay ahead of any cooling issues with the NIC we installed one of the temperature sensors from the ERM-3K3UC control board directly at the CPU heat sink of the NIC.

Of course the Tor machine needs a proper housing as well. In terms of price performance the SC835 from Supermicro seemed to be an ideal choice. It offers the required 5 1/4 inch slot as well as redundant power and slide out rails. The price for the case was ca. 550 Euro.

We had to sacrifice 2 PCI slots to make room for external connections like the tubes, Raspberry PI serial and Ethernet cable. The pre-installed fans inside the case ensure a proper airflow to cool the remaining active parts of the system even when the airflow is partially blocked by tubes and cables.

The default cable configuration from Koolance to connect the internal sensors to the flow meter was a bit short. We had to stretch the sensor cable by about 20cm to be able to connect it seamlessly to where the flow meter was actually installed inside the case. The whole setup fits very nicely into the case and required almost no customizations.

To fix the issue of the missing IPMI module to control the power and reset the machine if it decides to stop working, we installed a Raspberry PI (RPI) model B. Using the GPIO interface of the RPI we can now control the reset button of the motherboard. This means that if the machine freezes we can SSH into the RPI and run a small python script to reboot it.

The Lantronix spider is used to get access to the keyboard and graphics. Given that the BIOS likes to load safe values after some error conditions, for example disable the overclocking, it simplifies the recovery process quite a bit.

The RPI was strapped onto a common 19 inch board and seated in the front of the case where the disks are usually installed. To maintain the normal front panel reset button we used a simple Y cable to connect the GPIO control unit to the reset pins. The GPIO pins are not connected directly to the reset pins but instead drive a small relay that is also housed on the 19 inch board.

Unfortunately there are no pictures for this section because the photo team was MIA when the RPI setup was built and installed.