Richard Huff's Seattle Social Blog

Since I haven't had a chance to participate in one of Washington Trail Association's weeklong "volunteer vacations" this summer, I opted for one of their Wildland Discovery Hikes. These are hikes organized and led by WTA staff. The weekend of August 26th was the North Fork Skykomish to Dishpan Gap hike: ~15 miles of trail hiking, and ~3 miles of hiking to the trailhead. The blueberries were ripe, the sun was out, and you couldn't have asked for a better group to hike with. Enjoy the photo collect!

Update: I made a Topo map of the hike with elevation profile. It's an 11" x 17" format PDF created with National Geographic's Topo set for Washington state. You can download the PDF by [clicking here] (7.5 MB).

We immediately broke my cardinal rule of not talking about podcasting or blogging during the podcast. (I'll have to be more assertive during future shows.) Anyway, Keith Vaitkus pleaded the main topic of the show: bad bosses. My question was, "Why do we assume that bosses should be good?" That line of reasoning didn't get much traction with the other participants. In the end, this talk about bad bosses lessens my motivation to seek traditional employment. Yet, I realize that my skills and experience will be more valuable when combined with a well-known brand name (e.g., Google, Yahoo, Amazon, Digg, etc.).

My "online dating" update was lackluster. No news to report due to focus on other projects. I better get cracking before I have to join one of those senior citizen sites.

Guest List: Brian Gaither, David Goldstein, Schelley Olhava, Gavin Shearer, and Keith Vaitkus. Listen to the show at: www.confabshow.com.

Over the past week I've finished quite a few projects at the data center. These were the sorts of projects that took months to coordinate (as evidenced by my March 5th, 2006 post about being "halfway there"). I wish I could claim that all of these projects were proactive; that they were borne out of my impeccable engineering competence. Sadly, I have to admit that some of them were provoked by necessity.

Nathan Rolander sheds some light on the fundamental issue in his December 2005 masters thesis, "An Approach For The Robust Design of Air Cooled Data Center Server Cabinets", when he states:

"A lifecycle mismatch is present in data center operation. This is because data centers receive new high powered [sic] servers every 2 to 3 years, whereas the center infrastructure is only upgraded on the order of every 25 years. This means that the center must be reconfigured to handle the increased heat load quite frequently, and after a few iterations of the process the center is required to dissipate far greater loads than initially intended."

Since heat production in a data center is intimately tied to power consumption (every Watt of power used produces approximately 3.4 BTUs), the lifecycle mismatch should be understood to encompass both cooling and power. This is definitely true in our case, and is a problem exacerbated by what amounts to "absentee landlords" at most data center facilities. I don't wish to place all the blame on data center operators, but they really should be taking a position of leadership in this situation. The power outages on July 24th and July 28th at the Garland Building in Los Angeles, California (knocking out DreamHost and MySpace), and the power outage on July 30th at the Fisher Plaza building in Seattle, Washington (knocking out LiveJournal and us [Geckowerx]), are - at a minimum - ominous.

As background, each cabinet (28" W x 36" D x 84" H) at our data center is equipped with one 20 Amp primary and one 20 Amp secondary AC power circuit. The total power available to a cabinet should be understood as 20 Amps - not 40 Amps. (The secondary circuit is intended for redundancy purposes.) The cabinets are cooled using the vertical flow design, where chilled air enters the cabinet through an opening in the base and exits through a fan (rated at 500 CFM) in the top of the cabinet. The doors and sides of the cabinets are not perforated, whereas they would be perforated in a horizontal flow design.

It's important to note that - unlike power - cooling is an issue of degrees. (Yes, I'm sorry. That's a pun.) Data center equipment can operate within a fairly permissive temperature range (e.g., 41F to 104F, in the case of our servers); however, power is usually present or not present - there's very little leniency for "partial power". For this reason (and because heat production and power consumption are so closely correlated), adequate cooling tends to receive attention only after power shortages occur. The attention given our cooling and power adequacy happened, coincidentally, in parallel.

Following an upgrade to several servers, I become concerned with what appeared (anecdotally) to be a pooling of very hot air in the top of a cabinet. Not wanting to rely on anecdote, I ordered and installed AVTECH's TemPageR 4E with one sensor at the base of the cabinet and one sensor at the top (both at the rear). The empirical evidence confirmed the anecdote. The cabinet was experiencing a differential temperature of more than 20F (which is considered the maximum differential you would want in an enclosure of this sort).

After some trial and error, I settled on Delphi's Enclosure Blower (rated at 250 CFM) and sealed any empty rack spaces with blanks. Although this did not exactly resolve the pooling of hot air in the top of the cabinet, it did ensure that each device - despite its location in the cabinet - received a supply of chilled air. (Note: Since most of the equipment has internal temperature sensors that can be queried, I know whether or not each device is operating within its specified temperature range.) Adding two more temperature sensors, this time to the front of the cabinet, confirmed that the Delphi Enclosure Blower was effective. In fact, the temperature at the top, front of the cabinet is about 5F lower than its bottom, front. This is clearly the result of keeping the bottom temperature sensor outside of the blower's direct airflow, whereas the top temperature sensor cannot avoid the chilled air delivered by the blower.

The hotspot in the top of the cabinet still exists, but has been reduced by several degrees. Its potential for damage has also been reduced somewhat. Unfortunately, the 500 CFM exhaust fan simply isn't powerful enough to eliminate the hotspot in the top of the cabinet. My calculations indicate that a fan closer to 900 CFM would be necessary. Also, the delivery of chilled air should be increased to match this higher flow rate. A possible alternative is to implement a hybrid of the vertical and horizontal flow designs by adding a perforated back door to the cabinet, while leaving the front door unperforated. Due to the depth (which may constrain airflow) of several devices in the cabinet, a hybrid approach might be the better option.

"What about power," you ask? It was during some of the server upgrades responsible for the additional heat that I tripped the primary 20 Amp circuit during a boot cycle. It's not a pretty sight to watch a cabinet full of equipment lose power in an instant. Running everything with journaled filesystems and really good backups meant that I didn't need to pop the cyanide pill that all systems engineers and some administrators (the smart ones) carry with them in case of a really catastrophic failure of their own doing. Nonetheless, it was a serious slap across the face. My power requirement calculations were obviously off, even accounting for the inrush demand of booting.

In all honesty, I hadn't revised my power requirements to include the upgraded equipment. (There's a reason why 30% of system downtime is attributed to human error, and this was just such an illustration. Note to self: the additional heat should have been a clue that revising the power requirements was necessary.) Past estimates had indicated a fair capacity margin for a full cabinet (e.g., nominal use of approximately 12.7 Amps out of the 20 Amp circuit). The revised estimates freaked me out. The nominal use estimate was at 24.1 Amps. I had to power down equipment just to bring things back into a safe operational range. Not powering down equipment would have risked throwing a circuit breaker while physically away from the data center.

Two tasks were necessitated by the revised power estimates. First, we needed to start collecting empirical power usage data (similar to what we had already started doing for the temperature). Power estimates are just that: estimates. Real-time data is the experiment that can validate or invalidate your estimates (a.k.a., hypothesis).

Second, we needed larger circuits or additional cabinets.

Following an unhelpful conversation with the data center owner (InterNAP), I opted to order a pair of the APC AP7830 smart PDUs (rated for 20 Amps). At least I could begin gathering data even if they weren't able to provide circuits larger than 20 Amps. A subsequent conversation, involving a different representative, yielded a better result. InterNAP could, in fact, upgrade our 20 Amp circuits to 30 Amp circuits. Of course, this would come at a 41% increase in our per-cabinet, monthly fees (not including our bandwidth charges - which are billed separately). We had no choice. The power upgrade was essential. I immediately ordered a pair of the APC AP7832 smart PDUs (rated for 30 Amps) for the cabinet in question. I'll reserve the AP7830s for another cabinet that has lower power requirements.

The planned power upgrade and installation of the smart PDUs gave me an opportunity to test another power enhancement for the cabinet. Even though much of the data center equipment contains redundant power supplies, certain items (such as high-density servers) are notorious for their lack of power redundancy. A potential solution, which I'd had no prior experience with, is the "dual input PDU". The dual input PDU accepts power from two sources, and switches between them in less than one cycle if power is lost or restored on the primary source. Although dual input PDUs are no replacement for redundant power supplies, they do eliminate a single point of failure that exists when they aren't present. They also allow for complete dependence on a primary circuit, while failing all equipment over to the secondary circuit only when the primary fails. I was a little skeptical at first, but the engineers at Pulizzi were incredibly knowledgeable and informative. I purchased three of the EMF/RFI filtered TPC2234s for the cabinet getting the 30 Amp circuit upgrade. They've been tested under heavy load since installation, and operate exactly as advertised.

These projects have resulted in a cabinet with four points of real-time temperature monitoring and two 30 Amp circuits served through smart PDUs in a truly redundant configuration (thanks to the addition of dual input PDUs for all single power supply devices). Unfortunately, the knowledge gained through real-time temperature and power monitoring indicates that approximately half of a traditional 40U cabinet must remain empty to accommodate the cooling and power demands of the latest data center equipment. I will be curious to see how the necessary accommodations influence costing models that were based on larger unit densities of the newer equipment.

Confab continues its summer trend of light-hitting topics with this week's discussion about high-maintenance girls, appropriate dress for theatre, who pays for dinner on a date, and dating up or down (socio-economically speaking). All of this talk appears to be Gavin's way of prying the latest news out of me about my "online dating" exercise. The consensus is that I will need much editorial help with my "profile".

Guests: Rachele Cawaring, Elaine Chu, Nick Onken, Gavin Shearer, and Keith Vaitkus. Partake in the madness at: www.confabshow.com.