Technology

This year marks the beginning of my two-year appointment by Greg Nickels, Mayor of the City of Seattle, to the Citizens' Telecommunications and Technology Advisory Board (CTTAB). The board advises the Mayor's office and the City Council on technology and communications issues facing Seattle. Although my background includes a variety of technology competencies, my primary contribution will likely be with the committee consulting on Seattle's Broadband Initiative. Watch for the new FTTx (Fiber To The Home and Fiber To The Premises) category on this blog to follow developments over the next couple years.

Melissa -- now the proud owner of an 8GB iPod Nano (Product)^RED -- turned me on to the best workout mixes I've ever heard. DJ Steveboy mixes hour-long tracks of 130 to 180 beat-per-minute ear candy. It's the sort of music that makes you want to throw up your arms and dance while you run. Not even the music from Rocky touches the sort of nerve that PODRUNNER tracks tickle. Take a dose for your next run.

You can get it through iTunes:

In an age where progress is heavily constrained by the limits of the first mile networks (i.e., the twisted-pair copper from phone companies, and coax from cable companies that connect homes and business to core networks), it's extremely important that anyone approaching the issue of addressing that problem understand the weight that the late-1990's long-haul fiber bubble places on Fiber To The Home (FTTH) or Fiber To The Premesis (FTTP) initiatives (FTTx collectively). Fiber is, without question, the technology that will be deployed in place of twisted-pair and coax networks. The journey to this inevitable FTTx future is, however, obstructed by many countervailing interests and historical baggage. Om Malik's book, Broadbandits: Inside the $750 Billion Telecom Heist, documents how the telecommunications industry stumbled in a race to cash in on the build-out of long-haul fiber networks. It's my opinion that the market and public skepticism resulting from the telecom scandals have, and will, retard the oh-so-necessary deployment of fiber infrastructure in the first mile. In fact, it's this "hangover" that appears to be keeping the telecom industry from playing a leadership role in FTTx. Read the book.

Over the past week I've finished quite a few projects at the data center. These were the sorts of projects that took months to coordinate (as evidenced by my March 5th, 2006 post about being "halfway there"). I wish I could claim that all of these projects were proactive; that they were borne out of my impeccable engineering competence. Sadly, I have to admit that some of them were provoked by necessity.

Nathan Rolander sheds some light on the fundamental issue in his December 2005 masters thesis, "An Approach For The Robust Design of Air Cooled Data Center Server Cabinets", when he states:

"A lifecycle mismatch is present in data center operation. This is because data centers receive new high powered [sic] servers every 2 to 3 years, whereas the center infrastructure is only upgraded on the order of every 25 years. This means that the center must be reconfigured to handle the increased heat load quite frequently, and after a few iterations of the process the center is required to dissipate far greater loads than initially intended."

Since heat production in a data center is intimately tied to power consumption (every Watt of power used produces approximately 3.4 BTUs), the lifecycle mismatch should be understood to encompass both cooling and power. This is definitely true in our case, and is a problem exacerbated by what amounts to "absentee landlords" at most data center facilities. I don't wish to place all the blame on data center operators, but they really should be taking a position of leadership in this situation. The power outages on July 24th and July 28th at the Garland Building in Los Angeles, California (knocking out DreamHost and MySpace), and the power outage on July 30th at the Fisher Plaza building in Seattle, Washington (knocking out LiveJournal and us [Geckowerx]), are - at a minimum - ominous.

As background, each cabinet (28" W x 36" D x 84" H) at our data center is equipped with one 20 Amp primary and one 20 Amp secondary AC power circuit. The total power available to a cabinet should be understood as 20 Amps - not 40 Amps. (The secondary circuit is intended for redundancy purposes.) The cabinets are cooled using the vertical flow design, where chilled air enters the cabinet through an opening in the base and exits through a fan (rated at 500 CFM) in the top of the cabinet. The doors and sides of the cabinets are not perforated, whereas they would be perforated in a horizontal flow design.

It's important to note that - unlike power - cooling is an issue of degrees. (Yes, I'm sorry. That's a pun.) Data center equipment can operate within a fairly permissive temperature range (e.g., 41F to 104F, in the case of our servers); however, power is usually present or not present - there's very little leniency for "partial power". For this reason (and because heat production and power consumption are so closely correlated), adequate cooling tends to receive attention only after power shortages occur. The attention given our cooling and power adequacy happened, coincidentally, in parallel.

Following an upgrade to several servers, I become concerned with what appeared (anecdotally) to be a pooling of very hot air in the top of a cabinet. Not wanting to rely on anecdote, I ordered and installed AVTECH's TemPageR 4E with one sensor at the base of the cabinet and one sensor at the top (both at the rear). The empirical evidence confirmed the anecdote. The cabinet was experiencing a differential temperature of more than 20F (which is considered the maximum differential you would want in an enclosure of this sort).

After some trial and error, I settled on Delphi's Enclosure Blower (rated at 250 CFM) and sealed any empty rack spaces with blanks. Although this did not exactly resolve the pooling of hot air in the top of the cabinet, it did ensure that each device - despite its location in the cabinet - received a supply of chilled air. (Note: Since most of the equipment has internal temperature sensors that can be queried, I know whether or not each device is operating within its specified temperature range.) Adding two more temperature sensors, this time to the front of the cabinet, confirmed that the Delphi Enclosure Blower was effective. In fact, the temperature at the top, front of the cabinet is about 5F lower than its bottom, front. This is clearly the result of keeping the bottom temperature sensor outside of the blower's direct airflow, whereas the top temperature sensor cannot avoid the chilled air delivered by the blower.

The hotspot in the top of the cabinet still exists, but has been reduced by several degrees. Its potential for damage has also been reduced somewhat. Unfortunately, the 500 CFM exhaust fan simply isn't powerful enough to eliminate the hotspot in the top of the cabinet. My calculations indicate that a fan closer to 900 CFM would be necessary. Also, the delivery of chilled air should be increased to match this higher flow rate. A possible alternative is to implement a hybrid of the vertical and horizontal flow designs by adding a perforated back door to the cabinet, while leaving the front door unperforated. Due to the depth (which may constrain airflow) of several devices in the cabinet, a hybrid approach might be the better option.

"What about power," you ask? It was during some of the server upgrades responsible for the additional heat that I tripped the primary 20 Amp circuit during a boot cycle. It's not a pretty sight to watch a cabinet full of equipment lose power in an instant. Running everything with journaled filesystems and really good backups meant that I didn't need to pop the cyanide pill that all systems engineers and some administrators (the smart ones) carry with them in case of a really catastrophic failure of their own doing. Nonetheless, it was a serious slap across the face. My power requirement calculations were obviously off, even accounting for the inrush demand of booting.

In all honesty, I hadn't revised my power requirements to include the upgraded equipment. (There's a reason why 30% of system downtime is attributed to human error, and this was just such an illustration. Note to self: the additional heat should have been a clue that revising the power requirements was necessary.) Past estimates had indicated a fair capacity margin for a full cabinet (e.g., nominal use of approximately 12.7 Amps out of the 20 Amp circuit). The revised estimates freaked me out. The nominal use estimate was at 24.1 Amps. I had to power down equipment just to bring things back into a safe operational range. Not powering down equipment would have risked throwing a circuit breaker while physically away from the data center.

Two tasks were necessitated by the revised power estimates. First, we needed to start collecting empirical power usage data (similar to what we had already started doing for the temperature). Power estimates are just that: estimates. Real-time data is the experiment that can validate or invalidate your estimates (a.k.a., hypothesis).

Second, we needed larger circuits or additional cabinets.

Following an unhelpful conversation with the data center owner (InterNAP), I opted to order a pair of the APC AP7830 smart PDUs (rated for 20 Amps). At least I could begin gathering data even if they weren't able to provide circuits larger than 20 Amps. A subsequent conversation, involving a different representative, yielded a better result. InterNAP could, in fact, upgrade our 20 Amp circuits to 30 Amp circuits. Of course, this would come at a 41% increase in our per-cabinet, monthly fees (not including our bandwidth charges - which are billed separately). We had no choice. The power upgrade was essential. I immediately ordered a pair of the APC AP7832 smart PDUs (rated for 30 Amps) for the cabinet in question. I'll reserve the AP7830s for another cabinet that has lower power requirements.

The planned power upgrade and installation of the smart PDUs gave me an opportunity to test another power enhancement for the cabinet. Even though much of the data center equipment contains redundant power supplies, certain items (such as high-density servers) are notorious for their lack of power redundancy. A potential solution, which I'd had no prior experience with, is the "dual input PDU". The dual input PDU accepts power from two sources, and switches between them in less than one cycle if power is lost or restored on the primary source. Although dual input PDUs are no replacement for redundant power supplies, they do eliminate a single point of failure that exists when they aren't present. They also allow for complete dependence on a primary circuit, while failing all equipment over to the secondary circuit only when the primary fails. I was a little skeptical at first, but the engineers at Pulizzi were incredibly knowledgeable and informative. I purchased three of the EMF/RFI filtered TPC2234s for the cabinet getting the 30 Amp circuit upgrade. They've been tested under heavy load since installation, and operate exactly as advertised.

These projects have resulted in a cabinet with four points of real-time temperature monitoring and two 30 Amp circuits served through smart PDUs in a truly redundant configuration (thanks to the addition of dual input PDUs for all single power supply devices). Unfortunately, the knowledge gained through real-time temperature and power monitoring indicates that approximately half of a traditional 40U cabinet must remain empty to accommodate the cooling and power demands of the latest data center equipment. I will be curious to see how the necessary accommodations influence costing models that were based on larger unit densities of the newer equipment.

On a fairly regular basis I get asked about the equipment used for the various podcasts that Gavin and I produce (e.g., Confab and Podcasting Liberally). I, personally, have no notable background in audio engineering. When it came to shopping for our podcasting equipment, I had to piece together information from many different sources. Our goal was probably more ambitious than most, and included the requirement of supporting four or more participants from the very first recording. Luckily, only a few, minor mistakes were made on the path to achieving what we produce today.

In the most basic terms of podcasting, you need a way to record audio and convert it to a file format compatible with digital audio players. This typically means a microphone, computer (with audio recording software), and the MP3 file format. Distribution of the resulting podcast usually implies the Internet and Real Simply Syndication (RSS), but that's a whole other blog post. Let's stick with the production side for now, and start with microphones.

Following a series of recommendations and some testing, I decided on the Sennheiser HMD25-1 headphone and microphone combination. It includes a dynamic, super-cardioid microphone that, in the words of Sennheiser's marketing literature, "...has been specifically designed for use in noisy environments." They absolutely live up to the claim. Also, the benefits of a microphone and headphone combination should not be understated. For one thing, they tend to keep the microphone an equal distance from the participant at all points during the podcast recording, despite how much the participant moves their head. Although the MSRP for the Sennheiser HMD25-1 is $519.00 (USD), you can usually find them for ~$400.00 (USD). Be forewarned that the cable ends are going to be your additional responsibility - the HMD25-1s ship with bare wire ends to the headphone and microphone cables. Soldering male 1/4" jacks for the headphones, and male XLR connectors for the microphones prepared our HMD25-1s for actual use.

To improve the audio quality even further, our microphones connect to a series of tube pre-amps before feeding into the mixer. Although the mixer has built-in pre-amps for the microphone channels, their "gain" range is fairly limited. The tube pre-amps provide the necessary range, but also what audiophiles refer to as "warmth". Since each incoming microphone requires its own channel, we bought four of the 2-channel Behringer Ultragain Pro MIC2200s tube pre-amps. MSRP for the Behringer MIC2200s is $129.99 (USD), but you can readily find them for ~$99.00 (USD).

The eight, separate channels coming from the tube pre-amps feed individually into our mixer. In most traditional applications of a mixer, the input channels are all combined -- at various levels -- to produce a right and a left "main" channel (the output). Effects processors and EQ functions may also be added to modify the audio as it is being combined. The two resulting "mains" would normally be what get recorded for a podcast. In our case, we purchased a mixer that included a digital, Firewire (IEEE 1394) interface. The Firewire interface connects to our computer and delivers each of the input channels directly to our audio recording software. We not only use the mixer to combine the input channels, we also use audio software on a computer to perform the exact same task after the recording. Why? There are really two reasons. First, we need the ability to return the mixed channels to the participants' headphones in real time during the podcast recording. That's done through the mixer. Second, we want the ability to add effects and control levels on the individual channels in software, during or after the recording. The mixer and computer combination give us this remarkable flexibility.

In the very beginning, I purchased the Alesis Multimix 16 Firewire mixer. What I really wanted was the Mackie Onyx 1620 with Firewire option (~$1,300.00), but it (pictured) was more than twice as expensive as the Alesis (~$600.00). (You have to be somewhat conservative when you don't know how serious you'll be about a new hobby.) After almost eight months of regular podcasting, I felt that it was serious enough to warrant the Mackie Onyx 1620 purchase. The Onyx has several advantages over the Alesis, including visual level meters and low-cut filters for each of the eight microphone channels. It also produces better sounding audio.

For anyone considering a mixer for their podcast production, there are several Firewire and USB models from a handful of manufacturers. The number of channels delivered to your computer is the most significant difference between the Firewire and USB versions. In every model I've reviewed, the USB versions only deliver the left and right mixed channels to your computer - not each of the individual, pre-mixed channels. The lower cost of the USB versions is likely the result of this difference. Vendors need to make this difference more apparent in their marketing literature if they expect to have high rates of customer satisfaction within the podcasting community. There's absolutely nothing wrong with using a USB version, as long as you know about their limitation prior to spending the money.

The final piece of the system (excluding the computer and software) is the headphone distribution amp. The mixer has a right and left channel output, but definitely can't drive a series of headphones. To solve this challenge, I purchased the Behringer PowerPlay Pro-8 HA8000 (MSRP $179.99 / Street $140.00), which is an 8-channel headphone amp. The output from the mixer feeds into the amp, and the headphones connect to the individual distribution channels. Each channel of the distribution amp has the added benefit of it's own volume control. I eventually purchased two of the HA8000s. One is used to support the headphones of the podcast participants; one is used to support a set of headphones for listeners. This second headphone distribution amp is really important for Podcasting Liberally because it's held, weekly, at the Mountlake Alehouse. Anyone wishing to listen in on the live recording needs headphones to hear over the background noise.

All of our podcasts are recorded to disk using Apple's GarageBand software on my 17" Apple Powerbook. (I hope to be using a 17" Apple MacBook Pro in the very near future.) GarageBand's ability to perform compressor/limiter functions on a per-channel basis is the second most influential factor in making our podcasts sound good. We would have to spend another $2,000 to perform the same task in audio hardware. Gavin has been mixing and converting the shows on a Core Duo Mac Mini since the beginning of March, and I can't wait to have that sort of power in a portable form. All told, we've spent ~$6,300.00 (not including my laptop) on the podcasting gear. It has been worth every penny that we've spent to hang out with good friends for smart conversation on a weekly basis.

Go forth and podcast!

The new high-density servers in our data center have posed quite a serious cooling and power consumption anxiety. Within the span of two years, our typical 1U server purchases have more than tripled in BTU output (from 221 to 780)¹ and more than quadrupled in amperage draw (from .54 amps to 1.91 amps)¹. Despite how well the data center might have been designed, these demands are technically beyond its inherent capacity. Special steps must be taken for those cabinets where we have a high concentration of the newer servers.

One recent step was to optimize the cool airflow within the cabinet enclosure. After some trial and error with several different options, we settled on the Delphi Enclosure Blower -- a 2U, 250 CFM device uniquely designed for this particular application. The cooled air in our data center is supplied at the base of the cabinets through a raised floor, and exhausted into the room through the top of each cabinet. The Delphi Enclosure Blower directs a portion of the cooled air up the inside front to ensure that adequate cooling reaches the intake for all the equipment in the cabinet. Delphi's research claims a 15^o F improvement for equipment at the top of a cabinet (where traditional hotspots occur). Although our temperature monitoring equipment is not configured to measure the front of our cabinets, I do not doubt the claim's validity. My anecdotal experience leads me to believe that Delphi's research is pragmatic.

Improved exhaust flow remains a challenge for the cooling project. The 500 CFM fan at the top of the cabinets aren't powerful enough to expel the heated air so as to keep the temperature differential between the base and the top under 20^o F. Using the basic rule-of-thumb that the cabinet exhaust fan should have a rating roughly equivalent to the sum of all the housed equipments' exhaust fans, we're about 400 CFMs shy of what we really need.

¹ The noted BTU and amperage values are nominal, and can represent as little as 50% of the peak BTU or amperage ratings of the equipment. In practical terms, the nominal values are accurate for regular operation.