How Green Data Centers Save Money

Last summer when Wendy Cebula was shopping for a new vehicle, energy efficiency and lower emissions topped her list of requirements, along with four-wheel drive (her family lives on a hill). Cebula, then CIO at VistaPrint, a $152 million online supplier of custom print services, eventually chose a hybrid model instead of the traditional SUV. Even though she didn’t think the incremental savings on gas would make up for the higher sticker price, she says “it was the right thing to do as a human being.”

But as a corporate executive, Cebula, now VistaPrint’s COO, can’t lead with her heart. The right thing to do is whatever enables the business to grow. Those decisions come down to dollars and sense, not what’s best for the planet. But every now and then, the two converge.

In late 2005, Cebula noticed her data center’s costs were growing. The company, whose operations are almost completely automated, was adding 100,000 customers a month—and growing at nearly 60 percent a year. There was no sign the demand for data necessary to serve those customers would stabilize. When Cebula and Aaron Branham, VistaPrint’s vice president of technology and operations, dug into data center operations in detail, they discovered that energy costs were rising significantly.

If there were some way to lower power costs or increase energy efficiency, they could cut expenses. To Cebula, an avid recycler who tries to impart environmental awareness to her kids, doing so would be a double win. She could do “what’s good for the environment and what’s good for the bottom line,” she explains.

Until recently, the environmental impact of the data center was largely ignored. But today, energy experts estimate that data centers gobble up somewhere between 1.5 percent and 3 percent of all electricity generated in the United States. At the top of the range, that’s about the amount of electricity it takes to power the entire state of Michigan for a year. Market research company IDC (a sister company to CIO’s publisher) estimates that companies spent $26.1 billion to power and cool servers worldwide in 2005. That’s more than was spent to power all the commercial buildings in 17 states—from Delaware to Florida and west to Texas, according to the Department of Energy’s most recent energy consumption survey.

And that’s not all. According to the Uptime Institute, a consortium of companies devoted to maximizing efficiency and uptime in the data center, more than 60 percent of the power used to cool equipment in the data center is completely wasted. In fact, notes a recent study by the group, energy costs have replaced real estate as the primary data center expense. “Data centers that used to cost $10 million now cost $100 million,” says Jonathan Koomey, staff scientist at Lawrence Berkeley National Laboratory. “That kind of expenditure gets C-level attention.”

It’s garnered government interest too. A federal law enacted in December compels the U.S. Environmental Protection Agency to examine power consumption in data centers, evaluate what technology manufacturers are doing to increase energy efficiency and determine what incentives could convince companies to adopt more energy-efficient technology. The European Union is studying the level of carbon emissions from computer equipment. Down the road, local and federal governments in the United States and abroad may end up penalizing organizations that operate inefficient data centers, according to Rakesh Kumar, Gartner research vice president.

The combination of financial, environmental and legislative pressure will force IT organizations to develop greener data centers, says Kumar. By 2011, Gartner predicts, a quarter of new data centers will be designed for maximum energy efficiency and minimum negative environmental impact. But what that means may vary by organization. “There’s no generally accepted, standardized way to build a green data center,” says Kumar.

At VistaPrint, becoming green has proven easier than Cebula thought. The company bought more energy-efficient servers and improved utilization in its primary data center in Bermuda, steps that have reduced energy usage by 75 percent. As a result, the company expects to save nearly half a million dollars over three years and estimates it will reduce its output of carbon dioxide emissions by several hundred tons in this year alone. That’s equivalent to taking more than 100 cars off the road for a year.

VistaPrint also decided to locate a new data center in Canada, where hydroelectric power—a renewable energy source—keeps power costs stable and has potential to lower VistaPrint’s electricity bills by another 70 percent. “We were able to reduce our footprint at a time when it was very important to us financially,” says Cebula. “And it’s much more green.”

High-Density Power Surge

Back in 2000, when VistaPrint founder and CEO Robert Keane moved company headquarters from Paris to Waltham, Mass., the fledgling operation was bringing in just over $6 million in annual sales (the company makes business cards and other printed products to order). During the next five years, the company (which is officially based in Bermuda) saw its revenue explode. VistaPrint outgrew its 7,000 square feet of suburban Boston office space and moved a few miles north to a Lexington, Mass., location eight times larger.

During those years, the company focused on automating and optimizing the product design and manufacturing process at its Venlo, Netherlands, plant. The only concern when purchasing equipment—whether for the company’s main Bermuda data center (which runs VistaPrint’s website and transaction systems and is hosted by Cable & Wireless) or for the one in Lexington (which supports internal systems and IT production)—was that the equipment work. As a result, VistaPrint procured a hodgepodge of servers: back-end systems, front-end systems and databases each ran on different gear, says Cebula. By 2004, VistaPrint got caught up in the blade craze, purchasing machines from IBM.

In the fall of 2005, shortly before Cebula became CIO, VistaPrint’s then-COO, Alex Schowtka, hired Branham as director of IT operations. Branham thought blades were a mistake. The problem was the total power pull. “Blade servers look great on paper,” Branham says, “but you start piling them into a rack and suddenly you’re out of power, you’re out of AC.”

VistaPrint wasn’t the only company that was getting burned by its decision to buy high-density equipment. With quality data center space priced at a premium, companies sought out more compact gear, and vendors happily obliged. “The focus was on getting as much computer power in as small a package as possible,” explains Gartner’s Kumar. “Energy was not part of the design mentality.”

But blade servers need more power than less-dense hardware. A full rack of high-density servers requires 20 to 30 kilowatts of electricity, while traditional data centers are designed to provide 2 to 3 kilowatts per rack. Meanwhile, according to the Uptime Institute, using high-density equipment triples or quadruples facility cooling costs. Energy prices have risen as well.

CIOs who assumed that data center costs would decline as servers got cheaper and more powerful received a rude awakening. In what Kenneth Brill, founder and executive director of the Uptime Institute, calls “the meltdown of Moore’s Law,” the energy required to power and cool $1,000 worth of server equipment has skyrocketed from 8 watts in 2000 to 109 watts today—eroding some of the benefits from more powerful chips. In the best-case scenario, says Brill, within five years it could take 157 watts to run the same $1,000 worth of hardware; in the worst case, 1,650 watts. In fact, the blade server’s selling point—its size—has become irrelevant for some customers. Data center operators find themselves using only two servers in a rack designed for 10; packing any more than two high-density servers in a rack makes them too hard to cool.

That was VistaPrint’s quandary. It was the first Cable & Wireless customer in the Bermuda data center to install blades. And the outsourcer was none too thrilled. Cable & Wireless wanted to limit VistaPrint to one blade server per rack. VistaPrint got the vendor to agree to placing two servers per rack, but talk started of an energy surcharge. For years, most outsourcers charged for data center space based on square footage—a metric that did not take into account electricity costs. As a result, says Lawrence Berkeley’s Koomey, the outsourcers created a perverse incentive. “If you charge by the square foot, of course the customers want fully packed racks,” Koomey says.

A Virtual Solution

For Branham, joining VistaPrint in September 2005 was like “a trip back in time.” He had spent the eight previous years at Monster.com and had seen data center operations expand from one server to 1,000. He had spent the past year working on server virtualization. But Branham saw bigger challenges at VistaPrint. “From afar it looks like a print company. But once you get inside and see what’s really going on, it’s really a technology company,” Branham says, noting that everything from sales to manufacturing to shipping is run largely on custom-built software. “At Monster,” says Branham, “we weren’t anywhere near this data-focused.”

Like Monster in earlier days, VistaPrint wasn’t making the most efficient use of its data center. “They picked technologies that were right at the time, not right for where they were going.” Branham knew that virtualization would reduce power consumption. But Cebula knew that she wouldn’t convince anyone to spend more money on servers without a solid business case. So Branham ran the numbers, focusing on power expenditures.

Calculating the total energy consumption of a server isn’t straightforward. How much power a server consumes depends on how you use it. “It’s not something that our vendors talk about a lot,” says Cebula. The vendors were, nevertheless, able to provide data when asked. Using that information, Branham made the case for a greener data center crystal clear.

Branham calculated that if Vista­Print continued to use blades, this solution (conceived of as four racks housing eight IBM blade servers) would eat up approximately 32,000 watts (4,000 per server) and require 9.1 tons of air conditioning to cool. He estimated that the alternative, using eight HP Proliant DL 585 rack-mounted servers and 110 VMware instances, would require 5,500 watts (50 watts per virtual machine) and just 1.6 tons of AC. In addition, virtualization would enable VistaPrint to make better use of its CPU capacity. As is the case in many companies, VistaPrint was wasting additional energy by using only 20 percent of its server capacity. In a 24-hour period, the typical x86 server is used to only 5 percent or 10 percent of its capacity, says Gartner’s Kumar, while energy consumption even in idle mode is 60 percent to 80 percent of the energy consumed in use.

But the development team was concerned about stability and performance in a virtual environment. A year earlier, the IT group had attempted virtualization on the blades. Running VMware on the high-density hardware had been a bust. Memory limitations on the blades limited performance and the number of instances that could be run on each server. So when Branham used the “V”-word again, the development group got nervous. Branham won them over with a pilot project that proved the performance case. Swapping in the new servers in the Bermuda data center would mean eating the investment VistaPrint had made in the blades. But Cebula and the executive management team couldn’t argue with the ROI. Over three years, the greener solution would save VistaPrint $450,000—more than the cost of the hardware refresh.

Cleaner, Cheaper Electricity

By the fall of 2006, competition over parking in VistaPrint’s Lexington lot led its leaders to conclude it was time to move again, to a bigger space across the street.

As part of its earlier move in the summer of 2004, the company had inherited a data center in the basement of its existing building. The “free data center,” as Branham calls it, housed 18 racks of equipment by the time he arrived. Most of the gear was used for internal IT systems and development. They’d probably want to move that operation across the street with them. Or would they? The rework of the data center in Bermuda got Cebula and Branham thinking. If they could cut energy costs by making different gear choices, was there anything to be gained by moving the bulk of the Lexington data center elsewhere? While researching the Bermuda data center revamp, Cebula had learned that the power market in Massachusetts was perhaps the most expensive one in North America. “Outside of maybe Hawaii,” she adds. “Doing those cost comparisons opened our eyes and got us to run some of the numbers on power at different possible locations.”

VistaPrint had recently opened a 68,000-square-foot manufacturing facility in Windsor, Canada, which could be expanded to accommodate a 100-rack data center. Cebula charged Branham with creating a business case for three options: building the new data center in the new Lexington space, building it in Windsor, or outsourcing it.

Unlike most IT departments, Vista­Print had all the information it needed to make the analysis—from real estate costs to electric bills—at hand. The majority of IT shops are in the dark about their data center energy costs because facilities management pays the bills. Facilities and IT tend to operate on different planes. “Facilities people know how much power is being used, but they don’t know anything about the equipment,” says Koomey. The result is what the Uptime Institute’s Brill calls “the invisible crisis.” IT has little incentive to make more conservative choices about power consumption, and the results are not sustainable. AFCOM, an association for data center professionals, predicts that power failures or power limits will halt data center operations at more than 90 percent of companies over the next five years.

“Not a lot of IT people get down into the building of data centers,” admits Branham. “We definitely understand the total cost perspective,” says Cebula. “But even we were a bit surprised when doing the analysis of actual energy consumption.”

Branham’s analyses revealed some good news. Real estate costs would be cheaper in Canada ($7 per square foot in Windsor versus $27.45 in Lexington). But Windsor was a greener pasture in more ways than one. VistaPrint’s manufacturing facility is run on hydroelectric power, a renewable energy source that doesn’t entail fuel costs and comes at a lower and more stable price. (In the Boston area, electricity comes largely from generation facilities that run on coal, oil or nuclear power.) That’s why data hogs like Yahoo and Google have recently begun building operations in the Northwest near the Columbia River, a source of hydroelectric power. “Any small difference in energy consumption or energy costs can make quite a difference,” says Kumar. Hydroelectric power also creates little air pollution because, unlike fossil fuel–fired plants, it does not generate carbon dioxide. Even nuclear plants generate some thermal emissions.

Sure enough, thanks to a 12-cent per kilowatt-hour difference in electricity prices, Branham figured he would save more than $130,000 a year by locating in Canada. Those cost savings would improve as VistaPrint added racks to the facility—a near certainty at its current growth rate. Coupled with the discount on real estate, the Windsor option proved 10 percent cheaper than outsourcing—even with the capital expenditures.

Not everyone can just up and move a data center to a cheaper real estate market with a renewable energy source. “Banks, for example, may need to be in downtown Manhattan or central London,” says Kumar. “But what [VistaPrint is] doing makes a lot of sense for them.”

The Benefits of Green

The Bermuda project (which was completed at the end of last year) enables VistaPrint to shave 25 percent from its future hosting costs. And VistaPrint’s Windsor data center is scheduled to open this month. Branham plans to move many of the systems hosted in Lexington there (where the new virtual machines initially will occupy 10 to 12 racks), and the facility will also provide disaster recovery for the Bermuda data center (a service the company outsourced to Cable & Wireless). Cebula says VistaPrint is trying to make other eco-friendly choices, outside of IT. For example, the company bought an extraction system for its plant that recovers more paper waste, which VistaPrint can resell to recyclers. “What we’re finding is that the right thing to do environmentally and the right thing to do financially often go hand in hand,” she says. Some green options, like the new servers and the paper waste extractor, may involve up-front costs, Cebula says, but they usually pay for themselves and reap returns (such as revenue from reselling the excess paper and lower energy costs). Of course, not every green product or process has a payoff for everyone. Some of the new, environmentally friendly innovations in cooling technology, heat transfer and power supply for the data center (see “Fresh, Green Tech, Page 40) didn’t make the list at VistaPrint. “I did look at the price for a natural gas generator,” Branham says. “But the prices for natural gas fluctuate widely. Most people stay with diesel.”

Determining the environmental benefits of VistaPrint’s data center revamp is hard because of the complexities in how power is produced and consumed. But based on a model for calculating carbon emissions offered by Lawrence Berkeley, an average U.S. company that cuts its electricity consumption the way VistaPrint did in Bermuda would reduce its carbon emissions by nearly 612 metric tons.

Meanwhile, the benefits of VistaPrint’s data center changes go beyond saving money or saving the planet. Server standardization has made maintenance and upgrades easier. It’s also made IT more responsive to business needs. It used to take days to provision a new server in Bermuda. Now it takes minutes. “The business is much more satisfied with the time it takes to make changes or get new capacity,” says Cebula. “And with our better failover capabilities, we have fewer reductions in service.”

VistaPrint recently had a VMware server go down in Bermuda, and all the instances that were running on it restarted on another server. Total downtime was less than 10 minutes, with no human intervention. The business doesn’t understand why service is better, says Cebula, “but they understand the better service.” And the money IT might have been throwing away on sky-high electric bills in Massachusetts can be fueled into more sound IT investments.

“Green isn’t costing us money. Green is saving us money,” says Branham. “Interests are aligned.”

Senior Editor Stephanie Overby can be reached at soverby@cio.com.