Table of Contents

Contributors xxi


Preface xxiii


Acknowledgments xxv


Technical Advisory Board xxvii


Chapter Organization xxix


Part I Data Center Overview and Strategic Planning 1


1 Data Centers—Strategic Planning, Design, Construction, and Operations 3
Hwaiyu Geng


1.1 Introduction 3


1.2 Data Center Vision and Roadmap 6


1.3 Strategic Location Plan 7


1.4 Sustainable Design 8


1.5 Best Practices and Emerging Technologies 10


1.6 Operations Management and Disaster Management 10


1.7 Business Continuity and Disaster Recovery 12


1.8 Conclusion 12


References 13


Further Reading 14


2 Energy and Sustainability in Data Centers 15
William J. Kosik


2.1 Introduction 15


2.2 Flexible Facilities—Modularity in Data Centers 18


2.3 Water Use 21


2.4 Proper Operating Temperature and Humidity 21


2.5 Avoiding Common Planning Errors 23


2.6 Cooling System Concepts 26


2.7 Building Envelope and Energy Use 28


2.8 Air Management and Containment Strategies 30


2.9 Electrical System Efficiency 32


2.10 Energy Use of IT Equipment 33


2.11 Leveraging IT and Facilities 37


2.12 Determining Data Center Energy Use Effectiveness 39


2.13 Private Industry and Government Energy Efficiency Programs 42


2.14 USGBC—LEED Adaptations for Data Centers 42


2.15 Harmonizing Global Metrics for Data Center Energy Efficiency 42


2.16 Industry Consortium—Recommendations for Measuring and Reporting Overall Data Center Efficiency 42


2.17 Strategies for Operations Optimization 44


References 44


Further Reading 44


3 Hosting or Colocation Data Centers 47
Chris Crosby and Chris Curtis


3.1 Introduction 47


3.2 Hosting 47


3.3 Colocation (Wholesale) 48


3.4 Types of Data Centers 48


3.5 Scaling Data Centers 54


3.6 Selecting and Evaluating DC Hosting and Wholesale Providers 54


3.7 Build versus Buy 54


3.8 Future Trends 56


3.9 Conclusion 57


Further Reading 57


Sources for Data Center Industry News and Trends 57


4 Modular Data Centers: Design, Deployment, and Other Considerations 59
Wade Vinson, Matt Slaby, and Ian Levine


4.1 Modular Data Center Definition 59


4.2 MDC Benefits and Applications 59


4.3 Modularity Scalability Planning 61


4.4 MDC Anatomy 62


4.5 Site Preparation, Installation, Commissioning 80


4.6 How to Select an MDC Vendor 85


4.7 External Factors 86


4.8 Future Trend and Conclusion 86


Further Reading 87


5 Data Center Site Search and Selection 89
Ken Baudry


5.1 Introduction 89


5.2 Site Searches Versus Facility Searches 89


5.3 Globalization and the Speed of Light 90


5.4 The Site Selection Process 93


5.5 Industry Trends Affecting Site Selection 101


Further Reading 102


6 Data Center Financial Analysis, ROI and TCO 103
Liam Newcombe


6.1 Introduction to Financial Analysis, Return on Investment, and Total Cost of Ownership 103


6.2 Financial Measures of Cost and Return 109


6.3 Complications and Common Problems 116


6.4 A Realistic Example 126


6.5 Choosing to Build, Reinvest, Lease, or Rent 135


Further Reading 137


7 Overview of Data Centers in China 139
Zhe Liu, Jingyi Hu, Hongru Song, Yutao Yang, and Haibo Li


7.1 Introduction 139


7.2 Policies, Laws, Regulations, and Standards 141


7.3 Standards 145


7.4 Development Status of China’s Data Centers 147


7.5 Energy Efficiency Status 149


7.6 Development Tendency 150


References 151


8 Overview of Data Centers in Korea 153
Minseok Kwon, Mingoo Kim, and Hanwook Bae


8.1ntroduction 153


8.2 Korean Government Organizations for Data Center 154


8.3 Codes and Standards 154


8.4 Data Center Design and Construction 155


8.5 Data Center Market 159


8.6 Conclusion 160


References 160


Part II Data Center Design and Construction 161


9 Architecture Design: Data Center Rack Floor Plan and Facility Layout Design 163
Phil Isaak


9.1 Introduction 163


9.2 Overview of Rack and Cabinet Design 163


9.3 S pace and Power Design Criteria 166


9.4 Pathways 169


9.5 Coordination with Other Systems 170


9.6 Computer Room Design 174


9.7 Modular Design 177


9.8 CFD Modeling 178


9.9 Data Center Space Planning 179


9.10 Conclusion 181


Further Reading 181


10 Mechanical Design in Data Centers 183
John Weale


10.1 Introduction 183


10.2 Key Design Criteria 183


10.3 Mechanical Design Process 186


10.4 Data Center Considerations in Selecting Key Components 203


10.5 Primary Design Options 206


10.6 Current Best Practices 211


10.7 Future Trends 214


Reference 215


Further Reading 215


11 Electrical Design in Data Centers 217
Jay S. Park and Sarah Hanna


11.1 Uptime 217


11.2 Electrical Equipment to Deploy 217


11.3 Electrical Design 217


11.4 Availability 222


11.5 Determining Success 227


Appendix 11.A 228


Further Reading 228


12 F ire Protection and Life Safety Design in Data Centers 229
Sean S. Donohue


12.1 F ire Protection Fundamentals 229


12.2 AHJs Codes and Standards 230


12.3 Local Authorities, National Codes, and Standards 230


12.4 Life Safety 231


12.5 Passive Fire Protection 233


12.6 Active Fire Protection/Suppression 234


12.7 Detection, Alarm, and Signaling 239


12.8 Fire Protection Design 242


References 243


13 Structural Design in Data Centers: Natural Disaster Resilience 245
David Bonneville and Robert Pekelnicky


13.1 Introduction 245


13.2 Building Design Considerations 246


13.3 Earthquakes 248


13.4 Hurricanes Tornadoes and Other Windstorms 251


13.5 Snow and Rain 252


13.6 Flood and Tsunami 253


13.7 Comprehensive Resiliency Strategies 254


References 255


14 Data Center Telecommunications Cabling 257
Alexander Jew


14.1 Why Use Data Center Telecommunications Cabling Standards? 257


14.2 Telecommunications Cabling Standards Organizations 259


14.3 Data Center Telecommunications Cabling Infrastructure Standards 259


14.4 Telecommunications Spaces and Requirements 262


14.5 Structured Cabling Topology 264


14.6 Cable Types and Maximum Cable Lengths 267


14.7 Cabinet and Rack Placement (Hot Aisles and Cold Aisles) 269


14.8 Cabling and Energy Efficiency 270


14.9 Cable Pathways 271


14.10 Cabinets and Racks 272


14.11 Patch Panels and Cable Management 272


14.12 Reliability Levels and Cabling 272


14.13 Conclusion and Trends 273


Further Reading 273


15 Dependability Engineering for Data Center Infrastructures 275
Malik Megdiche


15.1 Introduction 275


15.2 Dependability Theory 276


15.3 System Dysfunctional Analysis 283


15.4 Application to Data Center Dependability 297


Reference 305


Further Reading 305


16 Particulate and Gaseous Contamination in Data Centers 307
Taewon Han


16.1 Introduction 307


16.2 Standards and Guidelines 307


16.3 Airborne Contamination 309


16.4 A Conventional Solution 309


16.5 Conclusions and Future Trends 311


Acknowledgment 311


References 312


Further Reading 312


17 Computational Fluid Dynamics Applications in Data Centers 313
Mark Seymour


17.1 Introduction 313


17.2 Fundamentals of CFD 313


17.3 Applications of CFD for Data Centers 321


17.4 Modeling the Data Center 325


17.5 Potential Additional Benefits of a CFD/Virtual Facility Model 340


17.6 The Future of Virtual Facility Models 341


References 341


18 Environmental Control of Data Centers 343
Veerendra Mulay


18.1 Data Center Power Trends 343


18.2 Thermal Management of Data Centers 343


18.3 Cooling System Design and Control 346


18.4 Performance Metrics 352


References 353


19 Data Center Project Management and Commissioning 359
Lynn Brown


19.1 Introduction 359


19.2 Project Management 359


19.3 Commissioning 367


19.4 Bidding Phase Tasks 376


19.5 Acceptance Phase Tasks 378


19.6 LEED-Required Commissioning Tasks 381


19.7 Minimum Commissioning Tasks 382


19.8 Commissioning Team Members 383


19.9 Data Center Trends 386


19.10 Conclusion 387


Further Reading 387


Part III Data Center Technology 389


20 Virtualization, Cloud SDN, and SDDC in Data Centers 391
Omar Cherkaoui and Ramesh Menon


20.1 Introduction 391


20.2 Virtualization in Data Centers 392


20.3 Cloud as an Extension of the Data Center 393


20.4 Networking in Data Center 394


20.5 SDN 396


20.6 SDDC 398


20.7 Roadmap to Cloud-Enabled Data Center 398


References 400


Further Reading 400


21 Green Microprocessor and Server Design 401
Guy AlLee


21.1 Introduction 401


21.2 Microprocessor 403


21.3 Server 407


21.4 Motherboard 409


21.5 Software 413


21.6 Benchmarks 415


21.7 Conclusions 416


Further Reading 417


22 Energy Efficiency Requirements in Information Technology Equipment Design 419
Joe Prisco and Jay Dietrich


22.1 Introduction 419


22.2 Computer Servers 421


22.3 Storage Systems 425


22.4 Uninterruptable Power Systems 426


22.5 Networking Equipment 427


22.6 Future Trends in Product Energy Efficiency Requirements 427


References 428


Further Reading 428


23 Raised Floor versus Overhead Cooling in Data Centers 429
Vali Sorell


23.1 Introduction 429


23.2 History of Raised Floor versus Overhead Air Distribution 429


23.3 Air Delivery Methodology as it Relates to Containment 430


23.4 Airflow Dynamics 430


23.5 Under-floor Air Distribution 433


23.6 Overhead Air Distribution 437


23.7 Conclusion 439


References 439


Further Reading 439


24 Hot Aisle versus Cold Aisle Containment 441
Dave Moody


24.1 Executive Summary 441


24.2 Containment: The Airflow Architecture Models 441


24.3 Return Air Temperature Trends in HAC and CAC 444


24.4 Run- or Ride-Through Impact of Higher RAT 446


24.5 Single-Geometry Passive Chimney Ducts as Part of HAC 448


24.6 Psychological Impacts of Higher RAT 450


24.7 Cooling System Airflow and Fan Power 453


24.8 Redundancy and Cooling Unit Location Impact 459


24.9 Impact on Conditions for Peripheral Equipment in the Data Center Outside any of the HAC or CAC Zone(s) 461


24.10 Impact on Economizer Operation Time Periods During Cooler Outside Ambient Temperatures 462


24.11 Conclusion and Future Trends 463


References 464


Further Reading 464


25 Free Cooling Technologies in Data Centers 465
Nicholas H. Des Champs and Keith Dunnavant


25.1 Introduction 465


25.2 Using Properties of Ambient Air to Cool a Data Center 466


25.3 Economizer Thermodynamic Process and Schematic of Equipment Layout 466


25.4 Comparative Potential Energy Savings and Required Trim Mechanical Refrigeration 475


25.5 Conventional Means for Cooling Datacom Facilities 478


References 478


Further Reading 478


26 Rack-Level Cooling and Cold Plate Cooling 479
Henry Coles, Steve Greenberg, and Phil Hughes


26.1 Introduction 479


26.2 Rack-Level Cooling Types 482


26.3 Rack-Level Cooler Selection and Installation 485


26.4 Conclusion and Future Trends 486


26.5 Rack-Level Cooling Using Cold Plates 486


26.6 Conclusions and Future Trends 492


References 493


Further Reading 493


27 Uninterruptible Power Supply System 495
Chris Loeffler and Ed Spears


27.1 Introduction 495


27.2 Principle of UPS and Application 496


27.3 Considerations in Selecting UPS 504


27.4 Reliability and Redundancy 507


27.5 Alternate Energy Sources: AC and DC 512


27.6 UPS Preventive Maintenance Requirements 516


27.7 UPS Management and Control 519


27.8 Conclusion and Trends 520


Reference 520


Further Reading 520


28 Using Direct Current Network in Data Centers 523
Sofia Bergqvist


28.1 Introduction 523


28.2 Edison’s Revenge 523


28.3 Data Center Power Design 525


28.4 Why Use the DC System in Data Centers 526


28.5 Examples of DC Data Centers in Operation 531


28.6 Future Trends and Conclusions 532


Acknowledgments 532


References 532


Further Reading 532


29 Rack PDU for Green Data Centers 533
Ching-I Hsu


29.1 Introduction 533


29.2 Fundamentals and Principles 534


29.3 Elements of the System 540


29.4 Considerations for Planning and Selecting Rack PDUs 548


29.5 Future Trends for Rack PDUs 555


Further Reading 557


30 Renewable and Clean Energy for Data Centers 559
William Kao


30.1 Introduction 559


30.2 Renewable Energy Basics 560


30.3 Renewable Energy Types 560


30.4 Alternative Energy: Fuel Cell 569


30.5 Case studies 573


30.6 Summary and Future Trends 575


References 576


Further Reading 576


31 Smart Grid-Responsive Data Centers 577
Girish Ghatikar, Mary Ann Piette, and Venkata Vish Ganti


31.1 Introduction and Context for Grid-Responsive Data Centers 577


31.2 S mart Grid and DR Applications in the United States 579


31.3 Site Infrastructure Control System Technologies 581


31.4 IT Infrastructure Virtualization Technologies 582


31.5 DR Opportunities, Challenges, and Automation Considerations 582


31.6 Data Centers with DR Provisions 583


31.7 AutoDR Using Open Standards 585


31.8 Grid-Distributed Data Centers and Networks 586


31.9 Summary of DR Strategies 586


31.10 Challenges to Grid-Responsive Data Centers 587


31.11 U.S. Policies Governing Smart Grid Emerging Technologies 588


31.12 The Energy Independence and Security Act of 2007 588


31.13 State Policies for Smart Grid Advancement 589


31.14 Conclusions and Next Steps 589


Acknowledgments 590


References 591


Further Reading 592


Part IV Data Center Operations and Management 593


32 Data Center Benchmark Metrics 595
William J. Kosik


32.1 Introduction 595


32.2 Origin and Application of PUE as a Metric 595


32.3 Metrics Used in Data Center Assessments 597


32.4 Green Grid’s xUE Metrics 597


32.5 Rack Cooling Index and Return Temperature Index 598


32.6 Additional Industry Metrics 598


32.7 European Commission Code of Conduct 598


32.8 International Telecommunication Union 599


32.9 Conclusion 599


Further Reading 599


33 Data Center Infrastructure Management 601
Mark Harris


33.1 What is Data Center Infrastructure Management? 601


33.2 Triggers for DCIM Acquisition and Deployment 604


33.3 What are the Modules of a DCIM Solution? 606


33.4 The DCIM System Itself. What to Expect and Plan for 611


33.5 Critical Success Factors when Implementing a DCIM System 614


33.6 Future Trends in DCIM 616


33.7 Conclusion 617


Reference 617


Further Reading 617


34 Computerized Maintenance Management System in Data Centers 619
Peter Sacco


34.1 Introduction 619


34.2 CMMS Basics 620


34.3 CMMS Modules 620


34.4 Considerations in Selecting CMMS 632


34.5 Conclusion 637


34.6 Trends 637


Further Reading 638


Part V Disaster Recovery and Business Continuity 639


35 Data Center Disaster Recovery and High Availability 641
Chris Gabriel


35.1 Introduction 641


35.2 The Evolution of the Data Center and Data Center Risk 642


35.3 Physical Data Center Design and Redundancy: Tiers and N+ What? 649


35.4 Virtualization Brings Out-of-the-Box DR Survivability 652


35.5 DR and Cloud 656


References 657


Further Reading 657


36 Lessons Learned from Natural Disasters and Preparedness of Data Centers 659
Hwaiyu Geng and Masatoshi Kajimoto


36.1 Introduction 659


36.2 Design for Business Continuity and Disaster Recovery 659


36.3 Natural Disasters 660


36.4 The 2011 Great East Japan Earthquake 660


36.5 The 2012 Eastern U.S. Coast Superstorm Sandy 663


36.6 Conclusions 666


References 666


Further Reading 666


Index 669

About the Author

Hwaiyu Geng, P.E., is a consultant with Amica Association, promoting green planning, design, and construction projects. He has had over 40 years of manufacturing and management experience, working with Westinghouse, Applied Materials, HP, and Intel on multi-million high-tech projects. He has written and presented numerous technical papers at DatacenterDynamics, ASME and IIE conferences, and is also the editor/author of Manufacturing Engineering Handbook (McGraw Hill, 2004), Semiconductor Manufacturing Handbook (McGraw Hill, 2005), and Data Center Handbook (Wiley, 2014). He is a patent holder.