ENT022 – Data Centre Efficiency & Expansion
|Tony Weir||Project Sponsor||Director, IT Infrastructure||29/07/2019|
|Graeme Wood||Senior User||
Head, ITI Enterprise Services
|Grant Ferguson||Senior Supplier||Director, Estates Operations|
ITI Portfolio Manager
|Alan Carracher||Lead Engineer||Estates|
|Alan Carstairs||Operations Manager||FES|
|Caroline Brown||Procurement Manager||Estates|
|Dean Carrick||QS Consultant||Doig & Smith|
Mechanical Engineering Consultant
|Gerry McCafferty||Operations Manager||FES|
|John Girdwood||Electrical Engineering Consultant||RSP|
|Lawrence Stevenson||Project Manager||ISG Project Services|
|Mike Dixon||Principal Designer||Lewis & Hickey|
|Paul Hutton||Data Centre Lead||ITI Enterprise Services|
|Sheila Scott||Building Services Manager||Estates|
This project was initiated to deliver power and cooling improvements to the ISG data centres at the James Clerk Maxwell Building (King’s Buildings) and at the Appleton Tower (George Square). The project was a major collaboration between the IT Infrastructure team and the University Estates group supported by external consultants and contractors.
The installation and commissioning work at the data centres was preceded by a resilience and efficiency review reported by AECOM in December 2016, a feasibility study completed by RSP in February 2018 and full design work, also by RSP, completed in May 2018. AECOM also produced an updated Data Centre Handbook, in collaboration with ITI and Estates staff, in August 2017.
After an open tender process run under the Estates Purchasing Protocol and Scottish Government/OJEU procurement rules, the full M&E installation and commissioning contract was awarded to FES, who started work in August 2018. All major work was completed on schedule in April 2019. Replacement of air conditioning units at JCMB, not in the original scope, and completion of cold aisle containment in Row F at JCMB, delayed by the installation of security cage around a tape unit for the Data Safe Haven (DSH) project, took project completion to June 2019.
The project implemented measures to ensure that in the University’s data centres, the exposure to the risk to power losses was significantly reduced, through the installation of continuous power supplies for servers, storage and networking equipment hosting University application services. In addition, through the installation of modern and efficient electrical equipment and cold aisle containment partitioning, power efficiency would be improved at both the JCMB and Appleton Tower data centres. A major goal of the project was to reduce the PUE figure of 1.37 recorded at JCMB at the start of the project. Finally, the project delivered a significant increased to UPS capacity at the JCMB data centre.
During the project, new dual power supply circuits were installed at JCMB. These circuits ensure that electrical supplies to the JCMB data centre will be maintained in the event of a component failure in one of the supply circuits and ensures the continual operation of servers and other equipment in the data centre, contributing to improved system services levels to the University community.
In addition, the existing UPS batteries in the Appleton Tower data centre was replaced and relocated from the main data hall to an external secure room to safeguard the lifetime of the batteries and mitigate against increased temperatures in the data hall.
Finally, to reduce temperatures and increase power efficiency, cold aisle partitioning, and entry doors were installed around the rows of server racks in both data centres. These measures provide more efficient power utilisation and support the University’s green policies.
The project scope covered the delivery of M&E improvements to the ISG data centres as JCMB and Appleton Tower. The original scope for both data centres covered -
- The design, supply, installation, test and commissioning of dual power circuits from local electrical substations to each data centre
- The design, supply, extension, test and commissioning of existing power distribution boards to monitor and control the new circuits
- The design, supply, installation, test and commissioning new PDUs, containment and cabling to all racks within the data hall
- The design, supply, installation, test and commissioning of new modular UPS units
- The design, supply, installation and commissioning of cold aisle containment measures
- The design, installation and test of additional circuits to enable the installation of a 2nd power generator, at a later date
However, in March 2018, due to budget limitations, the scope of the project at the Appleton Tower was reduced by shelving the main electrical work and limiting work at that data centre to replacing and relocating the existing UPS batteries and installing cold aisle containment. The full scope of work at JCMB was retained with the installation of dual end-to-end power circuits and cold aisle containment.
The original plan was to deliver the upgrades to both data centres during the summer break in August 2018. However, due to the length of the procurement cycle, the planned switchover to the new power string at JCMB in August 2018 was delayed to the winter break in January 2019. After the plan was revised to accommodate a longer than estimated procurement cycle, all major work was delivered on schedule.
The key objectives of the Data Centre project were to provide –
- Improved power redundancy at the JCMB data centre, through increased UPS capacity
- Improved service resilience at the JCMB and Appleton Tower data centres by installing dual end-to end power cabling, expanded distribution boards, new PDU units and new UPS systems
- Improved power efficiency by commissioning more effective electrical equipment and installing cold aisle at JCMB and Appleton Tower
- Classification of the JCMB data centre as a Tier II (Uptime Institute) data centre
Excluding the reduction of scope at the Appleton Tower, the main objectives of the project were achieved and confirmed. The principal goals of the project, improved resilience and reduced power consumption at both data centres have been met.
In addition, an unplanned activity, the replacement of air conditioning units in secure rooms at JCMB, was completed, in part due to the age and condition of these units after inspection during the installation work.
The key requirements identified during planning for the Data Centre project are summarised in the table below.
|Improved power resilience and UPS capacity in JCMB data centre||Must||Delivered|
|Improved power resilience in Appleton Tower data centre||Must||Partially delivered, replacement UPS batteries installed and commissioned|
|Improved cooling efficiency at JCMB data centre||Must||Delivered|
|Improved cooling efficiency at Appleton Tower data centre||Must||Delivered|
|Improved labelling of existing cabling and equipment at both data centres||Must||Delivered|
|Full up-to-date O&M documentation covering the M&E environments delivered by the project||Must||Delivered|
|Updated ITI Enterprise Services Data Centre handbook||Must||Delivered, but needs to be updated|
|Classification of JCMB as a Tier II data centre||Should||New infrastructure does not meet full Tier II classification|
|Replacement of air conditioning equipment for secure rooms at JCMB||Should||Delivered, added to scope|
More detail on the status of these objectives and the corresponding deliverables is given in the table below.
|Improved power resilience in JCMB data centre||Install and commission dual resilient power supplies to data hall - install new switchgear and expand existing switchgear; remove existing and install new UPS equipment, install new cabling from substation to switchgear, remove existing PDUs and install new PDUs in data hall, install cabling and from switchgear to PDUs, install cabling and containment from PDUs to racks; commissioning and load bank testing||Cabling and equipment identified, sourced, installed/upgraded, checked, integrated, tested and commissioned as specified and on schedule; new power string operational on 12/01/2019, upgraded power string operational on 06/04/2019; removal of decommissioned equipment|
|Improved power resilience in Appleton Tower data centre||Install and commission new UPS equipment in secure room, switchover from existing UPS in data hall to new UPS, remove old UPS equipment (reduced scope)||UPS batteries, racks and cabling identified, sourced, installed, tested and commissioned to replace existing UPS batteries on schedule on 22/03/2019; removal of decommissioned equipment|
|Improved cooling efficiency at JCMB data centre||Design, source and install cold aisle containment and entry doors around racks in data hall||Cold aisle partitioning and security doors designed, sourced and installed on schedule on 21/12/2018 apart from the thermal curtains installed around the DSH tape system/cage to complete Row F (14/06/2019)|
|Improved cooling efficiency at Appleton Tower data centre||Design, source and install cold aisle containment and entry doors around racks in data hall||Cold aisle partitioning and security doors designed, sourced and installed on schedule on 01/03/2019|
|Improved labelling of existing cabling and equipment at both data centres||Upgrade labelling on existing equipment to industry standards; ensure new cabling and equipment is labelled to industry standards||All labelling upgrade or installed, as required, to industry standards, confirmed 18/06/2019|
|Full up-to-date O&M documentation covering the M&E environments delivered by the project||Fully documented engineering documents and drawings reflecting the installed M&E environment in both data centres||Full operating and maintenance documents and drawings, in web, disk and paper formats, provided by FES to the University’s Estates lead engineer on 18/06/2019|
|Updated ITI Enterprise Services Data Centre handbook||Fully documented handbook describing data centre electrical, mechanical, fire prevention systems and single points of failure||Provided by AECOM in August 2017; to be updated|
|Classification of JCMB as a Tier II data centre||Tier II classification achieved by having data centre infrastructure with redundant capacity components (secondary generator, UPS, chillers, heat rejection equipment, pumps, cooling units, 12-hours onsite fuel) and non-redundant distribution paths||Many measures to meet Tier II classification have been implemented at JCMB and non-redundant distribution paths are in place, but the data centre is not fully compliant, at present|
|Replacement of air conditioning equipment for secure rooms at JCMB (not in original scope)||Air conditioning units in 5 secure rooms to be installed to replace end-of-line and faulty units||
Final air conditioning units installed and commissioned on 13/05/2019
The key benefits that would be realised after the M&E upgrades at the data centres, presented during project initiation, were as follows –
Improved availability (reduced MTBF) for systems and network services at both data centres
- Since April 2019 when the second power string was brought into operation no major service outage has occurred, but this needs to be monitored over a 2-3-year period
- No service outages during planned power maintenance activity
- No loss of any service occurred during the routine generator black start test in April 2019, but again this needs to be monitored over a longer period
- Reduction in energy consumption and costs, reduced carbon emissions
- Energy consumption figures in April-June 2019 were 4% less than in the corresponding quarter in 2018
- More accurate PUE data derived from improved metering
- The initial readings show a PUE figure of 1.37, up from 1.33, but this has been caused by an increase in power consumption by the air handling units (up 50%); this is being investigated and more savings are predicted when air flow rates and cooling set points are adjusted
- Potential to provide commercial host services to external organisations with a Tier 2 data centre (JCMB)
- This activity has not been pursued to date as Tier 2 accreditation is yet to be undertaken
Alignment with Strategy
University Strategic Vision
This section reviews the Data Centre project objectives, at the closure of this project, against the University’s strategic vision.
|A unique Edinburgh offer for all our students|
||Improved service continuity will indirectly affect student satisfaction|
|Strong and vibrant communities within and beyond the University – making the most of our unique offer of world-leading thinking and learning within one of the world’s most attractive cities||N/A|
|A larger, more international staff who feel valued and supported in a University that is a great and collegial place to work, develop and progress||N/A|
|More postgraduate students – underpinned by the best support in the sector to ensure we attract the brightest and best regardless of ability to pay||N/A|
|A strong culture of philanthropic support focussed especially on our students and on outstanding research capabilities.||N/A|
|Many more students benefiting from the Edinburgh experience (largely or entirely) in their own country – supported by deep international partnerships and world leading online distance learning||N/A|
|Sustained world leading reputation for the breadth, depth and inter-disciplinary of our research supported by strong growth in research funding and strong international partnerships – drawing from well-established and less well-developed sources||N/A|
|An estate that matches expectations, responds flexibly to changing student and staff needs, and showcases the University||The new data centres’ M&E infrastructures contributes to more reliable IT services for students and supports the University’s environmental goals|
|A deeper and earlier collaboration with industry, the public sector and the third sector – in terms of research; knowledge exchange; and in giving our students the best possible set of skills for their future||Potential for providing data centre services to external organisations; telecommunications and network equipment for various NGOs is currently hosted at JCMB who will benefit from the improved resilience and reduced energy consumption delivered by the project|
|Digital transformation||The project provides a data centre M&E infrastructure that indirectly supports the digital transformation of University services through increased power resilience|
ISG Strategic Vision
This section reviews retrospectively how the Data Centre project objectives, at the closure of this project, contributed to ISG’s strategic vision.
||More resilient IT data centre services may contribute to an improved student experience|
||More resilient IT data centre services may contribute to improved online learning|
||More resilient IT data centre services may contribute to improved library systems|
|Research and Innovation|
||More resilient IT data centre services may contribute to improved research IT and data science services|
||The project delivered more resilient IT data centre services and reduced energy consumption on a par with many UK universities|
||The project has delivered energy reduction measures that contribute to the University’s environmental goals|
|Process improvement, efficiency, quality and best practice||The project has delivered reduced energy consumption and improved service resilience within both data centres as JCMB moves towards Tier II data centre classification|
|Long-term IS strategic planning and linked professional services||The project provides the possibility of the University hosting external organisations at the JCMB data centre as it moves towards Tier II classification|
|Information Security||The project delivered entry doors at the end of each aisle in the data halls at JCMB and Appleton Tower, contributing to improved physical security measures|
IS Change Programme
This section reviews retrospectively how the Data Centre project, at closure, contributed to the themes of the IS Change Programme.
|Project Management||Formal ISG project management processes were applied successfully and effectively and the main contractor (FES) also followed a formal project management process for their work|
|Working Together||Excellent collaborative effort across team with internal representation from ISG, Estates, Procurement and Finance and external team members from RSP (design consultants), FES (main contractor), Doig & Smith (cost management) and Lewis and Hickey (H&S/principal designer)|
|Standards and Technical Leadership||All applicable M&E standards followed from design through to commissioning phases; latest electrical equipment (e.g. UPS) and cooling equipment installed and operational|
|Staff Learning and Development||Project provided ISG and Estates team members exposure to latest data centre M&E technologies and practices|
|Service Based Culture||Project provided more resilient power supplies to JCMB data centre, underpinning continuous operation of servers, storage and network equipment running the University’s application services|
|Equality and Diversity||N/A|
|Innovation||Latest data centre M&E equipment installed in JCMB and Appleton Tower data centres|
|Flexible Resourcing||Internal team supplemented by external consultancies and contractors to achieve project goals on schedule|
|Communication and Branding||All service owners informed and approved major work e.g. power string switch over – well in advance; JCMB data centre potentially a reference site for other Universities considering data centre improvements|
The Data Centre (ENT022) project was initiated in January 2017 after AECOM produced a data centre M&E review in December 2016. Much of 2017 was spent developing the project scope and project planning, plus preparing a resilience study for the data centres and producing an updated data centre handbook in conjunction with AECOM. Budget estimates were prepared, and funding was secured towards the end of 2017. The project did not benefit from having two changes of project manager during this period.
With funding approved in December 2017, RSP were engaged to develop a feasibility study for the proposed M&E upgrades. The study was concluded in February 2018 and RSP were then awarded the design contract and helped to prepare the invitation to tender (ITT) for the M&E contract. Design work was completed, and the ITT was published (under Scottish Government procurement and OJEU policies) at the end of May 2018.
During the preparation of the ITT, it became obvious that the planned delivery date for the new power string in August 2018 (summer break) could not be met with an ITT process that would extend into June and July. The plan was revised so that the switch over to the new power string in JCMB was scheduled for January 2019 (summer break) and the roll out of the upgraded power string was planned for March 2019. The latter activity was rescheduled to April 2019, after a major upgrade to the University’s HR system was scheduled for the March date.
Preparation for the electrical upgrades and installation of cold aisle containment were scheduled from the revised start of contractor (FES) engagement in August 2018. In consultation with FES and RSP, key milestone dates from the revised plan were agreed –
- Completion of cold aisle containment at JCMB – 21/12/2018
- Switch over to new (green) power string at JCMB – 12/01/2019
- Completion of cold aisle containment at Appleton Tower – 01/03/2019
- UPS battery relocation and replacement at Appleton Tower – 22/03/2019
- Upgraded of existing (grey) power string at JCMB – 06/04/2019
All these key dates were met. However, during the installation work at JCMB several air conditioning units in the UPS room were found to be faulty and required to be replaced. This was completed prior to the January switch over. However, it was decided to review the condition of other AC units in the secure rooms at JCMB and these were also found to be faulty or end of life, so additional work was awarded to FES to replace these units which was completed in mid-May 2019.
Finally, cold aisle work could not be completed on one part of Row F at JCMB in December 2018, as a tape system installed by the RSS team for the Data Safe Haven (DSH) project required to be enclosed in a secure cage as part of their ISO27001 work. The cage was not installed until April 2019 and survey and delivery lead times meant that the completion of cold aisle containment round the cage and tape system was not completed until mid-June 2019.
Once completed, the project moved into its contractor handover phase in the latter part of June and subsequently moved to the project closure phase in July 2019.
Internal resources were drawn from ISG and Estates with significant contributions from Paul Hutton, Alan Carracher and Caroline Brown. Extensive resources were also provided by the external consultancies and contractors.
Initial internal resource requirements, where estimated at 240 days. The estimated days of effort from internal staff is 263 days, an overrun of 9.5%, within tolerance. The main reason for the additional resource used was the rescheduling of the new power switch over at JCMB from August 2018 to January 2019 which meant project team members spent longer that planned on the project with an end date that moved for December 2018 to July 2019. Additional time in May-June 2019 for the replacement of AC units and completion cold aisle work of Row F at JCMB contributed to additional resourcing needs.
The RSP feasibility study in February concluded that the budget (£1.3m) could not meet the estimated cost of equipment and works (£1.9m) and so, the proposed electrical resilience work at Appleton Tower was reduced in scope, with only the replacement and relocation of UPS batteries retained. A review of the original budget planning showed that the bulk of the costs were for M&E equipment with limited provision for installation, testing and commissioning costs.
The overall project budget was £1,335,000 with project spend at £1,335,050, i.e. £50 [sic] over budget, which means the project was brought in on budget, including the additional unplanned spend for replacement AC units at JCMB. Actual costs for external equipment and works are summarised in the table below –
|FES||Main M&E contractor||£1,225,182|
|Doig & Smith||Cost Management||£12,042|
|Lewis & Hickey||Principal Designer||£3,300|
|FMS Fire & Safety||Alarm management||£112|
The key observations from the project are summarised in the table below –
|Inaccurate initial budget||Initial budget estimate including equipment costs but no installation and commissioning costs||Scope reduction in this project||Ensure M&E projects, particularly any additional work at Appleton Tower includes works based estimated|
|Time needed for SG/OJEU ITT process||Time for ITT procurement was over 5 months starting with P1 preparation in mid-February until end-July when contract was awarded||New power string switch-over at JCMB delayed from August 2018 (summer break) to January 2019 (winter break)||Ensure project planning includes appropriate time for preparation of ITT and SG/OJEU procurement process|
|Change of project manager during project||There were 3 different internal project managers during the project||Some delays caused by handover and familiarisation||Try to ensure major projects have a project manager who will be available for the full lifetime of the project|
|Project was on budget despite additional scope at JCMB||Replacement of AC units was not planned but were required to ensure effective cooling of UPS area and other secure rooms||AC units replaced at additional cost but offset by FES contract bid being lower than estimated||Ensure initial review, feasibility and design incorporate all impacted infrastructure|
|Effective use of Scottish Government/OJEU procurement process||Adherence to the Estates procurement policies, detailed planning of requirements and managed scoring of bids||The highest scoring and winning bid from FES was under the estimate budget figure, allowing costs to replace AC units to be covered||Ensure effective planning of ITTs to ensure competitive bids are received|
The following work, within the original project scope, was not completed –
- Installation of fully resilient power circuits at Appleton Tower
In addition, the following recommendations are made –
- Complete work at JCMB (e.g. 2nd generator) so that Tier II classification can be obtained
- Initiate a new project to complete power resilience work at Appleton Tower
- Update data centre handbook, as required
- Improve formal monitoring and recording of power usage and temperatures at both data centres
- Resolve issue with PUE increase at JCMB
- Integrate with the regular (6-monthly) power failover tests for JCMB data centre
Appendix 1 – Final Project Timeline
 Figure excludes any costs for Informatics UPS replacement
 Includes VAT