Report for May 2022

Report Date
May 2022

Achievements in Last Period

  • Issues are still being encountered with unconfirmed and revised delivery dates from ROC/Dell
    • Dell and ROC account managers have been actioned to provide firm delivery dates for orders and lead times for current quotes
  • Power calculations for AFC upgrades done; existing RFCs to be approved; remaining RFCs to be raised
  • DataStore
    • 15 x R640 servers delivered to JCMB; order for installation placed, awaiting confirmation of date
    • 2 x ME4084 0.5 PB storage racked at JCMB; installation date to be confirmed
    • 2 x ME4084 (1.0 PB) storage racked for DR at Appleton Tower; installation date to be confirmed
    • 1 x ME4084 for CDBS delivered; installation date to be confirmed
    • 3 x ME4084 ordered for ICG; delivery planned for 14/06/2022; to be confirmed
    • 18 x N1524 switches ordered; delivery date of December being queried with Dell; contingency – reconfigure existing switches
    • PCIe riser cards reordered; awaiting delivery date
    • Revised quote needed for 2 x R650 server; order to be placed
    • Quote received for 2 x ME5024 storage for DR; order to be placed
    • Identify storage arrays at JCMB to be powered down and removed prior to installation of new units
  • Eddie
    • Compute nodes (12 x C6400 chassis x 4 x C6420) in stress test/burn-in at OCF; completion of burn-in now estimated as 07/06/2022
      • Delivery to ACF planned for 11/07/2022 and installation week 01/08/2022
    • 4 x R640 storage servers racked at ACF; RFC raised;  installation date to be agreed; OCF to configure
    • 7 x DE6000 storage (Lenovo) delivered to ACF; RFC raised, installation date to be agreed; OCF to configure
    • 2 x 100GbE Mellanox network cards ordered; awaiting shipping/delivery date
    • 4 x 10/25GbE Mellanox network cards ordered; awaiting shipping/delivery date
    • Revised quote needed for 1 x R650 server plus storage; order to be placed
    • Quote received for 2 x C6400 x 4 x C6420 compute nodes (GridPP); date for order to be agreed
  • Eleanor
    • Compute nodes (2 x C6400 chassis x 4 x C6420) in stress test/burn-in at OCF
      • Delivery to ACF planned for 11/07/2022 and installation week 01/08/2022
      • OCF SoW for Eddie/Eleanor compute (and GPU) nodes approved
  • TSM
    • 2 x ME4084 (1.0 PB) storage racked at ACF; to be configured by RSS team
    • 3 x R650 servers ordered for ACF; current shipping date of 07/09/2022 being queried with Dell

Risks/Issues

  • Revised delivery and installation dates have introduced budget risk and significant re-planning effort  (-)

    • Some delivery dates have been revised and are now in FY2022-23 (-)
    • Full delivery dates and lead times to be confirmed by Dell week 06/06/2022 (+)
  • RSS resourcing on project is at risk from higher priority work and delivery/installation issues(-)
    • Plan needs to be revised to avoid higher priority work e.g., RSS301 (-)
    • Delayed delivery and installation dates are driving re-planning across all RSS projects (-)
  • Replacement needed for servers and storage approaching EoL (plus need for additional processing capacity) (-/+)
    • DataStore kit currently EoL with more going out of support in July/August (-)
  • Current allocated budget is £2.338M (including GridPP investment) and spend must be within tolerance (+/-)
    • Estimated overall spend is £2.320M and may be reduced if orders cannot be delivered in FY2021-22 (+)
  • Installation and/or configuration must be carefully coordinated with ACF (RFCs) and ISG (GoCAB) (+/-)
    • ISG GoCAB requests and ACF RFCs being raised as required (+)
  • Upgrade work will impact DataStore, Eddie and Eleanor services (-)
    • DataStore server configuration will mean momentary outages (-)
    • Reduced Eddie/Eleanor service when compute nodes are installed and configured (-)
  • Planned kit exceeds data centres’ power budget - review server/storage ratings against limit (-)
    • No issue at JCMB/AT (+)
    • Power analyses completed; reduction in ACF (+)
  • No space in racks at ACF and JCMB for additional kit (-)
    • Racking impact at ACF considered low risk (+)
    • Legacy storage will need to be removed at JCMB to accommodate new kit (-)
  • OCF and Lenovo resources may not be available for installation and configuration dates (-)
    • Date for Eddie/Eleanor compute nodes installation/ agreed with OCF (-)
    • Installation date for Eddie storage under discussion with Lenovo (+)
  • Storage of kit at ACF, JCMB and AT needs to be agreed and storage time minimised (-)
    • Storage needed are being communicated well in advance at ACF, JCMB, AT (+)
  • UCU industrial action may impact project progress (-)
    • Action in progress, impact consider minimal (+)

Next Steps

  • Revise plan in line with shipping delays, higher priority projects and RSS resourcing limitations
  • Update budget/spend figures when Dell delivery dates/lead times are provided
  • Place final orders to meet FY2021-22 budget deadline
    • 2 x R650 servers (DataStore)
    • 1 x R650 server and storage (Eddie)
    • 2 x ME5024 storage (DataStore DR)
    • 2 x C6400 x 4 X C6420 (Eddie/GridPP)
  • Complete installation planning of DataStore servers and storage at JCMB
    • Confirm installation date for DataStore R640s at JCMB with Dell
    • Configuration dependent on fibre channel work; estimated post August
    • Complete review for extended support of DataStore EoL kit
  • Complete installation planning of Eddie storage servers and storage at ACF
    • Finalise RFCs for Eddie storage servers and storage installation at ACF
    • Agree storage installation date with Lenovo
    • Review server and storage configuration requirements with OCF
  • Complete installation planning for Eddie/Eleanor compute node installation/configuration
    • Raise RFC with for installation/configuration
    • Installation/configuration to be done by OCF
  • Complete installation planning for TSM storage and server installation/configuration at ACF
    • Finalise RFC for installation/configuration
    • Confirm improved server shipping date (currently 07/09/2022) with Dell

Milestones

* = All work after this date to be reviewed/revised

Date Milestone Phase Complete
26/11/2021 Project brief and project plan drafted Plan Yes
10/12/2021 Project brief and plan reviewed and approved Plan Yes
10/12/2021 End of planning Plan Yes
17/12/2021 Requirements and priorities completed Analyse Yes
24/12/2021 Time critical orders raised Procure Yes
28/03/2022 Eddie/Eleanor nodes delivered to OCF for configuration/burn-in Build Yes
31/03/2022 JCMB/AT Data Centre planning complete Design Yes
03/06/2022 ACF Data Centre planning complete Design No
17/06/2022* Eddie storage servers and storage installed at ACF Build No
24/06/2022 DataStore upgrades configured and commissioned at JCMB/AT Integrate No
08/07/2022 DataStore DR upgrades installed at AT Build No
05/08/2022 Eleanor/Eddie server upgrades installed and commissioned at ACF Integrate No
05/08/2022 TSM servers upgraded at ACF Integrate No
31/08/2022 Other DataStore upgrades completed

Deliver

No
02/09/2022 All ECDF service documents updated Deliver No
30/09/2022 Project closed Close No

 

 

 

Project Status
RAG Status
Time
AMBER
Cost
AMBER
Scope
GREEN
Overall
GREEN
RAG Commentary
Unconfirmed and revised delivery dates have introduced budget risk and significant re-planning effort; RSS resourcing on project is at risk from higher priority work and delivery delays
Change Status
Time
Moderate
Cost
Moderate
Scope
No Change
Overall
Moderate
Has formal escalation taken place?
Yes
Escalated to
Tony Weir
Change Commentary
Re=planning required when Dell delivery dates are known
Activity
Approved budget
80.0 days
Activity this month
9.0 days
Activity this year
54.0 days
Activity to date
54.0 days
Estimate to complete current year
15.0 days
Estimate to complete future years
20.0 days

Project Info

Project
Infrastructure Refresh 21/22
Code
RSS402
Programme
ITI - Research Services (RSS)
Management Office
ISG PMO
Project Manager
Lawrence Stevenson
Project Sponsor
David Fergusson
Current Stage
Build
Status
In Progress
Project Classification
Run
Start Date
21-Oct-2021
Planning Date
10-Dec-2021
Delivery Date
16-Nov-2022
Close Date
14-Dec-2022
Programme Priority
1
Overall Priority
Highest
Category
Discretionary

Documentation