Report for April 2022

Report Date
April 2022

Achievements in Last Period

  • Eddie/Eleanor nodes (14 x C6400 chassis x 4 x C6420) have been delivered to OCF and are in stress test/burn-in (PO IS22125/IS22122)

    • Burn-in completion estimated 09/05/2022
  • DataStore upgrades (2 x 2 x ME4084, 15 x R640) are on site at JCMB (PO IS2121/IS22117)
    • Installation of storage completed at JCMB/AT on 14-15/04/2022
  • TSM storage arrays were installed at ACF on 13/04/2022 (PO IS22115)
    • Date for TSM array configuration needed
  • Eddie Dell storage servers (4 x R640) to be shipped direct to ACF on 05/05/2022 for installation on 06/05/2022 (PO IS22123)
    • RSS team onsite on 05/05/2022 to receive delivery and install on 06/06/2022
    • RFCs raised for servers and storage
  • Eddie 7 x Lenovo DE6000 storage; shipment (one pallet) delivered to ACF 25/04/2022 (PO IS22123)
    • Partial delivery, follow up with supplier
  • Orders placed on 22/04/2022 and 26/04/2022 for current quotes
    • PO IS22500 raised for quote 1003673743 for 2 x ME4024
    • PO IS22487 raised for quote 1003584021 for 3 x R650 server
    • PO IS22501 raised for quote 1003671671 for 18 X N1524 switches
    • PO IS22493 raised for quote 1003673974 for 4 x 10/25GbE Mellanox network cards
    • PO IS22488 raised for quote 1003673974 for 2 x 100GbE Mellanox network cards
    • PO IS22491 raised for quote 1003675841 for 1 x R640 server (Globus/Eddie)

Risks/Issues

  • Lack of information on order status from Esteem has meant that additional RSS time has been spent clarifying status (-)

    • Status of all original orders now known and being tracked (+)
    • Orders placed for all remaining quotes (+)
    • Weekly review meetings now in progress with Esteem/Dell/TechData (+)
  • Replacement needed for servers and storage approaching EoL (plus need for additional processing capacity) (-/+)
    • DataStore kit currently EoL with more in July/August (-)
  • RSS resource capacity is at risk as project (P1) will run in parallel with RSS401 GPU Refresh (P2) and RSS405 Tape Library (P1) (-)
    • Aggregate planning has identified dates that avoid other critical work (+)
  • Current allocated budget is £2.258M and spend must be within tolerance (+/-)
    • £7K projected overspend approved (+)
    • Dell price increases at end-May may impact planned spend (-)
  • Equipment lead times may delay project completion past July 2022 (-)
    • Bulk of orders will be received by July 2022 (+)
  • Installation and/or configuration must be carefully coordinated with ACF (RFCs) and ISG (GoCAB) (+/-)
    • ISG GoCAB requests and ACF RFCs being raised as required (+)
  • Upgrade work will impact DataStore, Eddie and Eleanor services (-)
    • DataStore server configuration will mean momentary outages (-)
    • Reduced Eddie/Eleanor service when compute nodes are installed and configured (-)
  • Planned kit may exceed data centres’ power budget (-)
    • Reviewing server/storage ratings against power thresholds (-)
    • No issue at JCMB/AT; estimated as a reduction in ACF (+)
  • No space in racks at ACF for additional kit (-)
    • Swaps out/in ACF will mean reduced service (-)
    • Power densities and racking impact considered low risk (AT)
  • OCF, StackHPC and IBM resources may not be available for installation and configuration dates (-)
    • Dates for Eddie/Eleanor nodes installation/configuration moved to week 01/08/2022 at request of OCF (-)
  • Storage areas at ACF and JCMB need to be agreed and storage time minimised (-)
    • ACF storage currently holding decommissioned tape library kit; due for to be removed 04/05/2022 (-)
  • COVID restrictions at ACF and JCMB/AT may restrict access and so impact dates (-)
    • Restrictions have been remove at ACF/JCMB/AT
  • UCU industrial action may impact project progress (-)
    • Potential further action at end-May (-)

Next Steps

  • Complete installation of DataStore servers at JCMB

    • Confirm racking date for DataStore R640s at JCMB with Esteem/Dell
    • Planned configuration date for servers 24/06/2022
  • Configuration date needed for TSM storage arrays at ACF; not in current plan
  • Delivery of Eddie Dell storage servers to ACF scheduled for 05/05/2022
    • Dependent on removal of old tape library kit by CCL North on 04/05/2022
    • Installation of Eddie Dell storage servers planned for 06/05/2022 by RSS team
  • Query status of rest of Eddie Lenovo storage delivery; review planned installation date of 06/05/2022; at risk
    • Arrange IBM support for Lenovo storage installation
  • Place any final orders by 13/05/2022
    • Review current order costs versus budget
  • Continue installation planning at ACF including onsite storage requirements
    • Confirm status of RFCs for Eddie Dell storage servers/storage
    • Raise RFC for Eddie/Eleanor compute nodes
    • Check with ACF at monthly meeting on earlier delivery date (July) for compute nodes
  • Review needed for extended support for DataStore kit currently EOL
  • Complete overall RSS resourcing planning in line with the aggregate plan
    • Confirm RSS402 timing and resources

Milestones

Date Milestone Phase Complete
26/11/2021 Project brief and project plan drafted Plan Yes
10/12/2021 Project brief and plan reviewed and approved Plan Yes
10/12/2021 End of planning Plan Yes
17/12/2021 Requirements and priorities completed Analyse Yes
24/12/2021 Time critical orders raised Procure Yes
28/03/2022 Eddie/Eleanor nodes delivered to OCF for configuration/burn-in Build Yes
31/03/2022 JCMB/AT Data Centre planning complete Design Yes
11/04/2022 DataStore servers/storage installed at JCMB/AT Build Yes
06/05/2022 Eddie Lenovo/Dell storage installed at ACF Build No
03/06/2022 ACF Data Centre planning complete Design No
24/06/2022 DataStore upgrades configured and commissioned at JCMB/AT Integrate No
08/07/2022 DataStore DR upgrades installed at AT Build No
05/08/2022 Eleanor/Eddie server upgrades installed and commissioned at ACF Integrate No
05/08/2022 TSM servers upgraded at ACF Integrate No
31/08/2022 Other DataStore upgrades completed

Deliver

No
02/09/2022 All ECDF service documents updated Deliver No
30/09/2022 Project closed Close No

 

 

 

Project Status
RAG Status
Time
GREEN
Cost
GREEN
Scope
GREEN
Overall
GREEN
Change Status
Time
Within Tolerance
Cost
No Change
Scope
No Change
Overall
Within Tolerance
Has formal escalation taken place?
No
Change Commentary
Eddie/Eleanor node upgrades moved to August due to higher priority RSS work and non-availability of OCF engineers at end-July
Activity
Approved budget
80.0 days
Activity this month
10.0 days
Activity this year
45.0 days
Activity to date
45.0 days
Estimate to complete current year
15.0 days
Estimate to complete future years
20.0 days

Project Info

Project
Infrastructure Refresh 21/22
Code
RSS402
Programme
ITI - Research Services (RSS)
Management Office
ISG PMO
Project Manager
Lawrence Stevenson
Project Sponsor
David Fergusson
Current Stage
Build
Status
In Progress
Project Classification
Run
Start Date
21-Oct-2021
Planning Date
10-Dec-2021
Delivery Date
16-Nov-2022
Close Date
14-Dec-2022
Programme Priority
1
Overall Priority
Highest
Category
Discretionary

Documentation