Project Brief

RSS212 - DataVault Development Phase IV

Approvals

[1] Please note Scott Renton will take over as Product Owner and Development Lead in January 2020

Distribution

Background

The University of Edinburgh’s DataVault, part of the Research Data Service is a facility supported by the Research Data Support (RDS) team and delivered by the Digital Library, both part of the Library and University Collections (L&UC) division, which assists researchers to comply with funding regulations by providing cost-effective, secure, long-term archiving of their research data.

In 2015-16, the JISC funded DataVault project run by the University’s RDS team, in collaboration with the University of Manchester, developed generic software designed to collect basic research metadata and deposit research datasets into an archive service. In parallel, the University’s ITI Research Services Section (RSS) implemented an initial systems architecture to support these archive services.

Due to demand and the time to develop a UoE specific DataVault service, an interim service was developed and launched in 2016. The interim service used the RSS Storage Manager facility to manage research data deposits on the disk based DataStore system.

In 2017, a project to deliver a fully supported cost-effective DataVault service began and the service was launched in January 2019. This main objectives of this DataVault project (RSS022) were to deliver the following -

• expand the interim service to include –

  • an improved (self-service) user interface
  • encryption of data
  • resilient storage on internal and external archive services
  • usage reporting
• design a robust system architecture for the service
• define the processes and policies of the full service
• achieve GDPR compliance for the service
• launch DataVault as a fully supported RDS service

In 2019, a third project, DataVault III (RSS044), funded from the FY2018-19 DRS budget, ran from February to November 2019 and delivered additional features to the DataVault service including –

• organisational structures and roles for access management
• automated billing and usage reporting
• storage of deposits up to 10TB
• reduced processing times and improved resilience
• the migration of deposits from the interim DataVault service
• improved release management for DataVault deployments to test and production

The latest project, RSS212 – DataVault Development Phase IV, will run from November 2019 to July 2020 and is chartered to deliver the following features –

• retention date management
• processing data from other file systems in addition to DataStore
• automated scheduling and processing of deposits
• additional resilience and recovery measures
• user interface improvements
• improved synchronisation of metadata between DataVault and Pure
• processing increased deposit sizes (above 10TB)
• evaluation of improved encryption key management
• improved processing times

Project Scope

DataVault has been developed to provide a low-cost and reliable archiving service for the University’s research community in compliance with research data management and funding policies. The scope of this project will concentrate on improving the features, resilience and performance of the service for the University’s research community.

Significant features were added to the DataVault service and delivered during the previous project (RSS044) including roles management, processing of datasets up to 10TB, storage auditing, automated billing and usage reporting. Work in this project will include improvements to retention date management, resilience and recovery, encryption key management, processing times, and metadata collection by DataVault synchronised with Pure.

In addition, options were presented to increase archive sizing above the 10TB limit and potentially processing deposits of an unlimited size. The project will, at least, develop proofs of concept for some of the options suggested and, depending on the development work required, could implement the processing of increased deposit sizes based on one of these options.

DataVault development has concentrated so far on the University’s research datasets stored in DataStore. The project plans to deliver the processing of data from other sources e.g. local drives, network drives, OneDrive. In addition, the scope will include the evaluation of automated, scheduled processing of data from research equipment.

Encryption key management is essential to providing a secure service for DataVault users. It is recognised that improvements in this area need to be reviewed to provide enhanced security management to counter the increased threat levels to a service that manages, in part, sensitive research data.

Out of Scope

Expansion of the service to support the long-term needs of other groups within the University or elsewhere, outside the research community, is not within scope for this project. Future projects could be initiated to expand the service to provide, inter alia, -

• archiving of University business data
• a DataVault service to other universities
• a DataVault service to non-academic organisations

Objectives

The key objectives of this project are to deliver additional functionality and improved processing goals for the DataVault service. These include –

• To provide an effective retention data management service which automatically tells users when deposits need to be reviewed for retention or deletion
• To increase maximum deposit sizes above the current 10TB limit (and potentially unlimited) to meet the demand to archive larger data sets e.g. imaging data
• To deliver a more resilient and recoverable service so that fewer failed deposits occur and, if they do, archiving, validation and retrieval can automatically restart from point of failure
• To reduce deposit and retrieval processing times to provide a basis for improved customer service levels while ensuring acceptable impact on other systems and network services
• To provide processing for other data sources, in additional to data store
• To implement automated processing of deposits from continuously created data sources
• To continue to develop the website user interface to ensure simple and effective access to DataVault services and features including the elimination or, at least, simplification of the authentication ‘cut-and-paste’ process currently used to access a user’s DataStore
• To improve the user experience when creating a vault, providing better synchronisation of research metadata between DataVault and Pure
• To research improved encryption key management methods to ensure more secure access to archived data and service recovery
• To evaluate how DataVault could be used for the low-cost archiving by other groups in the University (e.g. Finance), and even external organisations, that required regulatory data retention
• To continue to promote and provide workshops and training for researchers and support staff on the services offered by DataVault
• To implement more responsive release management processes, potentially using the new shared Puppet service, to ensure updates and fixes are installed effectively minimising outages to the production services

Deliverables

More details on specific deliverables to meet the objectives of the project are given in the table below.

Success Criteria

The following are the criteria to be met to ensure a successful completion of this project –

• An increase in the usage of the DataVault service through the promotion of the service, publicising the additional features and processing improvements to be delivered by this project
• The effective deployment of retention review reminders via email and other methods
• The elimination of deposit failures from unplanned outages and loss of TSM and Oracle subsystems
• A reduction in system outages and downtime through improved resilience and data driven parameterisation of the service
• A significant reduction in the processing time (e.g. 25%) for the archiving, validation and retrieval of large datasets
• Report published on the potential use of the DataVault service, internally and externally, in archiving other data requiring regulatory retention

Benefits

The successful delivery of this project is expected to provide the following benefits –

• Improved website user interface and features for users of DataVault which should attract new users and retain existing ones
• Better control of retention dates so that deposits can be reviewed and removed in a timely manner, reducing costs to data owners
• Enable researchers to archive deposits in excess of 10TB thus opening DataVault to new users with larger datasets
• Recovery, from point of failure, of deposits during processing, if impacted by planned or unplanned outages, again contributing to increased customer satisfaction
• Deposit and retrieval processing times improved so reducing wait times and risk of unplanned outages and improving customer satisfaction
• Identification of improved data security measures through more robust encryption key management processes making DataVault more attractive to users with sensitive data to archive
• Increased promotion of the service and training channels to publicise the service and expand the user base

Project Governance

Overall project governance will be provided by the project board whose members are listed in the next section. It is intended to hold project board meetings on a monthly basis.

After initial planning and design, a backlog will be generated in Jira for the project plan and all internal development will be run as sprints. The project plan will also be kept up to date for progress reporting purposes to the project board and other stakeholders.

Day to day project management and delivery is the responsibility of the project team. Weekly project (sprint review) meetings will be held along with sprint planning and retrospective sessions at the start and end of each sprint. Sprints will be set at either 2, 3 or 4 weeks depending on the level of demand held in the backlog.

DataVault Phase IV is an upgrade project so any detailed systems design activity will take place during the relevant development phase. On that basis, the project is divided into work packages to facilitate (a) assignment of work to Digital Library Systems or EDINA, (b) changes to scope and priorities and (c) the incremental design required within each work package.

Where work impacts other teams, e.g. ENT or RSS, the internal development team will engage with the systems experts in those teams to ensure the integration of system design and implementation into the parts of their infrastructure which supports DataVault.

Project Board

The project sponsor is Tony Weir, ITI Director. Project board meetings will be scheduled monthly to review progress, major escalations and resolution. The following senior stakeholders will be invited to participate in project board meetings.

Project Team

The diagram below shows the key members of the DataVault IV project team and their roles in the project.

Resources, Skills and Costs

Priority & Funding

DataVault Phase IV is a project in the University’s ITI Research Services Section (RSS) programme for FY2019-20 funded by the Digital Research Services (DRS) Steering Group.

Budget

The table below shows funding that has been allocated to the project for this financial year - FY2019-20.

Project Costs

In terms of current projected costs, the following expenditure has been estimated for the project as follows –

A capacity planning exercise, based on current and estimated growth data, will take place during the design phase of the project, to ascertain when additional compute power and/or storage will be required during the following 12-month period. This is not included in these project costs but (a) is not currently predicted and (b) would be covered by the ITI operational budget.

Resource Plan

The following table defines the role and skills of the project stakeholders and an estimate of the days needed to be allocated to work on the project. This represents a total of 318 days effort for the core team members and 370 days overall when other stakeholders are considered.

Core Team

Support Teams

Project Board

Project Planning

Outline Plan

The table below show the initial estimated duration for the main tasks in the project. Tasks to be allocated to Digital Library Systems for development are marked (D), with work to be allocated to a contract developer in the Digital Library Systems team marked (C). Tasks to be allocated to EDINA are marked (E). During the planning and design phase, modules will be reviewed to ascertain if work can be overlapped or reduced in duration to see if additional releases can be delivered before the end of FY2019-20.

Project Milestones

The currently estimated major milestones, with delivery dates, are outlined in the following table to provide estimates when product releases and other deliverables will be made. Development work has been allocated to Digital Library Systems and EDINA, as appropriate.

Work Breakdown

The following work breakdown structure (WBS) shows the main components of work to be undertaken in the project.

Project Timeline

The diagram below is the high-level project timeline, showing key project phases and delivery dates.

Assumptions

The following assumptions have been identified at the start of the project and may be expanded during the project.

• Formal planning and overall design will be completed prior to any development work starting
• The plan reflects EDINA resourcing levels previously agreed (28 days, February – March 2020)
• The current plan assumes a contract resource will be in place in Digital Library Systems during March – May 2020 but if this resource is not be available the plan will need to be revised
• DRS staff will be available for testing during integration test and UAT periods
• ENT and RSS technical staff will be available, as required, to support VMware/SAN and TSM changes and upgrades
• Senior stakeholders will be available for project boards, escalation and resolution of issues, etc. as required

Constraints  

The main constraint for the project is that team members are not fully assigned to the project and may be working on other projects and supporting BAU activities. This can be translated into a key risk where precedence for resources may go the BAU work or work on a higher priority project. Management commitment needs to be made to the days allocated in the estimates in the resource plan and to the successfully delivery of the project on schedule.

The current plan has been restricted in scope so that the project is completed by the end of FY2019-20. However, any significant delay to any phase in the project or lack of planned resourcing being available will necessitate a revision of the plan that could push work into FY2020-21.

Development resources on the project are coming from Digital Library Systems, potentially supplemented by contract resource, and EDINA. Work packages have been allocated to ensure work can be undertaken by both groups in parallel to reduce the project timeline. Carefully planning for source code and release management will be maintained to ensure the smooth running of parallel development, testing and release.

Risks

The initial project risks identified are as follows –

• Potentials users may not use the DataVault service because of quality issues and constraints e.g. user interface is too complex, processing takes too long, Pure metadata prerequisites
• Funding may not be available to cover the procurement of an encryption key management  product such as Thales SafeNet
• EDINA development resource is provided in monthly allocations that means some work may have to go on hold if that allocation is used up in any month
• Licence renewals for TSM may be higher than estimated
• Subscription renewals for Oracle may be higher than estimated
• Additional licencing (e.g. TSM) or hosting costs (Oracle Cloud Archive) may be incurred as the service is expanded
• Migration to new shared Puppet environment for release management, if approved, may delay planned work

Issues

If they arise, issues will be recorded, on the Projects website, and tracked throughout the project. Issues that are inside agreed tolerances will be resolved within the project team. Major issues will be escalated to the project board for review and advice on resolution. Issue resolution that requires a major change to the project in terms of time, quality or cost needs to be approved under formal change management by the project sponsor.

Project work that impacts production systems or networks must be communicated with the requisite lead time to the ISG CAB for review and approval.

Tolerances

The following standard ITI project tolerances will be applied and these are outline below. Any deviation outside these tolerances must be reported for review and resolution to the Project Board, if constituted, or the Project Sponsor and Portfolio Manager otherwise.

• Cost – over +/- 10% of original budget
• Plan – slippage of over 4 weeks to current plan
• Scope – extensive addition or reduction from current scope
• Quality – refusal to approve a project work package

Dependencies

The key external dependencies for the project lie with the issues experienced with the TSM service and the continuous changes to the Pure system. The Oracle archive service has been affected by the need to change the administration password regularly, but this issue has been mitigated, to some extent, by the retry features added to the service during the previous project and a schedule created to notify password expiry dates well in advance.

The following dependencies have been identified at the start of this project -

• Timely communication of Pure service and API updates to ensure development, testing and production environments are not adversely impacted
• Provision of a stable TSM service at JCMB to ensure test and production processing is not adversely affected
• Support needed from RSS, ENT and CIS to review processing times and identify bottlenecks during performance monitoring and upgrades

Lessons Learned

A log will be maintained on the Projects website to record lessons learned throughout the project. The key observations from the previous DataVault project (RSS044 DataVault III) are summarised in the table below.

Communication

The following meetings and communication channels will be put in place to support the effective communication of project progress, events, product releases, and risk and issue management.

Alignment with University Strategic Vision

This section defines how the areas of the upgraded service aligns with the University’s strategic vision and contribute to run, grow or transform services. DataVault Phase IV will extend the facilities of the existing service.

Alignment with ISG Strategic Vision

This section defines how the areas of the upgraded service aligns with the ISG’s strategic vision.

Alignment with IS Change Programme

The following table shows how the DataVault project contributes to the themes that underpin the University’s change programme.

Attachment	Size
wbs.gif	40.77 KB
plan.gif	42 KB
team.gif	71.54 KB