RSS044 - DataVault III
|Tony Weir||Project Sponsor||Head, ISG IT Infrastructure||12/04/2019|
|Robin Rice||Service Owner||Data Librarian & Head, Research Data Support, L&UC||28/03/2019|
|Fraser Muir||Senior User||Chief Information Officer, CAHSS||29/03/2019|
|Anthony Davie||Senior User||IS Campus Leader, MVM||29/03/2019|
|Colin Higgs||Senior User||Computing Officer, School of Engineering||29/03/2019|
|Kirsty Lingstadt||Senior Supplier||Head, Digital Library & Deputy Director, L&UC||29/03/2019|
|Senior Supplier||Head, ITI Research Services Section||29/03/2019|
|Ianthe Sutherland||Product Owner||Development & Systems Manager, Digital Library||08/03/2019|
|Maurice Franceschi||Programme Manager (RSS)||ITI Portfolio Manager, ISG||
|John Baker||Senior Supplier||Director, Digirati||15/04/2019|
|David Graham||Senior Supplier||Head, ITI Communication Infrastructure Services||15/04/2019|
|Graeme Wood||Senior Supplier||Head, ITI Enterprise Services||15/04/2019|
|Claire Young||Senior Supplier||Operations Manager, EDINA||15/04/2019|
The University of Edinburgh’s DataVault is a component of Research Data Management services which assists researchers to comply with funding requirements for the long-term retention of their research data. In 2015-16, the JISC funded DataVault project run by the Library and University Collections (L&UC) team at the University of Edinburgh, in collaboration with the University of Manchester, developed generic software designed to collect basic research metadata and deposit research datasets into an archive service. In parallel, the University’s ITI Research Services Section (RSS) defined an initial systems architecture to provide robust archive services.
Due to demand and the time to develop a UoE specific DataVault service, an interim service was launched in 2016. The interim service used the RSS Storage Manager facility to manage research data deposits on the disk-based DataStore system.
A project to deliver a full, support DataVault service began in 2017 and the service was launched in January 2019. This main objectives of the DataVault project (RSS022) were to deliver the following -
expand the interim service to include –
- an improved (self-service) user interface
- encryption of data
- resilient storage on internal and external archive services
- usage reporting
- design a robust system architecture for the service
- define the processes and policies of the full service
- achieve GDPR compliance for the service
- launch DataVault as a fully supported Research Data service
The latest project, DataVault III (RSS044), funded for FY2018-19, will run from February to September 2019 to deliver additional features to the DataVault service including –
- an improved user experience (UX) and user interface (UI)
- organisational structures and roles for data management
- automated billing and usage reports
- the migration of deposits from the interim DataVault service
- storage of deposits up to 10TB
- improved encryption management
- reduced processing times and improved resilience
DataVault has been developed to provide a low-cost and reliable archiving service for the University’s research community in compliance with research data management and funding policies. The scope of the current project will concentrate on improving the service for the University’s research community.
Significant work on the design and development of the DataVault user interface was achieved in the previous project. Further work in this project will include a review and redesign of the user experience and interface based on feedback received during user testing and in the early adoption period of the current DataVault.
Initial DataVault requirements included the need for secure, data management within a research group or department, by incorporating organisational structures and roles, so that the University retains control of data access and retention if the original researcher is no longer a member of staff. This is a major feature to be delivered by this latest project.
Currently only deposits of 2TB can be stored and retrieved in DataVault. An increase to 10TB deposits is a key deliverable of this project and has been requested by several researchers interested in storing their data in the ‘vault’.
The DataVault has been designed to archive personal and sensitive data, within the scope of GDPR regulations, by (a) advising users to anonymise or pseudo-anonymise personal data and (b) encrypting all data prior to transmission and deposit, using SHA-256 protocols, to the archive subsystems. Further work will be undertaken in this project to improve encryption and key management processes.
Out of Scope
Expansion of the service to support the long-term needs of other groups within the University or elsewhere, outside the research community, is not within scope for this project. Future projects could be initiated to expand the service to provide, inter alia, -
- archiving of University business data
- providing a DataVault service to other institutions
Objectives & Deliverables
The following key objectives are planned for the next DataVault project, DV III, with starts in February 2019.
- Increase maximum deposit sizes to 10TB (currently 2TB)
- Improve release management
- Introduce processing recovery procedures for partially failed dep
- Implement organisational structures and roles for data management
- Improve the website user experience (UX)
- Provide storage auditing features
- Implement automated billing
- Improve retention date management features
- Migrate deposits from the interim DataVault service
- Improve existing vault webpages where feedback has been received e.g. remove encryption key cut-and-paste
- Replace the PURE flat file interface with the PURE API for accessing metadata
- Provide usage reports for Schools administrators
- Simplify retention policy updates
- Improve encryption key management
- Optimise deposit and retrieval processing times
- Improve resilience and recovery features
- Provide workshops for researchers and support staff in all Colleges to present the features and benefits of using DataVault
More details on specific deliverables from the project are given in the table below.
|Increase maximum deposit sizes to 10TB (currently 2TB)||A DataVault service handling data sets up to 10TB in size||Should|
|Improve release management||An automated and reliable deployment service for release to test, demonstration and live environments||Should|
|Introduce processing recovery procedures||A recovery subsystem that restarts processing from point of failure (or start of current stage at minimum)||Must|
|Implement organisational structures and roles for data management||
A user interface and back-end subsystem that supports a research organisation structure and the ability for a PI to share a data archive with colleagues and department supervisors to reassign ownership of deposits
|Review the website user experience (UX)||A report on the user journey plus the suggested redesign of the user interface to simplify navigation and support additional features||Should|
|Provide storage auditing features||Regular data integrity checks (a) database records against archive storage data and (b) regular, random deposit retrievals to check datasets||Must|
|Implement automated billing||An automated billing system using usage statistics to provide DataVault users with invoices for costs of storage to replace the current manual methods||Should|
|Improve retention date management and review features||A subsystem that manages retention and review dates logically and effectively||Must|
|Migrate deposits from the interim DataVault service||To be done manually by retrieving from the interim vault and depositing to the DataVault||Must|
|Improve vault webpage features and PURE interface||Improvements to vault webpage; an improved metadata connection using PURE API||Should|
|Provide usage reports for Schools administrators||A reporting subsystem proving usage reports automatically to senior research staff||Must|
|Simplify retention policy updates||An automated data-driven subsystem that provides facilities for DataVault administrators to easily update retention policy information||Must|
|Improve encryption key management||An industry standard encryption and key management system integrated into DataVault||Should|
|Optimise deposit and retrieval processing times||Improved processing which reduces deposit and retrieval times by a significant margin||Must|
The successful delivery of this project is expected to provide the following benefits –
- Improved website features for users depositing, reviewing and retrieving research data sets to attract new users and retain existing ones
- Recovery, from point of failure, of deposits during processing if impacted by planned or unplanned outages
- The ability to share and take ownership of data sets within a research group
- Clear usage and billing information for users and supervisors with costs retained at current low levels
- Deposits in the interim DataVault moved to the new service, enabling decommissioning of the interim service
- Increased compliance for research datasets to be archived, through improved management of research metadata
- Better control of retention dates so that deposits can be reviewed and removed in a timely manner, reducing costs to data owners
- Deposit and retrieval processing times reduced and so reducing risk of unplanned outages
- Improved data security through more robust encryption key management and other features
- Enable researchers to archive deposits up to 10TB thus increasing user community
The following are the criteria to be met to ensure a successful completion of this project –
- An increase in the usage of the DataVault service through the provision of additional features, improved resilience, reduced processing times, increased deposit sizes and regular data audits
- Reduction in the loss of deposits in progress from unplanned outages
- Research datasets being shared between researchers and colleagues which is a prerequisite for the transfer of data in the interim vault to new vault
- Ownership of datasets taken over by senior or other colleagues when a principal investigator leaves University employment
- Increased awareness of research archive costs through the provision of clear and regular billing information
- Increased use of PURE to store research metadata, as a prerequisite for DataVault use
The currently estimated major milestones, with delivery dates, are outlined in the following table to provide estimates when product releases and other deliverables will be made. Development work has been allocated to EDINA and Digirati, as appropriate.
|Initial review of scope and requirements||Pauline Ward||Robin Rice||15/02/2019|
|Project brief and initial project plan approved||Lawrence Stevenson||Tony Weir Robin Rice||29/03/2019|
|System documentation complete||Robin Taylor||Tony Weir Senior Users||29/03/2019|
|Processing recovery procedures operational||
Robin Taylor David Speed
|Website user experience (UX) report, redesign and wireframes available||Digirati||Robin Rice||10/05/2019|
|Deposit and retrieval processing times improved||Robin Taylor David Speed||Ianthe Sutherland||24/05/2019|
|Organisational structures and roles for data management implemented||Digirati||Robin Rice||07/06/2019|
|Storage auditing reporting in place||Robin Taylor David Speed||Ianthe Sutherland||07/06/2019|
|Usage reports for Schools administrators available||Digirati||Robin Rice||21/06/2019|
Deposits from the interim DataVault service migrated
|Pauline Ward||Robin Rice||05/07/2019|
|Improved release management in place||Robin Taylor William Petit||Ianthe Sutherland||05/07/2019|
|Improved retention date management in place||Digirati||Robin Rice||19/07/2019|
|Improved vault webpage features and PURE connect||Robin Taylor David Speed||Ianthe Sutherland||02/08/2019|
|Automated billing launched||Digirati||Robin Rice||16/08/2019|
|Improved encryption key management integration||Robin Taylor William Petit||Ianthe Sutherland||30/08/2019|
|Maximum deposit sizes increased to 10TB||Robin Taylor||Ianthe Sutherland||27/09/2019|
|Project closed||Lawrence Stevenson||Tony Weir Senior Users||18/10/2019|
Alignment with University Strategic Vision
This section defines how the areas of the upgraded service aligns with the University’s strategic vision and contribute to run, grow or transform services. DataVault III will extend the facilities of the existing service.
|A unique Edinburgh offer for all our students|
|Strong and vibrant communities within and beyond the University – making the most of our unique offer of world-leading thinking and learning within one of the world’s most attractive cities||N/A|
|A larger, more international staff who feel valued and supported in a University that is a great and collegial place to work, develop and progress||DataVault supplements existing services to researchers|
|More postgraduate students – underpinned by the best support in the sector to ensure we attract the brightest and best regardless of ability to pay||Not currently supported|
|A strong culture of philanthropic support focussed especially on our students and on outstanding research capabilities.||N/A|
|Many more students benefiting from the Edinburgh experience (largely or entirely) in their own country – supported by deep international partnerships and world leading online distance learning||N/A|
|Sustained world leading reputation for the breadth, depth and inter-disciplinary of our research supported by strong growth in research funding and strong international partnerships – drawing from well-established and less well-developed sources||DataVault provides a unique service, not currently available to Universities in the UK|
|An estate that matches expectations, responds flexibly to changing student and staff needs, and showcases the University||The DataVault service meet the needs for data retention required by funding bodies as well as meeting GDPR regulations|
|A deeper and earlier collaboration with industry, the public sector and the third sector – in terms of research; knowledge exchange; and in giving our students the best possible set of skills for their future||DataVault has the potential to be extended to provide low-cost, secure storage for legacy data that require retention by other organisations in the University e.g. Finance and to external organisations seeking similar services|
|Digital Transformation||The DataVault will facilitate the preservation and creation of metadata for research outputs, enhancing the prospects for re-use of that data and reducing risk of loss of data which might previously not have been backed up. It will also allow researchers to free up space on their active data storage for use in further work.|
Alignment with ISG Strategic Vision
This section defines how the areas of the upgraded service aligns with the ISG’s strategic vision.
||DataVault development has been driven by the Library and University Collections team who recognised a gap in service provisioning in an area that could be exploited in the UK and overseas|
|Research and Innovation|
||The service provides the University’s research community with ability to store completed research data cheaply, meet regulatory requirements but be accessible as and when required|
||A market review has shown that DataVault provides a type of service that is currently not available but would benefit University researchers across the UK|
||The University, along with the University of Manchester, are taking the lead in delivering a JISC funded version of DataVault for use in UK Universities|
||DataVault provides researchers with a secure central facility to improve their research process management for storing, retrieving and sharing past research data|
||DataVault is one of the strands of development to meet the goals of the Research Data Management Roadmap, which also includes complementary initiatives such as DataShare and Data Safe Heaven|
||DataVault incorporates secure user account management by integrating with the University’s EASE and Shibboleth authentication services. The service also incorporates client-side encryption and decryption to ensure the secure transmission and storage of data|
The impact of the continuing development of the DataVault service is outlined in this section where priority, budget and contribution to IS change are presented.
Priority & Funding
DataVault III is a project in the University’s ITI Research Services Section (RSS) programme for FY2018-19 funded by the Digital Research Services (DRS) Steering Group.
The table below shows funding that has been allocated to the project for this financial year plus the forecast for continuing the project into FY2019-20, but excludes FY2018-19 costs for previous DataVault project (RSS022)
|Project Financial Summary||FY2018-19||FY2019-20||Total|
IS Change Programme
The following table shows how the DataVault project contributes to the themes that underpin the University’s change programme.
|IS Change Programme Theme||Contribution|
|Project Management||The ITI project management process is being used for this project.|
|Working Together||The project team includes representatives for each of the University’s schools, members from digital library and research data services, internal (EDINA) and external (Digirati) development staff and infrastructure teams from research and enterprise services. The team was formed to ensure all the required skills were available to ensure a successful project. In particular, a wide-ranging requirements exercise was undertaken to ensure the right work and priorities were encapsulated in the project.|
|Standards and Technical Leadership||DataVault is being developed, in collaboration with JISC, to provide a research data archiving and retrieval service that is will close a gap in service provisioning for UK Universities’ research communities.|
|Staff Learning and Development||Some members of the team have developed their documentation and presentation skills to deliver various customer-facing activities such as presentations and workshops in the previous DataVault project. The new project gives them an opportunity to further develop these skills. The project team have benefited and will continue to benefit from planning and delivering a service using formal project management practices.|
|Service Based Culture||Consultation with users and other stakeholders has been undertaken at every stage of requirements and design process. A web-based user interface has been developed to make the usability of the service as easy as possible. Support and assistance are provided by the Library Research Support services to users prior to, during and after they have used the service. Extensive web help pages, including a range of ‘how to’ videos have been developed and will continue to be improved in this project.|
|Equality and Diversity||The DataVault website is be developed to incorporate accessibility features to ensure access is available to all that require to use the service. The University’s equality policies are followed by all team members in all communications, meetings and other channels used in the project.|
|Flexible Resourcing||The project team is supplemented by both development staff from EDINA and an external design and development team from Digirati, a Glasgow-based software development company. Both these teams expand the resources and skills of the Digital Library development team.|
|Communication and Branding||
A communication plan section is included in this project brief. The existing DataVault service and new releases to be delivered by this project are/will be communicated to the University’s research communities by digital and print media, by presentations and workshops and by the cascade of announcements by schools’ representatives on the project board.
The design and development of the DataVault website and supporting materials follows, and will continue to follow, the University’s branding guidelines and web design (EdGEL) framework.
The following diagram shows the main components of work to be undertaken in the project
Resources & Skills
The following table defines the team members, role and skills and an estimate of the days allocated for work to the project. Days allocated for Digirati work still to be reviewed and agreed.
|Team Member||Role||Days Allocated|
|Tony Weir||Project Sponsor||4.0|
|Robin Rice||Service Owner||18.0|
|Ianthe Sutherland||Product Owner/Development Lead||25.0|
|Pauline Ward||Business Analyst/Senior Tester||44.0|
|Anthony Davie||Senior User (MVM)||4.0|
|Colin Higgs||Senior User (CSE)||4.0|
|Fraser Muir||Senior User (CAHSS)||4.0|
|Martin Donnelly||Operations Manager (L&UC)||4.0|
|Jennifer Daub||System Tester||6.0|
|Kerry Miller||Schools Liaison/System Tester||
|Kirsty Lingstadt||Senior Supplier (Digital Library)||
|Robin Taylor||Lead Developer||44.0|
|Claire Young||Senior Supplier (EDINA)||4.0|
|David Speed||Developer (EDINA)||48.0 (25.0)|
|William Petit||Developer (EDINA)||40.0 (35.0)|
|John Baker||Senior Supplier (Digirati)||4.0|
|Sarah O’Sullivan||Workstream Manager (Digirati)||
|Daniel Grant||System Architect (Digirati)||
|Diego Lago||UX/UI Designer (Digirati)||20.0|
Senior Supplier (RSS)
|Kevin Tomlinson||System Administrator (RSS)||
|TBA||System Advisor (RSS)||4.0|
|Graeme Wood||Senior Supplier (ENT)||2.0|
|TBA||System Administrator (ENT)||4.0|
|David Graham||Senior Supplier (CIS)||
|TBA||Network Administrator (CIS)||4.0|
|Lawrence Stevenson||Project Manager||24.0|
|Maurice Franceschi||Programme Manager||8.0|
In terms of current projected costs, the following expenditure has been estimated for the project as follows –
|Thales SafeNet Licences||£12,000|
|Oracle Cloud Archive Service (2-years)||£14,000|
|Internal||ENT SAN Increase to 10TB (per annum)||£16,740|
A capacity planning exercise, based on current and estimated growth data, will take place during the design phase of the project, to ascertain if and when additional compute power and/or storage will be required during the following 3-year period, in addition to the planned 10TB storage upgrade already planned.
Overall project governance will be provided by the project board whose members are listed in the next section. Day to day project management and delivery is the responsibility of the project team.
After initial planning and design, a backlog will be generated in Jira for the project plan and internal development will be run in sprints managed. The project plan will also be kept up to date for progress reporting purposes to the project board and other stakeholders.
Work packages to be developed by Digirati will be assigned to the nominated workstream manager to be managed internally. The workstream manager will provide regular project reports on a work package and raise any escalations that cannot be resolved internally.
Each work package, when delivered, must have clear documentation and, where appropriate, installation, configuration and test instructions. In some cases, Digirati staff will need to work with internal development staff to integrate and test work packages in the DataVault test environment.
Digirati will adhere to the current Data Vault development standards to ensure a product that can be maintained into the future. EDINA development staff will provide Digirati with the appropriate source management and deployment standards to ensure seamless integration of all code work packages that they deliver. University user and VPN accounts will be provided to Digirati staff where work package delivery makes this necessary.
The current system design is included in the systems documentation. DataVault III is an upgrade project so any systems design effort will be around the planned incremental work. On that basis, the project is divided into work packages to facilitate (a) assignment of work to EDINA or Digirati, (b) changes to scope and priorities and (c) the incremental design required within each work package.
The initial Digirati work package will review the current systems architecture against planned DV III work to provide design recommendations on the improvement and expansion to the system.
Where work impacts other teams, e.g. ENT or RSS, the internal development team will engage with the systems experts in those teams to ensure the integration of system design and implementation into the parts of their infrastructure which supports DataVault.
The project sponsor is Tony Weir, ITI Director. Project board meetings will be scheduled monthly to review progress, major escalations and resolution. The following senior stakeholders will be invited to participate in project board meetings.
|Project Sponsor||Tony Weir||Head, ISG IT Infrastructure|
|Service Owner||Robin Rice||Data Librarian & Head, Research Data Support, L&UC|
|Senior User||Fraser Muir||Chief Information Officer, CAHSS|
|Senior User||Anthony Davie||IS Campus Leader, MVM|
|Senior User||Colin Higgs||Computing Officer, School of Engineering|
|Senior Supplier||Kirsty Lingstadt||Head, Digital Library & Deputy Director, L&UC|
|Head, ITI Research Services Section|
|Product Owner||Ianthe Sutherland||Development & Systems Manager, Digital Library|
ITI Project Manager, ISG
|Programme Manager (RSS)||Maurice Franceschi||ITI Portfolio Manager, ISG|
The following diagram shows the key members of the DataVault III project team.
Outline Project Plan
The table below show the initial estimated duration for the main tasks in the project. Tasks to be allocated to EDINA for development are marked (E). Tasks to be allocated to Digirati are marked (D). Tasks in red indicate dates past the end of FY2018-19. During the planning and design phase, modules will be reviewed to ascertain if work can be overlapped or reduced in duration to see if additional releases can be delivered before the end of FY2018-19.
|Define initial scope, requirements and priorities||25||
|Project planning and design||30||
|Review system architecture and documents (D)||5||01/04/2019||05/04/2019|
|Implement resilience and recovery procedures (E)||20||
|Prepare UX proposal, design, wireframes (D)||15||
|Improve deposit and retrieval processing times (E)||20||
|Develop/integrate organisational structures and roles (D)||20||
|Implement storage auditing reporting (E)||10||27/05/2019||07/06/2019|
|Build/integrate usage reports for administrators (D)||10||
|Migrate deposits from the interim DataVault service||20||10/06/2019||05/07/2019|
|Improve release management processes (E)||20||10/06/2019||05/07/2019|
|Build/integrate improved retention date management (D)||20||10/06/2019||19/07/2019|
|Improve vault webpage and PURE interface (E)||20||08/07/2019||02/08/2019|
|Build/integrate automated billing (D)||20||
|Implement improved encryption key management (E)||20||05/08/2019||30/08/2019|
|Increase maximum deposit size to 10TB (E)||20||
The following assumptions have been identified at the start of the project and may be expanded during the project.
- Formal planning and design will be completed prior to any development work starting
- EDINA resources will be available at the required level for the duration of the project
- Digirati remote working will still underpin the momentum of the project
- Research testers will be available during test periods
- Senior project team members will be available for project boards, escalation and resolution of issues, etc. as required
The main constraint for the project is that some team members are not fully assigned to the project and are working on other projects and supporting BAU activities. This can be translated into a key risk were precedence for resources may go the BAU work or work on a higher priority project. Management commitment needs to be made to days allocated in the resource plan and to the overall commitment needed.
Additionally, it may be management decision to complete the project by the end of FY2018-19. If it is agreed to close the project in July 2019, the current outline plan shows that two work packages, Improved Encryption Key Management and Deposits up to 10TB, may not be delivered unless additional funding is obtained for the work.
Development resources on the project are coming from EDINA and Digirati, an external software development company. Work packages have been allocated to ensure work can be undertaken by both group in parallel to reduce the project timeline. However, some issues may arise in such areas as communication with remotely located teams and clashes for time to test on the single test environment available.
Unlike the previous project (e.g. TSM integration, cloud service procurement), there are limited external dependencies currently predicted. The following are the current dependencies that have been noted during project planning –
- DV III will be at a release level that supports migration from the interim vault e.g. roles and organisation in place
- Clarification of EDINA policy on deployment management e.g. move to new Puppet environment
- Availability of the PURE API to improve the metadata interface
- Support from ENT to upgrade SAN storage on live workers to 20TB (to support 10TB deposits)
- Support from RSS, ENT and CIS to review processing times and identify bottlenecks in the process
- Availability of a supported encryption and key management subsystem replacement e.g. Thales SafeNet
The initial project risks identified are as follows –
- Current system documentation will need to be reviewed and updated to provide a firm base for formal design
- The loss of a key project team member in April and their replacement not on board
- EDINA development staff not available towards the end of the project
- Budget will not cover Digirati costs for all work allocated to them
- Additional licencing (e.g. TSM) or hosting costs (Oracle Cloud Archive) may be incurred as the service is expanded
- Potentials users will not use the DataVault service e.g. user interface is too complex, processing takes too long, PURE metadata prerequisites
If they arise, issues will be recorded, on the Projects website, and tracked throughout the project. Issues that are not outside agreed tolerances will be resolved within the project team. Major issues need to be escalated to the project board for review and advice on resolution. Issue resolution that requires a major change to the project in terms of time, quality or cost needs to be approved under formal change management by the project board.
All project changes outside tolerances must be reviewed and approved by the project board. Project changes within tolerances can be managed with the project team.
Project work that impacts production systems or networks must be communicated with the requisite lead time to the ISG CAB for review and approval.
The following standard ITI project tolerances will be applied and these are outline below. Any deviation outside these tolerances must be reported for review and resolution to the Project Board, if constituted, or the Project Sponsor and Portfolio Manager otherwise.
- Cost – over +/- 10% of original budget
- Plan – slippage of over 4 weeks to current plan
- Scope – extensive addition or reduction from current scope
- Quality – refusal to approve a project work package
A log will be maintained on the Projects website to record lessons learned throughout the project.
The key observations from the previous DataVault project are summarised in the table below –
|Limited knowledge transfer from interim system||Development team for DV II not the same as used on interim system but there was limited knowledge transfer from interim system which put the new development team at a disadvantage. Minimal documentation produced from interim system.||Ensure DV III architecture and operations documentation is of a detail and quality that benefits DV III developers.||High|
|New development team engaged distinct for interim system team||
Development for DV II was subcontracted to EDINA. The lack of knowledge transfer and documentation provided a steep learning curve for the developers.
|Ensure DV III development team includes EDINA developers, if possible. If not ensure L&UC development staff are involved in the final stages of DV II and the system is fully documented at the architecture and operations level.||Medium|
|No detailed system design||There was no appointed system architect, so no detailed system design was undertaken. A 3-page outline system design document was produced. Extensive design decisions were done 'on the fly' during the build phase of the project. System evolved rather than being designed.||Ensure system architect in place and detailed design - functional and infrastructure - is completed in a distinct design phase prior to system build.||High|
|Workflow has high working storage overheads and long processing times||Workflow is inefficient with high working storage overheads and long processing times in what is essentially a system integration exercise e.g. encryption, TSM, Oracle. DV is fundamentally a secure archiving service.||A detailed redesign exercise is recommended to ensure the service provides acceptable archive and retrieval times. The system design evolved and includes various steps that could be replaced by more efficient methods.||High|
|Planning moved to Jira when project was in flight||A decision was made to use Jira (Scrum version) for planning by adding functional requirements directly into the Jira backlog in November 2017 (project started February 2017). Limited formal planning seems to have been in place. A formal plan was reverse-engineering from Jira in January 2018. Jira is a short-term planning and tracking tool. It does not provide facilities for overall project planning such as scheduling, resourcing and dependencies.||Use a formal planning process and tools for DV III. Jira can still be used for task level initiation and tracking.||Medium|
|Adoption of Jira forced a Scrum based approach||DataVault is not best managed overall as an Agile project for the following reasons - The system is quasi-regulatory (e.g. GDPR required). The target system was well defined. The project team was not 100% assigned to the project and had limited Agile/Scrum experience.
|DV III project should be managed using formal requirements-planning-design-build-test-launch approach. Once complete, the plan can be added to Jira and a sprint or Kanban based delivery initiated.||Medium|
|Long lead time identifying cloud supplier||The selection of Oracle cloud as the external archive supplier took over 6 months using the G-Cloud framework. With the value of the contract (£13,500 over 3 years) a single supplier could have been engaged directly.
|Ensure procurement approach is appropriate for the costs involved.||High|
|TSM licences not available as agreed||TSM licences from the RSS pool were to be used but when they were required they were not available. Unplanned expenditure (£10,600) was required to purchase the requisite licences.
|Ensure commitment to supply licences is formally confirmed.||Medium|
|Processing on DataStore impacted other users||Processing on DataStore, a networked shared storage system, impacted other users including staff network drives. Testing had to be curtailed and a request to move to ENT SAN raised where performance and reliability improved.||Ensure appropriate storage technology is used for DV working storage during the design phase.||High|
|Design of the system infrastructure was ad hoc||Design of the system infrastructure was done just prior to the provisioning of final test and production environments. The team had limited infrastructure design knowledge.||
Ensure infrastructure design is included at the start of the project as part of the system design process.
|Significant budget overspend on development resources||Spending on development resources ate into the FY2018-19 project (DV III) budget potentially reducing the work that could be done on the next phase of DV.||Effective planning at the start of the project identifies resource and scheduling requirements that can be used to control project spend more efficiently.||High|
|Three changes of project manager||During the project, four different project managers were assigned to the project which contributed to delays in progress as handovers were completed||Ensure a single project manager is assigned and retained on the project throughout.||Medium|
The following meetings and communication channels will be put in place to support the effective communication of project progress, events, product releases, and risk and issue management.
|Sprint Planning||Start of sprint||Robin Rice Ianthe Sutherland Robin Taylor David Speed Pauline Ward Lawrence Stevenson||Updated plan and backlog, next sprint|
|Sprint Meetings||Weekly or daily, as required||Robin Rice Ianthe Sutherland Robin Taylor David Speed Pauline Ward Lawrence Stevenson||Progress report, updated risks & issues, next steps|
|Sprint Retrospective||End of sprint||Robin Rice Ianthe Sutherland Robin Taylor David Speed Pauline Ward Lawrence Stevenson||
Lessons learned, actions
|External Workstream Reviews||Fortnightly||Digirati team Ianthe Sutherland Lawrence Stevenson||Work package requirements, progress reports, and modules for test and integration|
|Project Board||Monthly||Tony Weir Robin Rice Fraser Muir Colin Higgs Anthony Davie Kirsty Lingstadt David Fergusson Ianthe Sutherland Lawrence Stevenson Maurice Franceschi||Meeting minutes & progress report, escalations and resolutions|
Project Manager All stakeholders
|Project announcements & updates||As required||Project Manager or Service Owner, as appropriate||
Email, IS Alerts, website announcements
|ISG CAB||As required||Appropriate Senior Supplier or Project Manager||Change requests|