MVM’s main medical UG programme - Undergraduate Medicine (MBChB) - was currently in the first year of transitioning their assessed portfolio to PebblePad with marking and feedback delivered online via the PebblePad/Turnitin integration. The adoption of PebblePad has been as a mechanism to improve online assessment and feedback, and is very much a part of TEL assessment and feedback programme of project work.
The academic year 2014/15 concentrated on trialing the online portfolio with Year 1 undergraduate assignments which included ~250 students - this alone has been the biggest single uptake of PebblePad and, while successful, has been a big task ensuring coordination of key dates, engagement of key stakeholder etc.
This project would concentrate on scaling this migration to all five years of the MBChB programme for the 2015 academic year (starting in June 2015 for Year 4). This meant ~1,200 students submitting a total of approx. 10,000 assignments throughout this academic year. The Programme Owner felt that the Apps Project Management support would greatly improve the likelihood of a successful roll out.
The project itself involved a huge amount of work to make each year ready for submissions. This involved mapping out all the submissions for all the courses over each year, liaising with Brendan Owers of EEMeC for the initial access setup, setting up workspaces and rotations for each year in PebblePad, identifying and setting up markers in PebblePad, setting up personalised workspaces for markers to test their access, setting up assignments, assigning markers and students to each submission, setting up additional skills portfolios workspaces, populating workspaces and portfolios, setting up and adding feedback templates, updating EEMeC with project updates, guidance and FAQs for staff, students, markers and moderators, training for staff in PebblePad, training and guidance for markers and guidance for students on how to submit their materials for marking plus producing training guidance for all these groups plus moderator guidance. IS Helpline also helped with the updating of the relevant wikis. The project also engaged IS Skills to deliver PebblePad training to staff members. Each year was also monitored as submissions started to ensure there were no issues.
Issues encountered and resolved
Non-UoE addresses were needed in PebblePad: without these addresses non-UoE markers could not access PebblePad and therefore not mark MBChB assessments. These were currently being manually added to VRS which in turn updated PebblePad but with the increase numbers from 1 year to 5 years of students meant this became unsustainable. Graeme Wood of the UNIX team however found a way where the addresses could be added in bulk from csv files. It has now become apparent that this was a GradeMark issue and PebblePad has not been affected as originally envisaged.
Access by NHS NHS markers were unable to access our system through the firewalls on NHS premises meaning they could not access PebblePad to download the marking. NHS Lothian prevented downloads in Internet Explorer, but all machines had Firefox installed which allowed download. NHS Borders were able to access in Google Chrome. This was therefore managed by communication rather than requiring any additional software and was due to the efforts from Paul Clark and Lydia Crow for chasing up, sourcing and testing alternatives to the status quo.
Feed between IDM and PebblePad not working: to get students and staff into PebblePad the process had to be done manually which created longer timescales and required additional resource. A fix was not anticipated for this year but with the additional years being added in 15/16 the issues would be greatly increased. However the feed was quickly fixed in May 15 due to the efforts from Steph Hay and her team and IS Applications.
Issue with assignments open to modification: MVM have adjusted how they are handling this. This was resolved by MVM adjusting how they handled modification and instead created an assignment for each rotation rather than rolling assignments.
Provide Initial Access to PebblePad by beginning of June 2015
Training materials ready by beginning of June 2015
– slight delay but ready in time for first submissions
Each year prepared for submissions 7 days before submissions begin
Completion of courses
Access sub project completed
Communications plan completed by end of April 15
Training materials ready for training
Markers identified, trained and set up in PebblePad
Training of course organisers and markers completed
Plan for 2015/16 (includes moderation, release of marks and feedback, record keeping and review etc)
Please note that the following have all been created over and above the original scope of the project:
- MBChB portfolio additions and guidance
- Clinical Skills Log (Student)
- Clinical Log Book Guidance (Student)
- Clinical Skills Big 5
- Record of Achievement (Student)
- Notes of concern (Student)
- Patient Logs (Student)
- CVs and GPS (Generic Professional Skills) (Student)
- Student FAQs (Student)
- CTA Guidance (Staff) : is ready to go, but isn't public yet as waiting on final sign-off about who is marking what – will be confirmed within next week
- Personal Tutor Guidance (Staff)
- External Examiner Guidance (Staff) https://www.eemec.med.ed.ac.uk/pages/external-examiners-1
- Guidance for SSC4a Project Markers (Staff) https://www.eemec.med.ed.ac.uk/pages/ssc4a-project-markers
- Staff FAQs
- Pre-prescribing: Lydia Crow has been scoping this but it has not yet been approved as part of the formal MBChB (so hasn't been rolled out): Currently awaiting signoff from MVM. Once agreed will be added to PebblePad by Lydia Crow.
- Markers all received personalised emails and a personalised test environment set up for them to test access
- Feedback templates
- Rotations handled with individual assignments
- Year 5 Electives: SSC5 Elective Form and Risk Assessment will be hosted on PebblePad (which will be reviewed by Personal Tutors) plus the Completion Certificate
- Access to MVM into EEMeC – form developed that allowed MVM to deal with directly
Success Criteria met
Students on years 1-5 are able to submit their assignments electronically through PebblePad
Markers can easily mark submissions in PebblePad
Eventually – there were issues with NHS browsers and blacklisting of URLs by NHS prevented a smooth transition for Years 4 and 5 but with hard work from Paul Clark and Lydia Crow Years 1-3 will be able to easily mark submissions in PebblePad
Students receive feedback and marks timeously
Analysis of Resource Usage:
Staff Usage Estimate: 40 days
Staff Usage Actual: 37 days
Staff Usage Variance: -8%
Other Resource Estimate: 1 days
Other Resource Actual: 1 days
Other Resource Variance: 0%
Explanation for variance:
The main milestones did not waiver from their original deadlines due to the hard deadlines of meeting semester dates.
For minor milestones the following changes did not impact the critical path
- Training materials and documentation was pushed back by a fortnight so MVM and LTW could concentrate on getting assignments into PebblePad by 5 June then by a further fortnight in order to create video guidance. The materials were still ready in time for Year 4 submissions starting.
- The year 2015/1+6 plans were delayed due to project manager’s time from other projects but were completed by September.
An additional 10 days was added to the project for 15/16. This was due to a risk being realised (risk 1: Concentrated effort on project may demand more project management time than initially expected which will result in more budget required from the programme.). There was sufficient contingency in the programme to accommodate unplanned carry-over of this scale.
Key Learning Points:
- It was a small project team with Sue Woodger as Project Manager, Lydia Crow as MVM representative and Ellen Spaeth the LTW representative. For sign offs the team also included Nicola Crowley from MVM who was designated to sign off on behalf of the project sponsor. Despite the amount of work in the project the team worked well as a small team plus the team worked well together.
- The Project Manager would like to acknowledge the project team commitment and effort to ensure the project delivered. Both Lydia Crow and Ellen Spaeth were exceptional in their commitment to ensuring the project was a success and the amount of work they input – to some very rigorous deadlines – should be acknowledged. From Sue: It was a joy to work with both team members.
- While there was a small project team there was a large interested and diverse stakeholder group. To improve communications to this group , rather than have them ploughing through the weekly minutes, the PM set up a weekly newsletter for them highlighting the achievements made that week, any issues and how they were being dealt with, task updates, next steps, items coming up soon plus providing a list of links for the project – this was very successful and has already been picked up by other project managers in IS Apps and LTW
Blank names in PebblePad: happens when adding students and markers to PebblePad they show up with blank names and appears to be a timing issue. Once people log into PebblePad the issue rectifies itself but this isn’t helpful when managing a workspace. This is a known issue to Pebble Learning and has been logged as a critical issue. Service Manager for PebblePad has not been given a timeline for the fix but will be made aware when the issue is resolved. This will be taken forward by PebblePad Service Manager supported by MVM.
Feedback Postcards for CTAs and PTs – with CME to make a decision on whether they want to continue as-is (taking full responsibility for its execution) or for the Feedback postcards to be incorporated into PebblePad At the moment it looks like this item is staying with CME but that is to be fully confirmed.
Closure Survey responses