Pilot Evaluation

Pilot Evaluation

Evaluation 

Considerations 

• Evaluation - were all of the messages sent out in a timely manner. 
• Baseline - all the other programmes. Couple of equivalent to answer the same questions. Evaluate the 2 other platforms against that. For students and programme reps.  
• Focus group - via Collaborate 
• % of students who've accessed the platform - built-in analytics on how the channels are being used.  
• Owner of the team - can click through to analytics - Manage Team - then analytics tab. 
• Gillian - have to go via surveys unit to send this out.  
  • https://www.ed.ac.uk/staff/data-matters/about-surveys-reporting/student-surveys/surveys-ethics-committee/ethics-application
  • Also attached the Student Voice Diagram. 
• Natalie - for reps at end semester - from March to end April it's a handover doc for all reps (in pilot and not in pilot) 
  • - first half are for reps next year.  
  • - 2nd half more about comms from Student Association, plus looking at and analysing what happened in their role from the operational side.  
• Adapting what we have rather than additional questionnaire.  So go to all reps.  
• Compare those responses.  
• Still students also - need comparable programmes to send survey to students on the pilot and others for a baseline. 

Focus group with the reps - collaborate and possibly in-person sessions. 

What questions to ask in the focus group? 

• Timing for student surveys - one eval per semester at the end of the whole pilot?  
• Before Christmas - NH had week 10 as check-up with online comms.   
• Decision: eval semester 1 with reps (re the tools, and suggestions to improve)  
• and then semester 2 with reps (through normal survey from Student Association for handover) and students (plus focus group).  
• Need to identify equivalent programmes that haven't taken part, to be able to have a baseline for the comparison. 
• Has the Student Association training taken place? In Natalie’s reply above 
• Has the Student Association Pilot training taken place? In Natalie’s reply above 

Intro for Teams 

Intro for Learn 

How will it deemed to be successful? 

• What is the uptake – what % makes it successful 
• Can we see the uptake by each method, e.g. comparing a programme not in the pilot with the programmes that are 
• Teams Analytics – Stephen 
• Looked at what Teams provides 
• Query if Learn can provide something similar Karen? 
• Natalie will see if the Marketing Team can communicate with the other means of communication and stream like Learn in  MYED?    

Consider a midpoint review, maybe Sem1 week 10 - Hadn't been rolled out by them

• Any suggested changes in case they can be incorporated easily before Sem 2 – ALL    
• Is this a better way to communicate with students  and reps? 

Student has query posted on platforms, rep responds and fees back to student? 

• Is it an improvement? 
• Is it a better way than other media?  
• Are you (the students) more likely to use it? 

Usability of pilot 

Other methods of evaluations to consider at this point 

• None 

Sheila and Natalie swapping useful contact names    

Evaluation  - Survey Unit. Gillian provided a link to it 

• https://www.ed.ac.uk/staff/data-matters/about-surveys-reporting/student-surveys/surveys-ethics-committee/ethics-application 

Need to contact surveys unit 6 weeks prior to when we want it