Friday, May 15, 2009

Draft Evaluation Plan

Evaluation Plan

I would be looking at effectiveness evaluation for unit 2780 – Operate a Personal Computer and 2781 Manage and Protect Data. Unit 20780/2781 is part of National Certificate in Computing level 2 units offered by MIT. I was initially thinking of only doing 2780, but since both of these units are offered in combination and students do both of them together. I thought the comments the students give would be for both of them so I better evaluate it as a whole cluster that contains 2780 & 2781.

Unit 2780 and 2781 is used to be offered via a text book, that students went through the theory followed by assignments and assessments. Just last year the whole course was changed to an online environment. Students work online with all the reading materials and quizzes and activities to work with and after completing the online materials than the students do the assessments.

My evaluation plan is to check:
- If delivering the course online makes it more efficient
- Whether students are able to use the online tools
The best way to do this evaluation is to get feedback from past students who had books and then compare it with current students who have everything online. Also look at their perspective by letting them know about the change and what would they have preferred.

The Sample
I would like to do the evaluation on students:
- Who has completed the two units
- Who are currently doing the units
- Who have done it by using the books (may be bit difficult to get hold of them)
- Will look at the age factor ( will try and keep it balanced)
- Will look at different branch students as well – to get a wider cultural group covered
Finding the sample of students and getting them to participate in the evaluation is a bit hard part. I would prefer having students who have at least done half of the course work – so that they are ready to give feedback. The ones who have just started the course – might prefer to take more time to give better feedback and I might not have time available to do my evaluation. I am planning to finish everything by the end of the semester so I will try and keep everything in the time line as such to allow me to be able to gather all the information be able to provide a report by end of the semester.

I will conduct the evaluation through:
- Questionnaires (suitable for most students)
- Interview
- Group discussion

I would prefer using the Multiple Methods evaluation model Mark & Shotland 1987 to conduct my evaluation project
Using the following guidelines:
TD2: Do students get clearly defined learning objectives that assist them in focusing on their learning activities?
- Are they able to follow the student guide and navigate through the course materials easily
- Are they able to complete the activities successfully
- Able to complete the assessments
TD5: Have a representative sample of students tested the e-learning materials, and any necessary modifications been made?

- To evaluate and see if putting the course online is made any changes
- To get feedback whether the students were comfortable using those tools.
- If any changes – what needs to be changed

Week 10 and 11 - Develop evaluation Methods. Find sample and Conduct evaluation
Week 12 - Gather all information and analyze it.
Week 13 - Publish results and receive feedbacks
Week 14 and Week 15 - Write a report based on the whole evaluation process and eh outcomes of it.

Time and Cost
I am planning to spend approximately 25 hours to complete the evaluation.
Cost - ?

There is more details to be added to this - please feel free to comment on it, while i am fine tunning it.


  1. Krishan,
    I don't think you would need to bother chasing past students who have done the course with textbooks, it sounds like alot of work and the timeline for our evaluation project is tight! You mentioned that you would prefer if the present students have done more of the course to give feedback on effectiveness... could you not just do a formal usability evaluation (you mentioned in an earlier post you like the concept i think?) I may be bias as I am doing one myself:) however if your course has just gone online it may be useful to find out how the present students are finding using it... do an effectiveness evaluation at a later date. Your elearning guidelines are great they could just be tweaked a bit for measuring usability vs effectiveness. Just a thought anyway. Good luck with your project .... hard juggling two papers hey! :)

  2. Hi Krishan
    I agree with Catherine's comments, particularly as it may be hard to get them to participate!

    Rather than compare how past students learnt with textbooks vs how current students learn online - I would just focus on the current students that you have now and evaluate how effective the online course is.

    There are some good evaluation tools on that would help you with this (questionnaire sample).

    Good luck

  3. Hi Krishan,

    I can see that you are trying to gain a comparison between the old course format and the new one, but does gathering data on the old course really answer your key questions? I think your questions are good, but are really focussed on the clarity and usability of the new course format - like Michelle and Catherine, I think it might be a waste of resources in gathering data from your old students.


    Iain W

  4. Hi Krishan,

    I agree with Michelle. It would be very difficult to get past students to participate into a past paper.
    In the "purpose" part, how do you define efficient? for the staff point of view (time) or the student point of view (easier to learn)?
    For the time and cost, you can get information from



  5. so Krishan it looks like four people think it is not going to be worthwhile finding out if students prefer textbooks. I am not so sure. it will be interesting to see what you find out. what do you think you will do?