If you want to develop a successful online course, you must keep your audience in mind. Online course evaluation is  a solid basis for course creation and in this article we’ll discuss online course evaluation by feedback from your audience. Using feedback is the only way of creating a course that will create any interest, and therefore ultimately sell.

Update: Literally 1 hour after reading this article and the associated workbook, Paco Vermeulen of https://pianoforedm.com/ had implemented a working evaluation system. Two weeks later, he had already collected 62 complete survey entries (8 questions each) with a lot of valuable feedback.  

 

testimonial

testimonial2

Why Online Instructors Fail

accidental-slip-542551_1280

Online Instructors define success by being able to generate interest for their courses, sell courses and enrol students. A course that nobody uses may be the best course in the world – the effort that went into creating it was still wasted. Good and bad, right and wrong are very relative. If you want to create a successful course, “wrong” is anything that your learners do not expect and want or would like you to change. Arguing that your way of constructing a course is still better is of little value when students disagree and will simply avoid your course.

Experts often make the mistake of thoroughly believing in a course idea that they’ve come up with based on their own experiences. But being an expert also means that you’re not in the same situation as your audience and they may have an entirely different idea of what your course should teach.

Without feedback for your online courses, you may

  • pursue the wrong learning goals,
  • choose the wrong pace,
  • make examples so difficult that nobody can follow or
  • so easy that there is no learning effect
  • use language that makes it hard for learners to be successful, and
  • you may assume previous knowledge that isn’t there.

You can prevent most of that by listening to your learners early on and getting feedback. In other words, you need to evaluate your course.

 

Why Online Course Evaluation Leads to Successful Courses

Online Course Feedback - Required for Success

 

As a course builder, you absolutely rely on your learner’s feedback to

  • Discover what your audience wants
  • Find out what works and what doesn’t work
  • Find out how effective your learning units and exercises are
  • Adjust course development accordingly.

In her 2012 article, Debbie Morrison points out an interesting factor to consider, as she indicates that unfavourable feedback may be more important in order to create the course your students actually want than favourable feedback.

An online course creator relies even more on feedback than school teachers and university professors. Lacking the immediate feedback a classroom teacher can get by interacting with students and reading their body language, feedback is often the only way of ensuring you’re on the right track.

3dcover_feedback-workbookWant to take all of this information home? Download your free PDF workbook containing this article, further details and a question pool!

Get the workbook

Getting feedback early-on also makes your life as a course creator so much easier. Why invest countless hours into tweaking course structure, without knowing if you’re on the right track?

Just ask, and get better quality information with so much less effort!

First time instructors often neglect feedback because they’re busy building courses and struggling with technology. This is a mistake, as all the effort spent on course creation may be in vain if the course fails to find an audience. Even if you’re lucky and find your audience by coincidence – your courses could be so much better if you listened to your learners.

Even experienced instructors struggle with the challenge of getting feedback though.

  • When should I request feedback?
  • Which questions should I ask?
  • How can I get feedback without spending all day on evaluating it?

 

Online Course Evaluation Foundations

medieval-castle-84336_1280

We can use Donald Kirkpatrick’s training evaluation model to structure feedback in general and try to place online course feedback. In this model, there are 4 levels of feedback:

  1. Reaction: What participants felt about the training.
  2. Learning: What students learned.
  3. Behaviour: How well did the increased knowledge/skills transfer to tasks/a job?
  4. Results: What changed because of the training? (i.e. promotion, raise, …)

In general, the earlier levels are easier to evaluate, while the latter levels provide more interesting data. In the context of online courses, levels 3 and 4 are really hard to evaluate reliably. Traditionally, you’d ask supervisors for a comparison of on-the-job performance before and after the training, which you obviously can’t do very easily online.

Feedback deals mostly with levels 1 and 2. You want to find out how students perceived a course, and how much they think they learned. You will also want to find out how much they actually learned (as opposed to what they believe they learned), and quizzes are a typical tool for level 2 evaluations. However, that is an entire discussion of its own. Levels 3 and 4 are difficult to evaluate for online courses, but we can still try by asking relevant questions.

For more background on feedback and a research based view, you might want to listen to the elearningCoach podcast with Will Thalheimer.

Choosing Your Evaluation Strategy

1603800

Your evaluation strategy will cover aspects such as

  • Which questions will I ask?
  • How will I accomplish that?
  • Which tools will I use?

Here are some general recommendations to decide when and how to get feedback from your students.

9 Essential Recommendations for your Course Evaluation Strategy

  1. Feedback is part of your ramp-up
    Feedback is not an afterthought. Tackle your feedback system early in your course creator’s career. Feedback will allow you to get better, and if you implement it early on, you will have valuable data to work with and you’ll be able to track your progress.  If you don’t, you’re losing time and data.
  2. Feedback is a part of course building
    Setup your feedback system early when creating a new course. Do not go perfecting your course before you actually know what your audience wants. Your idea of your course is what you want – what your audience wants is in the feedback.
  3. Be Concise
    Keep feedback forms short. Do you like filling out 3 pages of questions? Neither do your students. If you go overboard, you’ll receive ultra-short answers without any useful content.
  4. Sufficient Information
    You want to improve and get valuable insights into your training. If you limit the number of questions too much out of fear of annoying your students, you won’t get the data you need. Find a compromise that works for you!
  5. The Right Frequency
    Try to get feedback as soon as possible. Decide when (and how often) you will ask for feedback! Only at the end of the course? At the end of each unit? Not every learning platform will let you choose, but you can usually get by with 3rd party tools. Asking for quick feedback after each lesson will allow you to compare data for each lesson, and by applying the 80/20 rule you will be able to make major improvements by investing a reasonable amount of time by tackling the biggest issues first. If you think your learners will not like that, and you go for a survey a the end of a course, there will be a limit of what students will be able to tell you.
    A great way to get feedback during the course is using discussions/forums within courses. You won’t have a formal feedback mechanisms and usually no anonymity, but you should still be able to see how well students are doing and how much they appreciate the material. When students use discussions, make sure to reply in a timely fashion. Do not create the impression that nobody is monitoring discussions. A platform/course that seems abandoned will be abandoned.
  6. The Right Time
    Not all data will still be available if you only have a survey at the end of a course (students will have forgotten lesson specific remarks by then). If you need data concerning the beginning of the course – ask it earlier.
    At the latest, you should ask for feedback right after the end of the last lesson. Don’t wait for 2 weeks to send out a request for feedback via email – that’s too late, and much valuable information will be gone. However, you may still send a link to an additional survey to find out how well students were able to apply knowledge a while after the end of the course.
  7. The Same Questions Every Time
    Stick to the same form every time (unless change is unavoidable). This allows you to do automated and graphical comparisons, identify your weak spots and apply the pareto principle to increase quality and results quickly and efficiently. You’ll probably require changes sooner or later, but be aware of the disadvantages of a changed form and balance the pros against the cons.
  8. Automate
    It’s best to use an online tool to get feedback. Evaluating answers manually is time consuming and error-prone. Using a tool allows you to minimize the effort spent on evaluating feedback and get the answers you need at a mouse click. Many course platforms have features to achieve this. If your’s doesn’t, you can use services such as surveymonkey or typeform. If you don’t want to spend money, you can construct a solution with google forms/sheets.
  9. Ask Specific Questions
    You want to ask specific questions which will yield specific answers. This is hard to do for some categories of questions, but you need to keep it in mind. When constructing a question, try to think of possible answers. If the answer isn’t going to be specific, it probably won’t help you. Try not to ask too many questions where answers will be subjective.

Choosing your questions

When choosing questions, please bear in mind that when you’re filling out a survey, anything past the 5th question starts to feel tedious, and anything more than 15 questions is going to feel like a hassle to your learners. Evaluate for each question whether or not it’s necessary.

Screenshot 2016-04-23 13.05.37

Choose your questions now and write them down. Have a look at the overall survey, and adjust if necessary.

  • Does the survey cover all aspects you’re interested in?
  • Did you avoid repetition?
  • Are all questions really necessary? Will you gain additional, valuable knowledge from each question?

Feedback Tools

Screenshot 2016-04-23 13.07.54

Once you’ve chosen your questions, it’s time to set up your tools. How you go about this will depend on what your course platform offers.

If integrated surveys are offered, use them. Using integrated solutions is always preferable: It’s easy, the look & feel stays the same and the user experience isn’t broken, and you have all your data in one place.

If there’s no functionality called “surveys” or “feedback forms”, you could just use quizzing functionality for the same purpose. You just need to make sure that these quizzes won’t be graded automatically.

If this doesn’t work on your platform, you may resort to external helpers, such as

You can just set up surveys on any of these platforms and then include the link in your learning units, lesson descriptions, or you might email it to your students. The look&feel of your survey will be a little different, and accessing your data will require a visit to a site other than your school. Some learning platforms will also allow you to embed a form inside your course so that users don’t even notice that your form is hosted elsewhere.

You may want to consider using these external helpers even in case you have built-in survey functionality, but you have decided for a strategy that isn’t supported by your platform (e.g. a quick, two question feedback form after each lesson).

My personal recommendation for online course creators is Typeform. It’s the single most user-friendly survey tool in existence, and the awesome UX will increase your survey completion rates. Premium features such as logic jumps will potentially shorten your survey if you’re willing to pay for them.

Implementing Your Evaluation Strategy

chess-1214226_1280

Preparing Users

You’re done now, right? Wrong. If you chose your questions correctly and set up your toolchain, you’re off to a good start. But now the real work begins.

Educate your learners about feedback! Ask them to provide feedback, and tell them why it’s important to you.

If they don’t know why they’re giving it, and you won’t tell them, they might assume it’s an annoying feature of the course hosting platform that has no relevance anyway. As a result, the answers you receive will be of little value.

If your students don’t know how their feedback is going to be used and what is going to happen with the data, or if they don’t know what kind of answers are expected (“I’m supposed to be nice and praise the instructor, right?”) – don’t expect any useful results.

When preparing your users, don’t sound corporate! “I”, not “we” (unless you’re a team). No clauses and formulas, but actual conversation between people. Be nice, and give your participants all the information they expect.

Collecting Feedback

If you’ve set up your tools and given your users the information they need, you’ll start to see feedback trickle in over time. If set up correctly, this is an automatic process for the most part (unless you’re using forums/groups).

Dealing with Feedback

Screenshot 2016-04-23 13.10.58

Here are some considerations for dealing with feedback. They’re not rocket science, but they are difficult to carry out at times when your own work is being criticized. It may help to take a look at these items and become more aware of them.

  • Attitude
    Be ready to receive feedback. This may sound trivial, but if you’re not open to feedback, you might as well leave it. You will not like everything you hear, but hearing it is the only way to get better – even if “better” sometimes is just different. You don’t know what your students want until they tell you. Your idea of the best possible course may be fine for you, but your students may prefer something different.
  • Objectiveness
    Analyze feedback objectively. Why did the student feel this way? What are the reasons for this? What could be done to avoid it in the future? Often, simple additions to courses are sufficient to avoid criticism. Another short sentence to set up expectations, a picture to make sure it is understood what you’re referring to. Sometimes, there’s really nothing you can do – but you should not approach feedback with the attitude that there’s nothing you can do.
  • Don’t block
    Don’t start preparing a rebuttal while reading feedback. This is not the point here. You may disagree, but that ultimately doesn’t matter. Your students are telling you about their perception, and – if necessary – it is your job to change that perception if you can. You need to do this by adjusting the way you teach, not by telling them that they’re wrong. The latter is a surefire way of guaranteeing no more feedback from them in the future.
  • Trick Yourself
    Focus on positive feedback first. This will make it easier for you to accept negative feedback. Nothing will ever appeal to everybody. A certain degree of negativity is ok. If you put yourself in a positive mood, you’ll find it easier to process negative comments more objectively.
    Don’t take feedback too hard. Your material will never appeal to everyone, and when you put yourself out there, you’re bound to get some negative feedback.
  • Granularity/Efficiency
    When feedback is specific/objective (“there is a mistake in paragraph 3 of slide 17”), check it out individually. When feedback is general/subjective (“lesson 3 sucked”), look at feedback overall. If the opinion is shared by a number of participants, do further analyses.
  • Urgent Cases
    If a majority thinks a lesson was bad, it needs to be reviewed urgently. If a minority thinks a lesson was bad, but many others liked it, this is no indication that the lesson needs to be changed immediately. Talk to the minority and find out where the discrepancy comes from. Maybe the course prerequisites/expectations weren’t clear for them. If there are equal parts who dislike and like something, you might want to adjust the course description/prerequisites.
  • Dealing with Unhappy Customers
    If a single person (or very few people) think the lesson was bad, talk to them. Maybe you can help them, or maybe they want to leave the course. Offer a refund. Single sales are not worth bad reviews.

Making Use of Online Course Feedback

Feedback is only valuable if you’re going to act upon it.

Once you’ve finished your feedback analyzes, make an improvement plan by grouping the feedback you’ve received in three categories (you’re free to invent further groups if you’d like).

    • Group 1: Urgent. Mistakes should be corrected immediately. Don’t wait until your next launch to correct these – you can fix them immediately and improve the experience of other students.
    • Group 2: Adaptations, extensions, fixes, changes to better serve the needs of your students. Not urgent, and often time-intensive. This group should be incorporated into the next course update/launch.
    • Group 3: Won’t fix. Put items in this group when you cannot accommodate learners. This may be subjective feedback, or feedback that contradicts what the majority of your students expects.

Screenshot 2016-04-23 12.18.28

  • Publish Feedback: The bests way to ensure people know that you’re taking their feedback seriously is to collect and aggregate it, and in regular intervals publish the feedback as well as your action steps. You could do this via your mailing list, or on your blog.
  • Thank your students for their feedback. This can be combined with the previous step. If you decide that you won’t publish feedback results and your planned actions, at least send a thank you note so that they’ll be motivated to give feedback again.

The Online Course Feedback Workbook

3dcover_feedback-workbook

I’ve created a comprehensive feedback workbook for online course creators to get you started. It contains the information from the article above with additional details and more than 50 feedback questions you can use.

Get the workbook

This workbook is catered to online course creators, who need to solve the following major problems:

  • Content: Which questions to ask and when and how to ask these questions
  • Process & Tools: How to set up a feedback mechanism

This valuable workbook is available for FREE. Click below to receive it immediately! 🙂