Home » E-Learning » Evaluation
Category Archives: Evaluation
Tough Love
The Learning Thermal
Every day, as a Coursera Mentor, I receive notification from them in the form of an email indicating which student queries in the common forum require a response; every day I respond to one or two and in due course I get feedback. It is always a pleasure that my reassurance, prompt or suggestion is welcome. Having done this for 7 months I recognise a pattern: students (as I did) worry about assignments, which are of two types: peer reviews of submitted work and graded quizzes (typically around 12-16 carefully crafted questions). Both are tightly controlled: as a student you must review three pieces of work submitted by others (selected or offered from a rolling list as your cohort moves through the course) and to pass the course in its entirety, the quiz grade required, individually and collectively is high, certainly over 60%.
Having done four courses, each with several increasingly demanding parts, I know how much anxiety these can cause, more so where I have paid a fee to be part of the assessed cohort so that I get the most learning out of it and gain a certificate too. We don’t like being judged or criticised, so peer reviews need to be done with sensitivity and completely fairly. The instructions for assignments are specific: whether an essay, proposal or project write up, or another piece of submitted work such as a photograph, the factors that will result in marks being awarded and carefully spelled out. The student must then trust his or her work to a fellow student. Being a global and open platform Coursera attracts everybody who has access to the Internet: young, old, English a second or third language, at High School or a post-grad (even doctoral research students boosting their ego or doing a refresher). Some people take the peer review more seriously than others: you’d be unlucky if more than one student gave no more than a cursory review and worse, if they marked your work down on a trivial technicality (or plainly get their review wrong). On the better courses, and all I have experienced on five different Coursera courses (each having between One and Five substantial parts) the multiple-choice quizzes are well written and thought through; what is more, when you redo the quiz if your score is too low you will at least find the questions are in a different order, and nay find some questions are even different – you can’t ‘game it’ by repeating the test over and over trying different combinations of responses. Students new to this kind of learning experience and to opening their minds (and to some their soul) to strangers, get agitated: they panic, they fret, they even become inpatient and angry. The educators who ‘wrote’ the course or the platform itself can come in for a lot of Schtick.
Standing back from it all, were I advising a group of educators about to embark on the creation of a Massive Open Online Course (MOOC), I’d advise ‘tough love’ – some educators, through experience, and by their nature, might be like this anyhow. It does no one any favours to produce a course where any component may be deemed at best ‘edutainment’ and at w0rst ‘amateurish’ or plain wrong.
‘Tough love’ gets results and in the longer term is what the student and educator wants. In a different field, coaching swimmers, something that grew out of helping out at a club where my kids were learning to swim competively, it has taken me a while (over ten years) through training and experience, to deliver ‘tough love’ whether the swimmer is 8 or 18. Unlike in an online course, in these live situations I can respond to strengths and weaknesses, spot failings and go back. Crucially where something isn’t getting the results I need I can call upon a collection of approaches. They cannot progress until they can do a thing correctly. Likewise, online, the smarter courses, blocking the way to advancing, will send a student back to review a part of the programme that ought to help them with an answer they are failing to get right. Better still the student is offered a different way of seeing the problem; the better courses providing more than just a reading list – they link to a specific paper, or book, or video.
A few times, taking a series of courses on ‘Search Engine Optimisation’ I became stuck and had to go back over two weeks of material and find a different person giving a different explanation before the ‘penny would drop’ – it really was a moment of enlightenment, and getting the quiz score from something like 4/16 to 11/16 gave me such a great sense of achievement that I persevered until I settled for 15/16. I have to trust the designers and writers of the course than 16/16 is possible : students can get online and blame the course, not themselves, for being unable to ‘meet the grade’. Shame on the educators if they have made a mistake. Best practice is, in any case, to use information on any sticking point and go and fix it. It was one of the earliest findings shared by Daphne Keller in her TED lecture on the first course offered on Coursera that with tens of thousands of students the data would identify spots where the educators weren’t teaching something very well – too many students were making the same error to blame them so they went in and fixed it.
I like the idea of building a course as best you can and then using the learning stats and student experience to go back and fix problems and make improvements: you build an obstacle course for the brain, but it is no good if too many people fail where they shouldn’t. Nor is it of any value to make it so easy to render the course worthless. I know Courses, entire platforms, that are educationally valueless ‘edutainment’, PR for the educational institution – a taster of a campus based course. learning is not viewing: a timeline that indicates that you have clicked through 50% of the material is not worthy of a pass. Being flippant with a formative quiz is not funny; I’ve seen ‘educational’ quizzes where, like in a TV gambit aimed to get tens of thousands of punters wasting £1 on a punt the right answer is blindingly obvious from a choice of three. And paying to study at postgraduate level where 40% is the pass rate is so indulgent as to be diminishing of the institution.
The ‘toughest love’ is the examination hall; the ‘toughest test’ the real world. Would you want to be seen by a doctor whose course work in training repeatedly only scrapped above a 40% grade? And if the ‘learning’ was so easy did much go in and stick?
At school, and since, the educators I have most admired were always the toughest: their demands and expectations were high. However ignorant you were, if you stuck at it and showed willing they got you over the bar, they’d never lower it.
Watching TV you sit back and let it wash over you; you may even fall asleep. Learning on an interactive platform, as you should do in a lecture or tutorial, and certainly when writing an essay or sitting an exam, you ‘lean forward’ – you engage the brain – the harder you are made to think, the greater the struggle, the more likely you have learnt something lasting and of value on which you can build.
9 ways to create the perfect online course
Fig.1 Mosaic by featured in the University of Cape Town FutureLearn course ‘Medicine and the Arts’
Don’t call MOOCs MOOCs, they are ‘courses.’
Don’t even call them online courses. I suppose therefore, don’t call it e-learning either or even online learning … it is simply ‘learning’. I am on my eighth or ninth course with FutureLearn. I may have three or four open at any one time and complete two of these at least. I love ‘Medicine and the Arts’ from the University of Cape Town while I am both maddened and intrigued by ‘The Mind if Flat’ from Nick Chater. I’m certain that online courses longer than a couple of weeks should not be treated like books or TV programmes. What works best, as the University of Cape Town shows, is to get the entire team involved. They have a lead host and presenter who each week introduces several colleagues, something like four to six each week. It is stimulating and necessary to hear from so many different voices.
1. The Platform Provider
Brand and technical aspects. Think of this as the channel. It has both technical and brand qualities. Is it smart? Is it current? Does it all work faultlessly? Is it intuitive? Is it simple? I’ve done many FutureLearn courses but struggle every time with Coursera and EdX. Feedback on Udacity is dire from both suppliers and users.
2. Funding/Cost or Cost Benefit
You can’t make a movie in $125,000 dollars. If a 30 point 16 week distance learning course from the OU costs £1.5m to produce should a 3 week MOOC cost up to £300k? It’s a poor comparison is the cash cost may be a fraction of this: a university team’s job is to plan a programme of teaching anyhow. What matters is how a budget is spent. The learning designer for an online course is like the scriptwriter for a movie: they provide the blueprint. Is the investment worth it?
3. The Subject Matter
Are you true to your subject? Don’t try to be something you are not. Is it ‘made’ for an online course, rather than shoe-horned from a regular, traditional ‘classroom’ lesson plan? Would it be better served on a different platform in a different way? Can you teach sports coaches or movie directors online? Or rather, what can you, and what can you not teach them? Are you fully exploiting the affordances of the platform and easily linked to alternatives on the Internet.
4. Audience
Who do you attract and is this the same as who you get? Who do you attract by level of education, age, gender, culture and location. Are you getting the audience you want as participants? The contribution participants make is crucial. Are there enough active voices to sustain this? Be aware of the extreme differences in digital literacy skills and competences. Do you know your audience? How do you relate to those who start the course? Do you try to appeal to multiple ‘personas’ – a dozen student character types, as the Open University does, or do you appeal to one person only, as an author would do?
5. Champions
One advocate over more than a couple of weeks will tire. It will feel like an ego trip any way. How good is the mix of contributors? Both in what they have to saw and show, and their levels of and variety of experience. An online course is not necessarily akin to a TV documentary that can be carried by a single presenter. Is it a one man show or a team effort?
6. Objectives
What are the hidden and implicit goals? To attract students, to build reputation, for the good of mankind? To make money? To massage an ego? What do results say in terms of those completing a course? Doing assignments and getting to the end then singing the praises of the team? Another guide can be whether as a production fulfils the initial Creative Brief. Both qualitative and quantitative research is required to provide answers.
7. Brand and creation/production values
Is is possible to stay true to your own brand, even have a distinct image, when on someone else’s platform? Are the values of the design, creation and delivery consistent with the standards and image of your institution? If not publishing, and not TV what is it? It cannot be a lecture series with a reading list and essay put online. It has to pick the strengths from individual media platforms to succeed in this multimedia setting.
8. Assessment
These must never be taken lightly. There are examples of trite, ill-thought through multiple-choice quizzes: these are a learning opportunity. A good quiz makes you think, challenges your knowledge, and provides feedback whether you get it right or wrong. Bravely ‘Medicine and the Arts’ has both quizzes and a regular written assignments. These are not onerous yet some participants are scared by a 300 to 500 word piece of writing. They oblige you to read back through the week’s activities before replying.
9. Connectedness
How ‘sticky’ is the content? Has it got people talking to each other, not simply replying to the headline. Are people connecting as ‘friends?’ Are they continuing this relationship beyond the ‘walled garden’ of the ‘open’ learning site? Does interest in the subject, in the presenters and the institution ‘have legs’ – does it last for the years before a person may make the time, and raise the funds, to take a formal course?
How to take notes for effective studying and assessment
Fig.1. The muddy sides of the River Ouse, Piddinghoe. At low tide.
We are very good at forgetting: it’s vital.
We see, feel, sense far too much in our daily lives (which includes when asleep). Come to think of it what on earth was I doing on a student exchange to North America last night where I am twenty years older than my hosts … (probably sums up how I feel about the workplace).
See. Some memories are made for us, or by us whether or not we want them.
Learning though requires us to gather, create and retain stuff. Some of this stuff is forgettable; it doesn’t resonate, or is poorly taught or expressed. Or we simply don’t get it the way it is expressed, or the first time around.
Make it a memory
At an OU Residential School the session on revision was packed. The tips made us laugh: sucking a choice of Polo Fruit sweets by subject theme – when you come to the exam repeat and each sweet will link you to that period of revision. Odd. But it worked often enough for me to convince me of its value.
Fig.4. Ebbinghaus and his ‘Forgetting Curve’
The science from the likes of Hermann Ebbinghaus and his ‘Forgetting Curve’ simply indicates how something fades, unless you go back to it a few times over several days over which period you make it stick. It doesn’t say anything about the ‘stickiness’ of the memory in the first place. Sometimes this stickiness is made for you. There is drama, there is an explosion. Most likely, by chance, the learning is anchored by some unrelated event like the fire alarm going off – that won’t work for 50 different things though.
Fig. 5 Multiple ways of making ‘it’ stick: read (book and e-book), highlight, tag and take notes.
If the module, or your tutor isn’t doing it for you then the next step is to dig around for a book, video or image that does it for you.
Most likely, and of far greater value, is for you to turn that lesson into a memory of your own creation. There is always value in taking notes, so never listen to the presenter who says ‘no need to take notes I’ll give you the slides afterwards’. Never trust the quality of the slides. What the person said will be of more value then the slides. You, and your handwriting, and your doodles are how it starts to become a memory. Then when you write up or rewrite those notes you do it again. You make it into something.
Fig.6 The River Ouse at low tide.
I’m fixating on the horror of drowning in a shell-hole in the First World War.
Ever since I was a boy those images of cowboys and Arabian princes sinking into quicksand has horrified me. What must it have been like? Walking the dog by the River Ouse at low tide just as it turned the gurgling of water backing up and filtering into the muddy bank gave me the shivers. That sound was ominous. It made a memory of the walk and the thought. It’s also what is sustaining me as I work at a short story.
Fig. 7 A family memory of a wedding in California. Will it stick?
We’ve talked about ‘memory making’ in the family.
It is the event, and the sharing of the event. My late mother-in-law was horrified that her daughter couldn’t remember a road-trip they did across the US when she was 13. I concluded that she hadn’t remembered much, or couldn’t remember much when it was mentioned out of the blue, as the trip was never shared. Conversations are and were always about current and future events. This is why it helps to get the old photo albums out from time to time. But there’s a loss. Do we make them anymore? Visiting a mislabelled album online is never the same.
Fig. 8. My late grandfather John Arthur Wilson MM with the author Lyn Macdonald at the spot north of Poelcappelle, Belgium where he buried two of his mates – 75 years after the event. He recalled it ‘like yesterday’.
Recalling the First World War
Some veterans would talk, others remained silent. Those who did not want to remember could and did forget. My late grandfather was a talker; it drove my mother mad. I came to love his recollections. Clearly, there were events that would have burned themselves into the memories of these men, but unless they talked about it, in a veteran’s association or with family and friends it was not going to stick. No wonder veterans would seek each other out over the decades. Nudged by histories and movies their memories could be changed though; sometimes they came to say what was expected of them ‘the rats were huge, the generals useless, the German bunkers impenetrable, the mud up to your waist, the sound of the individual shells … ‘
In conclusion
Whatever activities and devices are built into your module, you are responsible and can only be responsible for making something of it. Take the hint. Engagement takes time so make the time for it. These days it is made easier through the Internet. You can keep a blog to share or as a learning journal; you can talk it over with fellow students either asynchronously in a forum (or blog), or synchronously in a webinar. You can ‘mash it up’ with images, grabs, doodles and annotations. You can make it your own. It’ll stick if you want it to but superglue requires effort. Someone else ‘sticks it’ for you and it won’t happen.
Towards an assessment of my teaching
I often share a post I am writing as I do so. In this case having identified the story to tell : running a workshop to solve a ‘messy’ business problem I am pulling together or creating supporting images, in the above case a grab and mashup from Martin Weller’s book ‘The Digital Scholar’ – my goal is to be recognised as one. In a forum post as an Open University ‘Master of arts in Open and Distance Education’ (I graduated in early 2013) I suggested this could be achieved in four years – John Seely Brown thinks that eLearning speeds things up, while Weller reckons on ten years.
Reflecting on a presentation on learning to students at Oxford Brookes University
Fig.1. Rescue having failed a 4 tonne whale is dragged from Stinson Beach.
As a student on Oxford Brookes University’s online course ‘First Steps into Teaching and Learning 2014’ here in week 4 we have been challenged to consider an experience from teaching or being taught and in a five minute presentation reflect on this.
My interest is teaching postgraduates and/or ‘in the workplace’.
I should be feeling I’ve stumbled into the right time and place with this one having just given a ten minute presentation online as part of the Open University Masters in Open and Distance Education module H818: The Networked Practitioner, however with that one, despite every expectation to exploit my love of and experience with linear and interactive media I resorted to a Powerpoint. I needed to improve the script up to the line and this offered the flexibility I could not have had with a Prezi or video. There were too many cumbersome technical barriers and trips that I wasn’t happy to pursue or risk.
What I’m doing here is thinking through a presentation I need to prepare. Sharing this, if and where feedback can be garnered, then informs the decisions I take.
My immediate idea, often my best, is to do a selfie-video talking to camera while hurtling around a roller-coaster at Thorp Park. It would sum up the terror, thrill, highs and lows of taking a day long workshop with a class of some 40 year 9s (12/13 year olds) in a secondary school that had/has a checkered history.
The second idea, to change the setting radically, would be a workshop with nine on creative problem solving – the objective was to come up with answers to a messy problem, though the motivation to be present for most was to experience a variety of creative problem solving activities that I had lined up. This nine in an organisation, included MBAs, prospective MBAs, a senior lecture, junior and senior managers and officers: colleagues and invited guests from different departments. This example is probably the most appropriate.
A third might be something I attended as a student – apt because doing this in 2009/2010 in part stimulated me to take an interest in learning: I wanted to know what was going wrong. Here we had prospective club swimming coaches doing everything that was unnatural to them – working from a hefty tome of paper, sitting through a lecture/seminar and expecting assessment to be achieved by filling in the blanks on course sheet handouts. This from people with few exceptions who left school with few or no qualifications – often troubled by Dyslexia. They were swimming coaches to dodge this very kind of experience. It was, you could tell, hell for some. The misalignment could not have been greater. Here the immediate visual image, apt given the subject matter, would be to watch a fish out of water drown – or nearly drown and be rescued. What really grated for me in this course was the rubbish that was taught – too many gross simplifications and spurious science.
Based on the above I should challenge myself to do the video as I need to crack loading and editing. The fish out of water, whale actually, I can illustrate from photographs and the experience this summer of being present as a 4 tonne whale beached and drowned on Stinson Beach, California (See Fig.1. above).
We remember thanks to repetition
 |
| From E-Learning III |
Repetition or re-visiting is vital. We cannot help but change our perspective as we gain more experience, insights and knowledge. We need repetition in order to get ‘stuff’ into the deeper recesses of our brains where wonders are worked. Therefore, far better to exposure to brilliance often, rather than giving them something less than brilliant simply because it is new, or an alternative. If nothing else Web 2.0 ought to be giving students the chance to find and limit themselves to the best.
Web Networks – from the micro to the macro
We are each unique – our brains make us so. At the microlevel the network in our heads is then tickled out into the the Web in, at first. the simplest of ways. Our first post, our first comment is that first baby-step. Unlike our firsf steps though, online everything we do is saved, is monitored, is shared. It takes on a life of its own. Multiplied billions of times now many millions of us have learnt to crawl, then walk, then run online. As we are virtual we can split into many versions or parts of ourselves too – the professional and private the immediate split, but then into hobby groups and as here, a study group. The network of networks is a living thing that mathematics can help to weight and categorize, even to visualise, but crucially – the point made here, humanising the maths requires the insight of someone asking questions, seeking to interpret what it taking place. I see currents in a digital ocean that transpires into a cloud that then precipitates digital artefacts in a myriad of other places. Others, like Yrjo Engegstrom, see the growing tendrils of a funghi. Either way it is fascinating to condense, simplify and sharing the thinking.
What is digital ‘academic’ scholarship? Should 19th and 20th century definitions even apply?
Martin Weller published ‘The Digital Scholar’ in 2011 on a Creative Commons Licence. You can download it for free, or purchase the book or eBook, and then do as you will with it. When I read it I share short excerpts on Twitter. I’ve blogged it from end to end and am now having fun with a simple tool for ‘mashing up’ designs called ‘Studio’. It’s a photo editing tool that allows you to add multiple layers of stuff. I rather see it as a revision tool – it makes you spend more time with the excerpts you pick out.
You cannot be so open that you become an empty vessel … you have to create stuff, get your thoughts out there in one way or another so that others can knock ’em down and make more of them. Ideas need legs. In all this ‘play’ though have I burried my head in its contents and with effort read it deeply? Do we invoke shallow learning and distraction with openness? If we each read the book and met for a tutorial is that not, educationally, a more focused and constructive form of ‘oppenness’?
In relation to scholarship shoulf the old rules, the ‘measures’ of academic prowess count? In the connected world of the 21st century ‘scholarship’ is able to emerge in unconventional ways, freed of the school-to-university conveyor belt.
REFERENCE
Weller, M (2011) The Digital scholar
University Of California Approves Major Open Access Policy To Make Research Free
Over 125 million people want a university education – current global provision struggles to cater for 5 milllion – making the source content free doesn’t create an educational package – it lack the scaffolding, assessment and accreditation, but it is food for the hungry.
Why skiing is my metphor for life and learning
Fig.1. Mont Turia from the summit of Aiguille Rouge, Les Arcs at 3250m
On the last day, on the last run of my first week’s skiing I broke my leg rather badly. I was 13. I was in hospital for a week. In a wheelchair for two months and had the leg re-broken as it wasn’t setting properly. I spent six months at home. Idiot. But most 13 year old boys are.
I missed the next season.
For the following 20 years skiing mattered – a gap year working in the Alps (Val D’Isere in the Sofitel Hotel working 13 hours a day 7 days a week), a decade later researching a TV documentary and book (Oxford Scientific Films, Skieasy Ski Guides), falling in love with a fellow skiing enthusiast (we’ve been married 20 years), a honeymoon on the slopes and ten years later, on the slopes with a 4 and 6 year old, then again when they were 10 and 12.
I miss it.
(See above – the last week of the season, Tignes. The only people on the slopes are the ‘seasoniers’ who have worked since December. It is like being on the beach. A stream that flows above Val Claret melts and various ponds form. We ski it.)
Early in the afternoon I’d asked my girlfriend if she’d marry me. I was feeling cock-a-hoop.
We’ve been back twice in the last decade. There have been other priorities. I’ll be taking my 14 year old son out later this month or in April. Is that wise? At this age teenagers really are prone to take risks and can lack the physique.
Reasons to celebrate and look forwards
37 months to the day after starting the Masters in Open & Distance Education (MAODE) I got the final result, for H810: Accessibility in Open Learning – supporting students with disabilities, today. 84.
It has been so worth it and such a better, engaging, effective, experience than my undergraduate degree in a traditional university some decades ago. I feel as if I have earned it for a start. I have survived disasters rather than succumbed to them.
I am a reading, thinking, writing machine.
I feel like someone who has come to skiing late in life and has caught the bug. My mother started skinning in her mid 40s … and in her 50th year (unencumbered by her husband who was with wife three by then) sold the house and did a belated ‘gap year’ working a season in the Alps. The equivalent for me has to be the intellectual challenge of doctoral research.
More reading, thinking and writing – with research and teaching too I hope.
Onwards.
Tutor Marked Assignment One (TMA01) for H809 (Practice-based research in educational technology) is due on Monday.
Why more?
‘Practice-based research in educational technology’, to use skiing as a metaphor, is like learning to ski ‘off-piste’. Apt, as the tracks I make are ones I have planned, rather than keeping to the groomed, signed and patrolled ‘safety’ of the regular runs.
And my reward?
Fig. 2. Mont Blanc – From the Ski Resort of La Plagne, Above Montchavin. Les Arc on the right . The road to Val d’Isere clinging to the mountain in the middle distance Bourg St. Maurice in the bottom of the Valley
Skiing en famille.
We’ve not been out for five years so it should be a treat. It has to be on a shoestring, so short of hitching to Bulgaria can anyone recommend ways to keep the cost down?!!
Related articles
- An Insider’s Guide to Skiing Europe: Where (outsideonline.com)
- Ski Report (kotokydo.wordpress.com)
- Best of Val d’isere (besttripever.wordpress.com)
- Ski centers think spring and hope skiers do too (poststar.com)
Mind Bursts 