Home » Posts tagged 'assessment'
Tag Archives: assessment
|From E-Learning V|
More than any module or exercise I have done over my four years with The OU, it is a MOOC in FutureLearn that is giving me the most thorough experience of where the future or learning lies. I’m in week seven of eight weeks of ‘Start Writing Fiction’ from The OU, on the FutureLearn platform. Just in these few weeks I’ve seen the site change to solve problems or to enhance the experience. Subtle lifts and adjustments that make a positive out of constant adjustment. Those tabs along the top: activity, replies where under a tab. I think ‘to do’ is new while ‘progress’ was elsewhere. This is a responsive platform that listens to its students.
In the final week we submit our third piece of work.
As assessments go these are far less nerve racking than a TMA. The first piece was 300, the second 500 and the last will be 1000. These are assessed by fellow students. In my case I had one, then two reviews. Most people seem to get at least two sometimes three. The system is designed, I’m sure, to try and ensure that everyone’s work is reviewed at least once. Tens of thousands, certainly thousands of people are on the course.
We’re here to the 19th of December or so … if you follow the tracks as laid.
I hazard a guess that between 20-100 have posted there final piece already. Some, I know, got to the end of the entire course a few weeks ago; I looked ahead to see out of curiosity. There have always been 20 who post comments one, two even three weeks ahead. If 20 are posting I hazard a guess knowing my stats on these things that another couple of hundred could be clicking through the pages to read and observe. They may, like me, be coming back later. They may only be following the course, but not participating. Often, it is like standing on a stage looking into the gloom of the auditorium. Someone probably out there. One or two let you know. The rest don’t.
I hope those that race ahead come back …
I find that if I get ahead then I slow down and retrace my steps. To learn in this connected and collaborative way you are far better off in the pack … it is not a race to get to the end first. In fact, those who do this have already lost. They’ve missed the point. I’d suggest to people that if they have the time to do the week over. That’s been my approach anyway – the beauty of these things is everyone can come and go as they please, at a pace that suits them. Skip a bit. Go back. Follow it week by week, day by day … or not. Whatever works works?
There’s another very good reason to stay with the ‘pack’ or to come back and do a week over – the platform depends not on tutors and moderators commenting and assessing work, but us students doing a kind of amateur, though smart, peer review. This is what make a MOOC particularly vibrant, memorable and effective. Not listening to an educator telling us what’s what, but the contributors sharing, figuring it out, answering each other’s problems in multiple ways. We all learn in different ways and at a pace that shifts too. I find that often a point I don’t get first time round, on the second, or third, or even the fourth visit to an activity someone, somewhere puts it in a way that suddenly brings complete clarity – their way of seeing a thing, or expressing it, makes more sense than the writes of the course could manage. Because they can only write one version, not the ‘tartan’ that comes from an intelligent, threaded online conversation.
Facts in an essays are like pepper in soup – the wonders of FutureLearn, the Paris Conference of 1919 and the birth of the League of Nations
|From E-Learning V|
Fig.1. Facts in an essays are like pepper in soup
How do you compare and mark a variety of Massive Open Online Courses (MOOCs)?
We need to treat them like one of those challenges they do on Top Gear, where Jeremy Clarkson – Richard Hammond – James May set off to Lapland in a Reliant Robin or some such and then get marks across six or so criteria. Hardly scientific, but it splits the pack.
So, let’s say we take THREE MOOCs, what criteria should there be?
- Commitment. What percentage of participants signing up complete the course?
- Comments. I use the word ‘vibrancy’ to judge the amount and nature of activity in the MOOC, so this is crudely reduced to the number of comments left.
- Likes. Another form of vibrancy where comments left by the team and by participants are ‘liked’. It has to be a measure of participation, engagement and even enjoyment
- Correct answers. Assuming, without any means to verify this, that participants don’t cheat, when tested are they getting the answers right. This is tricky as there ought to be a before and after test. Tricky to as how one is tested should relate directly to how one is taught. However, few MOOCs if any are designed as rote learning.
You could still end up, potentially, comparing a leaflet with an Encyclopaedia. Or as the Senior Tutor on something I have been on, a rhinoceros with a giraffe.
It helps to know your audience and play to a niche.
It helps to concentrate on the quality of content too, rather than more obviously pushing your faculty and university. Enthusiasm, desire to impart and share knowledge, wit, intelligence … And followers with many points of view, ideally from around the globe I’ve found as this will ‘keep the kettle bowling’. There is never a quiet moment, is there?
I did badly on a quiz in a FutureLearn Free Online Course (FOC). World War 1. Paris 1919. A new world order …
I think I got half right. I chose not to cheat, not to go back or to do a Google search; what’s the point in that. I haven’t taken notes. I wanted to get a handle on how much is going in … or not. Actually, in this context, the quiz isn’t surely a test of what has been learnt, but a bit of fun. Learning facts and dates is, or used to be, what you did in formal education at 15 or 16. This course is about issues and ideas. A ‘test’ therefore, would be to respond to an essay title. And the only way to grade that, which I’ve seen successfully achieved in MOOCs, is for us lot to mark each others’ work. Just thinking out loud. In this instance the course team, understandably could not, nor did they try, to respond to some 7,000 comments. They could never read, assess, grade and give feedback to a thousand 4,000 word essays. Unless, as I have experienced, you pay a fee. I did a MOOC with Oxford Brookes and paid a fee, achieved a distinction and have a certificate on ‘First Steps in Teaching in Higher Education’.
As facts are like pins that secure larger chunks of knowledge I ought to study such a FutureLearn FOC with a notepad; just a few notes on salient facts would help so that’s what I’ll do next week and see how I get on. Not slavishly. I’ll use a pack of old envelopes or some such. For facts to stick, rather than ideas to develop, the platform would have needed to have had a lot of repetition built into it. Facts in an essays are like pepper in soup.
Armed with an entire module on research techniques for studying e-learning – H809: Practice-based research in educational technology – I ought to be able to go about this in a more academic, and less flippant fashion.
|From E-Learning V|
Fig.1. Grainne Conole’s 7s of Learning Design
7Cs is an OU with OU Learning Design Initiative with JISC through the Curriculum Design Programme. Activity Profile and Course Map. Trialed thoroughly.
Gráinne Conole continued this work with the JISC funded CARPE Dium learning design workshops at Leicester whiuch provides a ‘ rich storyboard of learning design’.
Overarching conceptual framework a lot of Cs here:
- Course features – the essence of it.
- Creative activity
- Combine – into course map and activity profile
- Consolidate – running it as face to face, or VLE, or more specialised learning design tool
or …. From Gráinne‘s blog:
|From E-Learning V|
With current thinking on 7Cs Various systems offered and can be tried.
Listening to OLDs MOOCers it appears that the 7Cs framework has been received well
- It articulates what teachers already do.
- There are 7 aspects in a whole design process.
- What level are you teaching, what level of support do they need etc:
- Teachers (all of us I would say, educators, learning designers, L&D managers) are bewildered by the range of tools, the range of approaches so fall back on their own content. So use the tools to think about the activities, the core essence of hte course.
- Indigenous Culture on locality.
- Introducing elements of serendipity.
- Activity profile
- Is it the right mix of learning for what you want the students to do.
- Correlation of time mapped out to what students are achieving … so she is poor at communication in Spanish … and there is little communication in the course she is doing.
Is this the right tool set?
- Covers all the aspects of design.
- Getting a taster for these in the course.
‘A huge amount in the MOOC is mix and pick, so take your time, come back to the resources. Six months down the line, you discover which ones you like’.
- Some love the activity profiles some don’t, so find the mix that works for you.
- Some with learning outcomes.
- Some with the content.
- Some with the characteristics of the context of the learners.
- Different tools will mean different things to different people.
‘We’re offering a Smörgåsbord of offerings that you can develop and use over time. Pick the ones that are relevant to you, don’t feel that you have to use all of them’.
(More coming up in WK 8 to act as a springboard to reflect)
- What is learning design?
- How has it come about?
- Why is it different to structural design?
2011 ALTC National Teaching Fellow
- Driven by people in Europe and colleagues in Australia.
- What is learning design? How has it come about?
- How is it distinct from instructional design?
- Major Epiphany moment Sept 2012
- Two days in Cyprus
- Timeline of key moments since 199 learning design
REF: Key books on design science (Dianna Laurillard) Teaching Design as a Science It’s aimed to be pedagogically neutral so that it can be used across a range of methodologies and pedagogies.
- Tools for guidance and support
- Tools for visualisation
- Tools for sharing like Cloudworks
What works for you
- It depends on the nature of how people want to go about things
- Connect and be sociable
- Open, unstructured … to form some kind of navigatable way through, as well as enjoying the serendipity. Having the options of the long and short routes.
- Is something more needed in the middle ground. B MOOCs.
‘Teachers want support and guidance to help them rethink their design practice, to think beyond content to and activities to make pedagogically informed design decisions that make good use of technologies’. Grainne Conole.
I’ve just been listening over the OLDs MOOC hangout for Week 3 and particularly enjoyed the Q&A with
The sentence above stood out from the 60 minutes, as well as how this was put into context for the MOOC in Week 3 and coming up in Week 8.
Personally I wish that we’d had something like this to begin the week. I got in early, did a couple of activities then followed the noise from the active design group I’ve joined. Give others a turn. Let things roll over. This works. Leave gaps and sometimes others will come along and think, OK, he’s done that so I can see how it works, or might work for me. I won’t bother with that tool, I’ll try something else and see what people make of it.
I cherry picked and as this hangout suggests and recommends; I’ll go back and pick out more as required.
I enjoyed downloading, colouring in, cutting out then using the Activity Cards. This is more my thing than the EXCEL spreadsheet – which I planned on a sheet of paper then transferred over. I might use an APP to generate such a thing. I find EXCEL somewhat heavy handed, or I’d want to design it in a way that I like. We learnt about the background to 7Cs. The background and context was invaluable. Credibility ought not be taken for granted. Work like this needs to be put on a pedestal and people told of its credentials and worth – i.e sell it to me!
Had this been the title of a post-graduate diploma in e–learning it would have been precisely what I was looking for a decade ago – the application of theory, based on research and case studies, to the design and production of interactive learning – whether DVD or online.
A few excellent, practical guides did this, but as a statement of fact, like a recipe in a cook book: do this and it’ll work, rather than suggesting actions based on research, evidence-based understanding and case studies.
Mayes and de Frietas (2004) are featured in detail in Appendix 1 of Rethinking Pedagogy for a Digital Age (2007) Beetham and Sharpe.
Four types of learning are featured:
- 1. associative
- 2. constructive (individual)
- 3. constructive (social)
- 4. and situative.
Of these I see associative used in corporate training online – with some constructive (individual), while constructive (social) is surely the OU’s approach?
Situative learning may be the most powerful – through application in a collaborative, working environment I can see that this is perhaps describes what goes on in any case, with the wiser and experienced passing on knowledge and know how to juniors, formally as trainees or apprentices, or informally by ‘being there’ and taking part.
Each if these approaches have their champions:
Associative – Skinner, Gagné (1985).
Constructive (individual) – Piaget (1970), Papert (1993), Kolb (1984), Biggs (1999).
Constructive (social) – Vygotsky (1978).
Situative – Wenger (1998), Cole (1993), Wertsch. (Also Cox, Seely Brown). Wertsch (1981), Engestrom (), Cole and Engeström (1993)
Beetham and Sharpe (2007:L5987) – the ‘L’ refers to the location in a Kindle Edition. I can’t figure out how to translate this into a page reference.
How people learn and the implications for design
Associative – Skinner, Gagné (1985) (in Mayes and de Frietas, 2004)
Building concepts or competences step by step.
People learn by association through:
- basic stimulus–response conditioning,
- later association concepts in a chain of reasoning,
- or associating steps in a chain of activity to build a composite skill.
Associativity leads to accuracy of reproduction. (Mnemonics are associative devices).
- Routines of organized activity.
- Progression through component concepts or skills.
- Clear goals and feedback.
- Individualized pathways matched to performance.
- Analysis into component units.
- Progressive sequences of component–to–composite skills or concepts.
- Clear instructional approach for each unit.
- Highly focused objectives.
- Accurate reproduction of knowledge.
- Component performance.
- Clear criteria: rapid, reliable feedback.
- Guided instruction.
- Drill and practice.
- Instructional design.
- Socratic dialogue.
FURTHER READING (and viewing)
Brown, J.S. (2002) The Social Life of Information
Brown, J.S. (2007) October 2007 webcast: http://stadium.open.ac.uk/stadia/preview.php?whichevent=1063&s=31
+My notes on this:
+The transcript of that session:
Biggs, J (1999) Teaching for Quality Learning at University, Buckingham: The Society for Research in Higher Education and Open University Press. (Constructive alignment)
Cole, M. and Engestrom, Y. (1993) ‘A cultural-historical approach to distributed cognition’, in G. Salomon (ed.) Distributed Cognitions: Psychological and Educational Considerations, New Work: Cambridge University Press.
Conole, G. (2004) Report on the Effectiveness of Tools for e-Learning, Bristol: JISC (Research Study on the Effectiveness of Resources, Tools and Support Services used by Practitioners in Designing and Delivering E-Learning Activities)
Cox, R. (2006) Vicarious Learning and Case-based Teaching of Clinical Reasoning Skills (2004–2006) [online], http://www.esrcsocietytoday.ac.uk/ esrcinfocentre/ viewawardpage.aspx?awardnumber=RES-139-25-0127 [(last accessed 10 March 2011).
Engeström, Y (1999) ‘Activity theory and individual and social transformation’, in Y. Engeström, R, Miettinen and R.-L. Punamaki (eds) Perspectives on Activity Theory, Cambridge: Cambridge University Press.
Eraut, M (2000) ‘Non-formal learning and tacit knowledge in professional work’, British Journal of Educational Psychology, 70:113-36
Gagné, R. (1985) The Conditions of Learning, New York: Holt, Rinehart & Winston.
Gagné, R.M., Briggs, L.J. and Wagner, W.W. (1992) Principles of Instructional Design, New Work: Hoplt, Reihhart & Winston Inc.
Kolb, D.A. (1984) Experiential Learning: Experience as a Source of Learning and Development, (Kolb’s Learning Cycle) Englewoods Cliffs, NJ: Prentice-Hall
Kolb, D.A. (1984) Experiential Learning: Experience as the Source of Learning and Development, Englewood Cliffs, N.J: Prentice Hall.
Littlejohn, A. and McGill, L. (2004) Effective Resources for E-learning, Bristol: JISC (Research Study on the Effectiveness of Resources, Tools and Support Services used by Practitioners in Designing and Delivering E-learning Activities).
Mayes, T. and de Frietas, S. (2004) ‘Review of e–learning theories, frameworks and models. Stage 2 of the e–learning models disk study’, Bristol. JISC. Online.
Piaget, J. (1970) Science of Education and the Psychology of the Child (Constructivist Theory of Knowledge), New Work: Orion Press.
Papert, S. (1993) Mindstorms: Children, Computers and Powerful Ideas, New Work: Perseus.
Piaget, J. (2001) The Language and Thought of the Child, London: Routledge Modern Classics.
Seely-Brown, J.S and Duguid, P. (1991) ‘Organizational learning and communities-of-practice: toward a unified view of working, learning and innovation’, Organizational Science, 2 (1): 40-57
Schon, D (1983) The Reflective Practioner: How Professional Think in Action, New York: Basic Books.
Sharpe, R (2004) ‘How do professionals learn and develop? In D.Baume and P.Kahn (eds) Enhancing Staff and Educational Development, London: Routledge-Flamer, pp. 132-53.
Vygotsky, L.S. (1978) Mind in Society, Cambridge, MA: Harvard University Press.
Vygotsky, L.S. (1986) Thought and Languages, Cambridge, MA: MIT Press.
Wenger, E. (1998) Communities of Practice: Learning, Meaning, and Identity, Cambridge: Cambridge University Press.
Wertsch, J.V. (1981) (ed.) The Concept of Activity in Soviet Psychology, Armonk, N
Appendix and references largely from Beetham, H, and Sharpe, R (2007) Rethinking Pedagogy in a digital age.
See also Appendix 4: Learning activity design: a checklist
Gagnés events of instruction:
1. Gaining attention . The scene opener, even the preview or title sequence.
2. Informing the learner of the objective . Laying out your stall
3. Stimulating recall of prerequisite learning . Tapping into what has already been understood – creating empathy.
4. Presenting the stimulus material . Presenting the case, offering evidence that might impress or inspire, that could be controversial and memorable.
5. Providing learning guidance. Offering a way through the maze, the thread through the labyrinth or the helping hand.
6. Eliciting the performance . Now it’s their turn.
7. Providing feedback . Sandwiched, constructive feedback on which to build.
8. Assessing the performance . How are targets going?
9. Enhancing retention and transfer . Did it stick, could they pass it on and so become the teacher?
You wouldn’t necessarily know it was a wiki either, rather it is a shared document held online with secure access by a group of people collaborating on a complex project.
The roles are well-defined, as clear as those on a film production team (with similar titles):
- Senior Production Manager
- Production Manger
- Learning Design
and so on
New to an e-learning office I find I am permanently online adding to a massive, collaborative wiki which is the e-learning course with its plethora of inputs.
Email rarely comes into it, why should it?
Wikis are lean production, they operate ‘just in time’ with each cell responsible for picking up their task as it best suits them.
The Open University provide an OU Student Blog platform, which you are required to use for some modules to build up reflective practice, they also provide a portfolio called MyStuff in which to dump stuff.
As portfolios either system can be used to aggregate content that can be shared, offered with restricted access or kept private.
I have been on the Masters in Open & Distance Education (MAODE) for two years, we have to give blogs, portfolios, wikis and other tools a go.
The interesting thing is to see how it plays out in practice during these MAODE modules.
I can cite failures as well as extraordinary successes.
Like learning to do anything new people/teams need to accept that at first they are getting into the sandpit to play.
Letting go of inhibition is tricky, academics in particular find it very hard to touch the words of another person.
The trick, I find, is to think of myself as a writing team, that the words that appear as text might just as well be a conversation around a meeting room table. Over time the ‘script’ will be bounced around.
- A wiki needs to be ‘populated’ with some text, ‘seeded’ by someone just so that there are some ingredients to get started on.
- Don’t fuss about spelling, grammar or even the accuracy of ideas that you present. Indeed, the rougher the initial input, even the presence of easy to fix mistakes, the more likely someone will dip their toe in the water and fix the obvious. The polished whole should be the product of the group enterprise.
- The magic isn’t the finished result, but the ability with current tools to trace back and forth through the ‘narrative’ of changes. In Google Docs you can contribute using different colour text which makes this ‘animation’ all the more easy to read. I found I got a fantastic sense of the thinking process, the logical changes, the ebb and flow, the occasional false trail corrected.
Have a go in Wikipedia
I was surprised how easily I signed in as an editor, found I subject I knew something about and jumped in with text and images. This felt like the first time I swam in a 50m pool.
My conclusion, shared amongst fellow students, is that the ‘modern’ blog platform, such as WordPress offers all of this, as in a wonderfully simple, bulletin board kind of way the OU’s own blog offering.
Six categories of eportfolio:
1) Assessment – used to demonstrate achievement against some criteria.
2) Presentation – used to evidence learning in a persuasive way, often relate to professional qualifications
3) Learning – used to document, guide and advance learning over time
4) Personal development – related to professional development and employment
5) Multiple owner– allow more than one person to participate in development of content
6) Working – combine previous types, with one or more eportfolios and also a wider archive.
Three kinds of e-portfolio (Matt Villano):
(A note on blogging. Spurred to say something about wikis based on my current experience in an international e-learning business with 70+ offices around the world I refer to the OU Student Blog I have kept since February 2010. Amongst its 1000+ pages there are 23 tags to wiki, or I can search ‘wiki’ in the blog. This reaches out to any notes I have taken during the FOUR modules I have thus far completed, where wikis, amongst many Web 2.0 tools are carefully introduced and discussed at length drawing on academic papers, the course content, input from out tutor, my student group and from the student cohort on this module who contribute to the vibrant asynchronous conversations in the various social learning environments offered).
Not only did I write one of these in my OU MBA Module exam (B822 Creativity, Innovation & Change), I even drew it as the essay plan for one question. Large and bold with space to annotate my six ideas into each petal and to scribble down some thoughts on the introduction and conclusion (up and down the stem).
I doodled mnemonics on the themes around the petals.
I gave myself 5 minutes per theme and 10 minutes to get into and out of the topic.
I suppose I wrote 200 to 350 words per theme.
Pegging works. They may not look very pretty but anagrams, sayings and mnemonics all have their place. However, you have to fix them in your head, which includes all of the senses. I had to visualise a couple of my mnemonic lists (VAN BECK CLIMB)as a short walk through our house seeing hockey sticks and flippers in the porch at the front door (playfulness), a woman sitting on the stairs with a broken tennis racket (adopt a set to break sets), at the top of the landing another woman cuddling an infant (it was always there, nurture it), then a massive Damien Hirst butterfly picture on the wall (broad picture, local detail) and so on ... people in a room, riding a bicycle (oh yes!), then for key components of a 'messy' problem '[I could use a colourful swimming costume: ICUACSC) a photograph of James Mason perched on the end of springboard with a Russian looking girl called Olga Mitroff. In both cases I used this in the exam, quoting 'Mason and Mitroff' and drawing in what I saw as I walked through the house.
I ended up with 5 mnemonics per block, some only three letters, as in VATCC and CAPO.
VATC the caveats to using a personality inventory:
- Technical considerations
That's 15 which a week on I still know as: POVCC, MHIVE, and MDMAP ...if you can see it these are 'point of view', 'Maldhives' and 'Mindmap'. These 15 key letters gave me the longer mnemonics and phrases. All of this I practiced and on the day filled a 'rough book' in 10 minutes. When it came to looking at the questions, with one exception, the fit was immediate. With the third I had to cherry pick from what I had laid out. It took time to embed these pegs in my cerebellum, but if you want to sit down to an exam, turn the question page and develop a knowing Mona Lisa smile this is the way to do it.
Mind Dump for a written exam on ‘Creativity, Innovation and Change’ B822 with the Open University Business School
This is my mind dump, something I lay-off onto a sheet or sheets in the first 10 minutes of sitting down to a written exam. In this case the few gaps I had covered by the afternoon when I sat the exam. What had taken me 20 even 30 minutes took me 10 minutes on the day as I produced something even more cryptic than you see here, only adding the detail once I’d chosen the questions I would answer.
My exam for the module ‘Creativity, Innovation & Change’ B822 of The OU MBA took place in Hove Town Hall, a 45 minute train and walk-it from Lewes. The last time I sat something like this? Finals, June 1984. There were fewer of us, some on other subjects, the desks were miniscule and bizarrely I found food and drink permitted.
I was also surprised when someone on our module left after 90 minutes. I suppose if you simply turned the question paper over and immediately wrote you response to each question this would explain it; I gave between 45-60 minutes to preparation, selecting the question and creating a mindmap for each as an essay plan.
The reality could be that I do well at exams, but can’t handle the assignments getting into deep mud as a consequence of having too much time. I should set myself parameters. I could only have learnt this from sitting an exam, the value of locking myself away for three hours with a blank sheet of paper and then editing this with course books and notes to prepare an assignment.
Each to their own.