Malicious metrics

Clark Quinn

Years ago, when we built an adaptive learning system (it would profile you versus me, and then even if we took the same course we’d be likely to have a different experience), we realized we’d need to profile learning a priori. Yes, in good test design, you ask a question a number of times to disambiguate. Like others, I have been seduced by the “what X are you” quizzes on FaceBook.

Social Media Metrics

Clark Quinn

I continue to get asked about social learning metrics. Until we get around to a whitepaper or something on metrics, here’re some thoughts: Frankly, the problem with Kirkpatrick (sort of like with LMS’ and ADDIE, *drink*) is not in the concept, but in the execution. And I also like what I heard added at a client meeting: level 0, are they even taking the course/accessing the system? Frankly, even activity is a metric.

Researching open online courses

George Siemens

In fall, we (TEKRI, NRC, UPEI and possibly a few other organizations) are hosting an open online course. The course will run for about 35 weeks and credit options will be available for students in the Athabasca University MDE program and with Georgia Tech. It’s fairly easy to take online courses or interact with your favorite academic these days. Since 2007, numerous people have offered some version of open online courses.

Course 164

Thoughts on Learning Design Strategy

Clark Quinn

At the DevLearn conference, I ran a Morning Buzz on Learning Design Strategy. I started with a set of questions to address, so I’ll go through their comments in roughly that order (though we didn’t exactly follow this structure): What is learning design strategy?

Egoless design

Clark Quinn

A number of years ago I wrote a series on design heuristics that emerged by looking at our cognitive limitations and practices from other field. One of the practices I covered briefly in one of the posts was egoless design, and a recent conversation reminded me of it. The context for this is talking about how to improve our designs. This applies to instructional design as well. One of the things we could, and should do, is design reviews. design social strateg

Design 129

ROI and Metrics in eLearning

Tony Karrer

I'm working on an article around the use of ROI and metrics in eLearning. To ensure adequate funding and organization-wide commitment to your training program, you must be equally committed to Online Course Development: What Does It Cost? Does it cost less to design and develop online teaching and learning today than it did a few years ago? Proving 900 Percent ROI The ROI for e-learning internally at Cisco is 900 percent per course.

ROI 110

Is the New Facebook “People Talking About” Metric As Meaningless as Number of Likes?

Beth Kanter

Mari Smith pointed to this blog post from All Facebook explaining the metric and how it has been implemented in apps. His take: “It is a very shallow metric. And it does feel somewhat gratifying that I think we’ve been right all along. Photo by lecercle.

What to evaluate?

Clark Quinn

To be fair (I know Will, and we had a comment exchange), he’s saying that there are important metrics we should be paying attention to about what we do and how we do it. Of course we have to be professional in what we do. I put it another way: they’re efficiency metrics.

Secrets of Working Smarter

Jay Cross

Those in the quantitative data/metrics/benchmarking camp argue against the legitimacy of the notion of “informal” learning. As often as not, they claim workplace learning is too important to be left up to happenstance, and requires planning and careful, thorough, design.

Social Learning - Instructional Design - Free - eLearning Hot Topics

Tony Karrer

May 26, 2009 The Excellent Inevitability of Online Courses - Chronicle.com , May 27, 2009 Creating scenarios for learning - a four-step guide , May 27, 2009 Learning and Working in Complexity Workshop , May 29, 2009 Will Higher Education Be the Next Bubble to Burst? , May 28, 2009 Integrating work support systems , May 27, 2009 Top Keywords Social Learning (18) Instructional Design (25) Free (36) Collaboration (17) Poll (4) Develop (15) eLearning Technology.

Significant Work Needed to Help Instructional Designers

Tony Karrer

There's little practical advice that can assist Instructional Designers in what the evolved forms really are. Dennis Coxe : The role of the instructional designer will be to use the ADDIE model to determine what baseline structure can be built into the formal piece of the learning and what parts of the knowledge base are fluid and need to maintained delivered in an informal venue, be it a blog, a podcast, or talking points delivered by a project manager to his or her team.

Design 100

Formal Learning is (or should be) Expensive!

Clark Quinn

Courses are more expensive. In other words, we need to take the time to identify what business problem this is solving that a course will affect, and the associated metric. Then we need to design an intervention that will address that skill shift: we need to determine what the change in the workplace behavior needs to be to impact that metric, and then design an objective that reflects that needed behavior change. design social strategy

Because quality matters

Clark Quinn

Yet we still see strategies limited to courses, and courses designed poorly, and thus money being wasted that could be doing good. They might meet one particular metric, but when you look at the big picture, aligning improvement with respect, you need to have a rich solution. We are seeing more people interested in improving their learning designs despite considerable budget and time pressures. design meta-learning strategy

The Only Person Who Behaves Sensibly Is My Tailor

Charles Jennings

Effective Metrics for Learning and Development I wonder what Shaw would think if he saw the way learning and development is predominantly measured in organisations today. Attending a course or completing an eLearning module tells us little apart from the fact that some activity occurred.

Pushing back

Clark Quinn

In a recent debate with my colleague on the Kirkpatrick model, our host/referee asked me whether I’d push back on a request for a course. Being cheeky, I said yes, but of course I know it’s harder than that. The technique is to drill in for more information under the guise of ensuring you’re making the right course. design strategy

ROI 184

A Competent Competency Process

Clark Quinn

In the process of looking at ways to improve the design of courses, the starting point is good objectives. As Van Merriënboer’s 4 Component Instructional Design talks about, we need to separate out the knowledge from the complex problems we apply it to. One thing is that if you’re trying to cover a whole college level course, you’ve got to break it up. This is heuristic, of course, but with a focus on ‘do’, you have a good likelihood to get here.

Agile?

Clark Quinn

The topic is interesting to me, because like with Design Thinking , it seems like well-known practices with a new branding. As context, during grad school I was in a research group focused on user-centered system design , which included design, processes, and more. What was current at the time were iterative, situated, collaborative, and participatory design processes, so I was pretty familiar with the principles and a fan. design strategy

What’s the CEO want to see

Clark Quinn

Of course, we have to admit problems we’re having as well. minutes” or “we identified the problem and were able to reduce the errors in manufacturing back to our y/100 baseline” These may even include “saving us $z” To do this, of course, means you’re actually addressing key business impact drivers. That strategy, of course, comes from new ideas. That’s the learning that occurs outside the course!

Digital Learning Survey 2019 – Most Tech Doesn’t Stack Up

Xyleme

Since content is centralized and easily searchable, authors can quickly find and reuse content for their courses, lessons or other learning objectives. Instructors can easily administrate all assessments and course materials from one location.

Survey 151

eLearning Process Survey results!

Clark Quinn

Order taking doesn’t allow for seeing if a course is needed and SMEs can’t tell you what they actually do. I assumed people would know to choose the lowest option in the list if they could, and I don’t know that (flaw in the survey design). design strategy

Survey 125

Re-thinking Workplace Learning: extracting rather than adding

Charles Jennings

Adding Learning to Work Many learning professionals and training companies have taken the lesson about the criticality of context to heart and are designing courses and programmes that link learning with work more closely than was done in the past.

5 Phrases to Make Mobile Work

Clark Quinn

The second one focuses on looking beyond the initial inference from the phrase “mlearning”: Anything but a course! Here we’re trying to help our stakeholders (and designers) think beyond the course and think about performance support, informal learning, collaboration, and more. While it might be about augmenting a course, it’s more likely to be access to information and people, as well as computational support.

Digital Learning Survey 2019 – Most Tech Doesn’t Stack Up

Xyleme

Since content is centralized and easily searchable, authors can quickly find and reuse content for their courses, lessons or other learning objectives. Instructors can easily administrate all assessments and course materials from one location.

Survey 130

Manifesting in practice extended

Clark Quinn

Those should be your examples, and guide your practice design. You should be developing some extra content at development time for streaming out over time, but breaking up your course so that the hour of seat time is 30 or 40 mins up front, and then 20 or 30 mins of followup spread out over days and with repeated practice will make learning stick way more than not. Of course, you should be measuring your own work, and similarly if your interventions are as efficient as possible.

a simpler approach to km

Harold Jarche

Here is a simple guide on how to enable organizational sense-making, not the mere management of data and metrics. of course, there has to be something to curate, and that is only available when most people in the organization freely share.

RFP 223

Never Under-Estimate the Immune System

John Hagel

Every large and successful institution has an immune system – a collection of individuals who are prepared to mobilize at the slightest sign of any “outside” ideas or people in order to ensure that these foreign bodies are neutralized and that the existing institution survives intact and can continue on course. In more stable times, institutional immune systems are very effective at keeping institutions focused and on course, resistant to the distractions that might lead to their downfall.

System 146

Quip: tuning

Clark Quinn

When I run my game design lectures and/or workshops, I point out that as Will Wright once told me, tuning is 9/10ths of the work. Now, in most of the formal learning situations we design for, we have sticks as well as carrots, so we aren’t expecting our learners to pay for the privilege of completing our learning experience, but it’s important to understand what learner experience we think would be reasonable and shoot for achieving that. design games

SUNY Initiatives Supporting Online Learning Panel

Stephen Downes: Half an Hour

Project metrics: ease of implementation, how it's used, minimum configuration, etc. Campuses reported they wanted faster process time, and they wanted the raw data instead of pre-designed format. Live notes from the SUNY OpenCOTE Conterence, Syracuse, NY.

Social Media Planning and Evaluation for NGOs | Full Circle Associates

Nancy White

This topic is designed to give you some tools and ideas for including social media appropriately in your overall organizational strategic plan and to measure its effectiveness. These are examples of using quantitative metrics.

e-Clippings (Learning As Art): CNET/Webwares Top 100 Web Apps

Mark Oehlert

Assessment in MOOCs

Stephen Downes: Half an Hour

First of all, the MOOCs I have worked on have not focused on assessment - they have been courses, yes, with a small number (20 or so) taking them for credit, but the vast majority of participants auditing. And like you, I would not contemplate multiple-choice exams in humanities and literature courses. peer assessment - essays are graded not by professors but by other course participants. But the course score is secondary to the artifact itself.

Elevating Advocacy Voices for Children Through Social Media

Beth Kanter

Note from Beth: As part of my work at the Packard Foundation, I’ve had the honor of designing and facilitating a “networked capacity building project” for a cluster of grantees, state-based groups, to be more effective in engaging their networks toward covering uninsured children.

Supporting self-managed team learning in the organisation

Jane Hart

So it will not involve designing a programme of instruction for a team, but rather will comprise a quite different set of “ scaffolding ” activities undertaken in conjunction with the team, which include: Understanding the sharing practices that are currently taking place - or not.

Complicit Clients

Clark Quinn

I regularly rail against cookie-cutter learning design, boring elearning, etc. I like to blame it on designers who don’t know the depths of learning behind the elements of design, and perhaps also on managers who don’t work to ensure that the learning objectives are tied closely to meaningful business outcome. And I think that’s true, but of course there’s another culprit as well: clients who just ask for the same old thing!

Trends for 2019?

Clark Quinn

I think the main trends will be an increasing exploration of alternatives to ‘courses’ This will include performance support, and social networks. When we don’t really understand learning, we can be swayed by well-designed distractors. It’s already started!

Trends 191

What is stopping companies from measuring learning: Skillsets, datasets, toolsets or mindsets?

Xyleme

By adapting our language and perspective to integrate terms like traffic, bounce rate, conversion, time on site and social sharing into an L&D context, we create a dynamic system that uses data to drive smarter decision-making and draw on accurate metrics for more actionable insights.

Never Under-Estimate the Immune System

John Hagel

Every large and successful institution has an immune system – a collection of individuals who are prepared to mobilize at the slightest sign of any “outside” ideas or people in order to ensure that these foreign bodies are neutralized and that the existing institution survives intact and can continue on course. In more stable times, institutional immune systems are very effective at keeping institutions focused and on course, resistant to the distractions that might lead to their downfall.

System 100

Social ROI?

Clark Quinn

In engineering we might get less time to design a requested feature, and in manufacturing we might increase usable yield. Of course, if you can correlate them, all the better. Sure, we also might affect indirect metrics – retention, workplace satisfaction, or customer satisfaction – with tangible value, but our real focus should be on direct metrics. Ok, so I’ve been thinking about this, but not sure what the current state of play is.

ROI 138

Evidence-based L&D

Clark Quinn

There’s also evidence that there are principles on which to design learning that leads to better outcomes.

Using Social Media Effectively and Powerful Tactics Workshop

Beth Kanter

The session was designed a mix of strategy with a deep dive into content, measurement, Facebook, and Twitter. And, of course, it helps to have free stuff to give away to motivate people to share their stories. Social media metrics for nonprofits. Social Media Master Class.

working smarter case study

Harold Jarche

The university transitioned from designing processes for formal learning to increasing support for informal learning by: Establishing a learning & performance innovation team. From pushing training and courses, to workers pulling learning as required.

Platforms Are Not Created Equal: Harnessing the Full Potential of Platforms

John Hagel

Just to get the semantics aside, I tend to prefer the term “performance ecosystems” to describe this space rather than platforms simply because, to me, the term “platforms” creates an image of a much more tightly designed and specified environment than the term “performance ecosystems.” Of course, the flip side of this is that, until and unless the platform draws in a critical mass of participants, it will have only minimal value to participants. 

Personal Learning: A Set of Three Talks

Stephen Downes: Half an Hour

First admissions were open, then educational resources were open and now whole courses are open. This formed the basis for the original design of the Massive Open Online Course as a connected environment in which participants created and reused resources. Education is no longer ‘delivered’ (for free or otherwise) and instruction is no longer ‘designed’ in the traditional sense. Today, though, alternative metrics are being deployed.