Science of Learning: When and How

2016-08-30_10-00-45The 2014 Association for Talent Development’s (ATD) State of the Industry report, shows organizations spending spent $1,208 on average, per employee, on training and development. Salas and the other authors of The Science of Training and Development in Organizations: What Matters in Practice, groundbreaking research using a series of meta-analyses, tell us that well-designed training is effective. It also says that the way we design, deliver, and implement training is what impacts the degree to which it is effective, asserting:

… (D)ecisions about what to train, how to train, and how to implement and evaluate training should be informed by the best information science has to offer.

Salas and fellow authors said reviews of training literature found many training efforts to be faddish, disconnected to the scientific literature, and lagging behind other sciences, with training programs implemented for inadequate and wasteful reasons.

Too many training myths still prevail. A 2008 Cisco whitepaper discussed the multitude of learning myths prevalent among learning practitioners. An entire book was recently published on learning myths.

Using research in practice aims to integrate scientific evidence with day-to-day practice in order to gain better outcomes. For example, carpenters may not know all the physics that goes into good practice, but they practice the science as shelving and framing would fall down if they didn’t.

What happens when L&D practitioners don’t practice the science of learning? Not practicing our science means our organizations and learners suffer and resources are wasted.

In this chat2lrn, we’ll discuss whether and how to use the science of learning in L&D practice.

References

C. Fadel. & C. Lemke. (2008). Cisco Systems. Multimodal learning through media: What the research says http://www.cisco.com/web/strategy/docs/education/Multimodal-Learning-Through-Media.pdf

Salas, E., Tannenbaum, S.I., Kraiger, K and Smith-Jentsch, K.A. (2012). The science of training and development in organizations: What matters in practice. Psychological Science in the Public Interest, 13 (2), pp. 74-101. 42. http://psi.sagepub.com/content/13/2/74.full.pdf+html

Tharenou, P., Saks, A., & Moore, C. (2007). A review and critique of research on training Psychological Science in the Public Interest-2012-Salas-74-101 http://www.celiamoore.com/uploads/9/3/2/1/9321973/tharenou_saks_moore_-_hrm_review_-_2007.pdf

R. E.  Mayer, R.E. (1997). Multimedia learning: Are we asking the right questions? Education Psychologist, 32, 1-19. http://www.uky.edu/~gmswan3/609/mayer_1997.pdfPsychological Science in the Public Interest-2012-Salas-74-101

Trends, Goals, and Professional Development…Planning for 2016

“Ajay is a Chartered Professional Accountant and a Certified Training and Development Professional but considers himself a Workforce Revolutionary. Ajay is a 3-time published author with John Wiley & Sons recently publishing his third book titled, “The Trainers Balanced Scorecard: A Complete Resource for Linking Learning and Growth to Organizational Strategy” (http://amzn.to/c3Qsk0). He is also a multi award-winning writer receiving the 2014 and 2015 prestigious TrainingIndustry.com Readership and Editors’ Award for Editor’s Choice and the Top 10 most read articles. Ajay regularly appears on the #1 Montreal Talk Radio morning show discussing workforce performance issues.”

Visit his (uncensored) workforce performance blog, Workforce Revolution

The 35th anniversary of John Lennon’s death just passed but his words remain…“So this is Christmas, and what have you done? Another year over. And a new one just begun.”

Yes. Another year is over and what have you done? If you are unable to fully answer this question don’t worry, you are not alone. The end of a year comes upon us quickly and it is often difficult to reconcile considering the promise a start of the year offers.

Each of us begins every year with renewed vigor and energy then in the end, which we believed was so long away, comes upon us with little notice. If you feel deflated from the passing of another year I have bad news…another one is around the corner. But here’s the good news. If you don’t want to experience disappointment the same time next year then do something about it now.

Moving Forward By Looking Back

The first step to achieving your end of year expectations is to first take a moment to reflect upon the past year. Learning only takes place if you are able to recognize what you can learn from past experience.

Begin by taking inventory of your experiences in the past twelve months. Human nature is to focus on the mistakes rather than successes. Both provide valuable learning lessons and it is essential that you not repeat the failures and attempt to leverage the successes.

When reflecting, take note of everything you observed/experienced in the past twelve months. What were the fads disguised as trends? What were the trends that meant well but never delivered on their promise? What did you learn from them? What experiences did you take away? Answering these questions is key to achieving your next twelve months objectives.

Goal Setting Isn’t Goal Achievement

Reflect upon the goals you set. Did you set any at all? Disappointment is only measured based upon the goals you initially set. Otherwise your disappointment is a waste of time…the only disappointment you experience is why you never set goals in the first place.

If you did set specific goals at the beginning of the year then measure how well you’ve achieved them. I assume you’ve achieved some to your complete satisfaction, you achieved others to the extent to say you met them (but not to your complete satisfaction) and then there are the remaining ones that you didn’t achieve at all. Focus first on the ones you didn’t achieve and then the ones you could have done better achieving.

If you failed to set goals then you are solely to blame for your end of year ineffectiveness. But don’t wallow in the disappointment learn from the experience. Set specific goals for the next twelve months but be sure to set them up so you can track your progress throughout the year. And, never make it a shopping list of goals. Plan too many and you will set yourself up for certain failure this time next year.

Professional Growth Is Not Optional

Finally, take care of your own professional growth. It simply amazes me how so many learning practitioners I come across don’t place any effort into their own learning. How can you not practice what you actually preach unto others?

The second error many practitioners make is to be myopic in their development. Stop focusing on what you actually do. While it is relevant to maintain your expertise it is equally important to develop holistically. To be taken seriously by your leadership, think about complimenting your skills with learning that lifts your value and expertise to be more inclusive of ancillary concepts and roles.

Give Yourself a Year-Round Present

Christmas is around the corner. Why not be kind to yourself. Plan your 2016 with learning events and opportunities by subscribing and/or registering for courses and conferences throughout the year. This will guarantee your professional growth and lead you to reflect favorably when 2016 comes to an end.

We hope that the past twelve months have been good to you. What I mean by that is not only in successful attempts but also if you didn’t achieve your objectives. You must be able to look back on both experiences to add to your personal growth. My wish for you is to take these experiences and leverage them for a more fruitful and exceptionally successful 2016.

Please share your experiences from the past year with the #Chat2lrn community. What are the fads or trends that you believe added value or were a waste of time? What was your wish list from last year that you achieved or wanted to achieve? What is going to inspire you to achieve your goals in twelve months? Join us on December 17th, 8:00am PDT, 11:00am EDT, 4:00pm BST for a #chat2lrn discussing these and other questions.

Allow me to leave you with a closing John Lennon thought and my sincerest wish to you that seems appropriate for this post:

“A very merry Christmas, And a happy New Year. Let’s hope it’s a good one. Without any fear.”

Please take this Poll:

Audience Analysis, Critical for Instructional Outcomes

Written by Patti Shank PhD, CPT
It may not reflect all of the chat2lrn moderator opinions.

Audience analysis is part of the need analysis process during instructional design. The purpose of audience analysis is to help us understand who we are dealing with (including the organizational system) and how to serve them most effectively.

What Happens During Audience Analysis?

Some of the things considered during audience analysis:

  • Target audience: Who is the target (primary) audience and any secondary audiences? What are their expectations and needs? What problems are they experiencing? What is their level of experience? How much will they participate? How much time do they have? How will we respond, with instructional and non-instructional interventions?
  • Environment analysis: The entire environment people operate in. Leadership, learning, performance, business, competitive, work, tools, the entire system…
  • Instructional analysis: What tasks are needed to learn? Do people all know the same thing? How quickly does the information change? Is this declarative or procedural information? Is this information that people need to memorize?
  • Technical analysis: Does this involve technology? Hardware/software?  Will it be changing? How does it tie into company infrastructure? Who will deal with the hardware and software? Does the audience have the ability and capacity to deal with the technology and keep up with it? Who will build and maintain it?

Why Should We Perform Audience Analysis?

In “The Science of Training and Development in Organizations: What Matters in Practice,” Eduardo Salas and his fellow authors proclaim that “decisions about what to train, how to train, and how to implement and evaluate training should be informed by the best information science has to offer.”

I write about the critical nature of needs analysis for good training outcomes, according to Salas and fellow authors research in my ATD Science of Learning Blog article, Science of Learning 101: The Latest Research on Needs Analysis and Learning Climate (https://www.td.org/Publications/Blogs/Science-of-Learning-Blog/2015/07/Science-of-Learning-101-the-Latest-Research-on-Needs-Analysis-and-Learning-Climate).

Below is Table 2 from the Salas paper (http://psi.sagepub.com/content/13/2/74.full.pdf+html?ijkey=g8tvuLmoeZfN2&keytype=ref&siteid=sppsi), which shows that needs analysis is the key factor for maximizing training outcomes before training.

Table 2

 

 

Below, Table 3 of the paper, the needs analysis items that are most critical are clarified.

Table 3

 

The very first item, Conduct training needs analysis, includes (emphasis is mine):

Determine what needs to be trained, who needs to be trained, and what type of organizational system you are dealing with.

Internal vs. External

Today’s post comes to us from #chat2lrn crew members, Meg Bertapelle & Holly MacDonald.

Meg is a Senior Instructional Designer of Clinical and Product Education at Intuitive Surgical, a medical device company which makes the da Vinci Surgical System. You can find her on twitter at @megbertapelle

Holly is the owner and principal consultant of Spark + Co, a boutique training company that provides custom training solutions to organizations for employees or customers. You can find her on twitter at @sparkandco  

Startup Stock Photos

Meg and Holly were chatting about the differences between internal and external L+D work, and captured some of their observations in this blog post.

What’s the biggest challenge you face as an internal L+D expert?

Meg: I would have to say that we run so lean sometimes, that our team isn’t able to really do our best work under the timelines & sheer number and scope of projects assigned to us. Always having to compromise on the best solution to get an OK solution out the door eventually gets exhausting.

What’s the biggest challenge you face as an external L+D expert?

Holly: Typically the biggest challenge is communication. Working with such a range of clients, some of whom are brand new to e-learning, others who are familiar with it means that we are constantly having to check assumptions, confirm things and keep those lines of communication open.

How do you deal with analysis as an internal/external?

Meg: Our fall-back position is always surveys and focus-groups, but sometimes the timeline of a given project just doesn’t allow for those methods, and we have to try to extrapolate information about the need from internal folks that work closely with the true audience. Our company just recently created a data analytics group that will work cross-functionally to gather what data we can directly from our products, and will advise on other ways to incorporate data gathering as learning experiences are designed and revised. I’m very excited about this because we might actually get real (not anecdotal) information about the gaps in our current materials and processes.

Holly:I think it’s easier as an external to do analysis, since you need to get information about the client and the learning need before moving ahead. I think as an external, you get more latitude to do an analysis. That being said, sometimes you find out that the problem is not a training one and those are not conversations the client always wants to have. But, if it won’t fix the problem, then they need to know.

What design challenges do you face as an internal/external

Meg: Usually time is my biggest challenge here. I would LOVE to be able to design tons of scenario-based practice activities; link directly to resources; provide everything our learners need in an easily-accessible, SINGLE place; and provide just-in-time and performance support for a truly flexible and end-to-end solution to all of our challenges. It just ends up being impossible while also keeping up with the project load on our team.

Another big challenge for us is that in order to meet deadlines, especially for product-related training materials, we have to split up design & development work between team members, and then struggle with the lack of consistency in the end result.

Like Holly, we have to adhere somewhat to the company brand guidelines, but thankfully (!!) more of the general feeling rather than the “letter of the law.”

Holly: Either the constraints of the “brand guidelines” where the client’s marketing team has decided to apply branding rules to elearning. This can really mean that you aren’t able to get as creative as you’d like. I usually try to find out if there’s a way to adapt the brand guidelines to elearning. To be honest, if not, then I’d actually consider walking away. If the branding overshadows the need to learn, then it can actually be an indicator of an organization that really doesn’t value learning.

The other common constraint is that the budget is not big enough to get custom design assets, so you head into your digital closet to see what you’ve collected and stockpiled over the years to use on the project. One other aspect that I’ve found challenging is to source great designers who get instructional design and/or elearning. I have found a few who kind of get it, but there is sometimes a tension around which designer knows best.

What implementation challenges exist as an internal/external?

Meg: Managing the different permutations of products released where & when – what system, what software version, where is it cleared, where is it launched, in what language… (did you hear that? it was my head exploding)

Holly: The LMS. That’s the biggest challenge we’ve faced with the implementation. Some clients engage us to work on their launch plan with them, but sometimes we hand it off to the LMS Administrator or IT department and that’s the end of it.

What do wish you could do that you can’t as internal/external?

Meg: I think I would love to be able to say “no” to a project that I just don’t want to do. LOL 🙂 Honestly, since my biggest constraint is usually time, and I imagine that’s not that different in an external role, I’m not sure what else to wish for! Hopefully some of you in the chat will give me some good ideas that I can try to make happen internally 😉

Holly: I have been an internal before and I think the thing I miss the most is the ability to modify the program once it’s launched, or having a more flexible role to extend the program. As an external, you live and die by your scope and once the program is launched, it’s gone. We’re very lucky to have long term relationships with our clients, so we do get to do some of that with them. But, for some it’s a budget decision.

What do you think you could teach internal/external?

Meg: I have a lot of “tribal knowledge” of our business, so I think I could help an external person come up with a solution that would fit our organization, and make a business case for it. Sometimes the things that matter to the organization are not as visible to someone external.

Holly: After doing this for so long with many different clients, I think the thing I’ve really mastered is how to understand a client’s business quickly. I get to use my “ignorance card” constantly and coming at things from the perspective where you know little or nothing means you have a unique point of view. I have one client who often says things like: “I love how you make us think about things in ways we haven’t thought before.” When you are internal, it’s much harder to maintain that perspective. You need to find ways to do that consciously, otherwise you just end up making assumptions.

What about you? What have you found to be the benefits and challenges of being either an internal or external learning expert?

Let’s discuss during #chat2lrn on Sep. 10th, 8:00 PDT/11:00 EDT/16:00 BST

Hope to see you there!

Total Cost of Ownership – What is the ‘real cost’ of a learning intervention?

This weeks post is written by Lesley Price (@lesleywprice).  Lesley is a co-founder of the #chat2lrn crew and now, although supposedly ‘semi-retired’, she works part-time for Learn Appeal  and continues to love challenging and being challenged!  Lesley is Scottish and the Scots have a reputation for being ‘canny’ with money…so her challenge to you is: Does the Total Cost of Ownership of a learning intervention really matter?

When we buy a car, some folks only look at the purchase price, others may also consider obvious running costs e.g. insurance, road tax, petrol consumption. Some may take into account the cost of servicing and replacement parts, but I wonder how many factor in depreciation cost and how many years we expect to have the car before we replace it? I have yet to meet anyone who does all of this, puts the information onto a spreadsheet and then calculates the cost of having the car over a number of years. If we carried out this exercise prior to purchase, would we be able to work out which car would offer us the best value for money and the optimum time to replace it? Logic would say yes, as we would then know the total cost of ownership.

IcebergSo what has buying a car to do with learning?  I would suggest that as the picture says, ‘what we see often is only a fractional part of what really is’.   So the question I ask is, ‘what is the real cost of a learning intervention?

All too often we only consider the cost of the course itself or the purchase cost/license fees of either an LMS and/or a content authoring tool, but what about the other ‘hidden’ costs? Do we even know what these are?

When we consider face-to-face training, these are relatively easy to calculate, or are they? If we send somebody on a course that is held elsewhere, there is generally a flat fee, but do we include the cost of the attendee’s time? We are told on a regular basis that time = money, so if we expect colleagues to disseminate what they have learned during the course, how much does that cost both in terms of their time and the time of others who are learning from them?

If face-to-face training is ‘in house’, what is the cost? Should we include the trainer’s delivery time, the time the trainer has spent on creating learning materials, the time of all those who attend the course, the cost of the space used for training which takes place on the premises or might the training involve room hire?

This becomes even more complex when we move into elearning. Yes, we think about the number of licenses we need, but do we consider whether we will need more IT equipment? Most people would say ‘yes of course we do’, but if the system needs a dedicated server what is the cost of IT support of both the software but also equipment?

If we are offering an elearning programme that we are going to create, how do we put a cost on that? We have to consider the time it will take to create…that’s easy….it’s the cost of an Instructional Designer (ID). Mmmm….. but most IDs refer to subject matter experts (SMEs) to ensure the content is fit for purpose and that then takes up the time of the SME and how many do we need to consult?

The other thing we know about elearning is that we cannot assume that just because we create and build systems and content that people will use it. So we have to generate interest and awareness otherwise all the time that has been spent creating the elearning content will be wasted, but that also takes time and to reiterate time = money!

Let’s not forget that to implement a new system; we will need the support of the senior management team (SMT). How many of us factor into the cost of the intervention the number of meetings we have attended, on-going conversations and reports we have written to get SMT buy-in?

Ooohhh and lets not forget all the conversations we have ‘out of hours’ with colleagues and pondering we do ‘in our heads’ about whether or not the learning intervention we feel passionately about will make a difference.

I guess the ultimate question is so what? Does the Total Cost of Ownership of a learning intervention really matter? Do we really need to know the real cost and if we do, what impact does that have on whether we proceed or not? So many questions and probably even more answers. Join #chat2lrn to share your views and thoughts on Total Cost of Ownership #TCO Thursday 30 July 8.00 PDT/11.00 EDT/16.00 BST

Is This a Training Problem?

by Patti Shank, PhD

I was very lucky when I was a young training manager and had the opportunity to learn with Geary Rummler (http://www.performancedesignlab.com/geary-rummler-founder). I truly believe that this training greatly helped my performance over the lifetime of my career. It provided a certain way of doing my work. The resource I will share with you will provide a brief synopsis of some of the thinking involved that I hope will intrigue you.

Why Care About This?

Training is an expensive intervention. We only need to provide training for one reason: People need skills they don’t have (or need to upgrade or re-establish their skills) and it makes sense to provide it in a formalized way.

When there are problems, such as people unable to do their jobs because of inadequate tools or not enough feedback about whether they are performing as needed (no performance standards), those problems must be fixed and training won’t solve the problem.

Example: A manager asks for team training for her staff because they are don’t work well together. In reality, she causes problems among them by how she treats them. She favors some over others. She provides more work and overtime to people she doesn’t like as much. Training might help this but she is the one that needs it. And before that, she needs coaching about the problems she is causing so the training might be valuable to her.

When we get requests (or demands) for training and we don’t determine if training has a good chance of solving the problem (or being part of the solution), we are creating a problem, not solving it.  Why?

We are using resources that could be better put elsewhere.

We are removing people’s time (when they are stuck in training) that they could be using towards better purposes. They could be using that time to get work done.
The problem doesn’t get fixed. (Think of all the resources used to not solve the problem!)
We look foolish and are unprofessional, and frankly, this happens too often. Who would hire a carpenter who couldn’t measure or build the right solution?

How Training Doesn’t Work

Example: When someone asks for customer service training but they have insufficient tools to answer customer questions or their process requires multiple workarounds, adding customer service training is a misplaced and expensive intervention.  They may need some training (or not) but they DO need better tools and an improved process so customers aren’t angry about being put on indefinite hold or sent to the wrong department.

Carl Binder’s discussion of Gilbert’s Six Boxes is a great introduction to thinking about what we need to do to have the type of performance organizations need and what influences these performance outcomes in the workplace. Read it and think about what part each part plays in your work. If you don’t think it fits in L&D’s world, we’ll have to disagree.

The Six Boxes: http://www.binder-riha.com/sixboxes.pdf

Much Ado About Microlearning

MightyMouseMicrolearning (1)Much is being made of the concept of microlearning these days, and perhaps rightly so. Microlearning products and collections, assembled and offered by learning and development organizations, fit into available time slots and busy work schedules. If available on mobile devices, they can also be used in performance support applications at the time and place of need.

From the producer’s perspective, they are also relatively quick to produce, and both easier to create and maintain then their larger, more complex e-learning counterparts.

But microlearning is not new at all. Countless how-to videos on YouTube have helped millions of people repair appliances or learn to better perform tasks or even hobbies. More interestingly, most of these products were created by people with no instructional design background, and yet we learn effectively from them.

So how can learning and development organizations use microlearning products to meet the needs of organizations? What can we learn from YouTube to encourage the participation of large numbers of employees? Discuss this and more about microlearning products in learning and development at #chat2lrn Thursday, 09 April at 16:00BST/11:00EDT/8:00PDT.

The Business of Learning Evaluation

#Chat2lrn is delighted to have a guest post from @AjayPangarkar.  Ajay M. Pangarkar CTDP, CPA, CMA is founder of CentralKnowledge.com and LearningSourceonline.com. He is a renowned employee performance management expert and 3-time author most recently publishing the leading performance book, “The Trainer’s Balanced Scorecard: A Complete Resource for Linking Learning to Organizational Strategy” (Wiley 2009), award-wining assessment specialist with Training Magazine, and award-winning writer winning the 2014 prestigious TrainingIndustry.com Readership and Editors’ Awards for the Top 10 most read articles. Help him start a, “Workplace Revolution” at blog.centralknowledge.com.

Learning practitioners are under tremendous pressure from business leaders to demonstrate that their learning efforts and initiatives are worth the budget they allocate. This has to be one of most daunting challenges facing those involved with any aspect of workplace learning.

There are many reasons why learning practitioners are unable to connect their efforts with actual workplace applications. One that stands out is that learning practitioners focus on the “learning” rather than on how learning “results” impact business performance.

Reality Check

Learning practitioners like to talk about being ‘accountable’ but behind the talk is an unfortunate reality where, like the three monkeys, this pesky ‘accountability’ issue will go away if we do not speak, see, or hear it. What learning practitioners really want to say to business leaders is, “Leave us alone to focus on the learning and stop bothering us with your trivial business issues!”

Regretfully, many learning practitioners remain under the impression that if proper learning takes place then everything else will take care of itself. Intuitively, this makes some sense but this causal relationship is too weak to be effective. Following this logic is the same as saying that, if you eat ice cream you’ll be cold; possibly, but there are many other reasons that also apply.

“If proper learning takes place then everything else will take care of itself is similar to saying that if you eat ice cream you’ll be cold; possibly but many other reasons also apply.”

Those involved with learning discover early to integrate and apply Kirkpatrick’s Four Levels of Evaluation. Yes, your organization explicitly hires learning practitioners for their expertise with level 1 (develop effective learning) and level 2 (learning retention). There isn’t one business leader that expects anything different. What’s more, however, is that they also expect their learning practitioners to ensure that the first two levels contribute to improving job performance (level 3) that will lead to business improvement (level 4).

Wolf In Sheep’s Clothing

Here lays every Learning Practitioner’s challenge…getting employees to learn the right skills and, ultimately, apply these skills to the job, again, ‘accountability’. In an attempt to answer this need, there are those proposing what appears as relevant solutions to this dilemma including measuring learning’s “return on investment” (training ROI) and how well learning meets business expectations (ROE). Again, the shortsightedness of these methodologies is just like the analogy of “ice cream making you cold”. The causal relationship is too weak to prove and too often inappropriate or irrelevant.

These solutions fall short to actually measure and evaluate how well learning contributes to on-the-job effectiveness and its role to achieving business objectives. With a growing need for innovation, creativity, and managing continuous market changes, business leaders are also under tremendous pressure to foster a knowledge-driven business environment. Leaders are increasingly depending on organizational knowledge to develop a strategic and business advantage that will help them to maintain relevance, let alone survival, within their market space.

“Rather than being viewed as a secondary role, workplace learning has quickly risen to the top of many business leaders to-do list.”

Rather than being viewed as a secondary role, workplace learning has quickly risen to the top of many business leaders to-do list. Furthermore, even though this is a learning practitioners dream, it also comes at a price…the need for accountability. So, what should learning practitioners do? How can they prove that their learning efforts actually improve employee and business performance? Is there anything currently available that works?

Let’s discuss these and other related questions to measuring and evaluating workplace learning impact at our next online gathering of #chat2lrn, Thursday 27 November at 16:00 BST / 12:00 EDT / 09:00 PDT. Come prepared, we look forward to seeing you!

How Do We Do It? Crafting Decision-Making Practice in eLearning

Today’s post comes to us from #chat2lrn crew member, Meg Bertapelle. Meg is a Senior Instructional Designer for Clinical & Product Education at Intuitive Surgical. You can find her on twitter at @megbertapelle.

Decision-makingI think most of us would be happy to build more scenarios and practice decision-making activities into our elearning projects (time permitting) – if we knew up-front how to plan and execute them. Sometimes the hardest part is knowing where to start.

This week, I’ve asked a couple of our #chat2lrn community members to share their experiences crafting decision-making practice elearning activities. Fiona Quigley (@FionaQuigs), one of our #chat2lrn crew,  is Head of Learning Innovation at Logicearth, an Irish learning services company with a global client base. They specialise in the production of modern multi-device elearning content, learning technologies and training support services. Laura Payette (@ljwp) now works at Nielsen, but is coming off a three-year stint designing and developing elearning and corresponding product/marketing communication for KPA, a dealer services and internet marketing provider for over 5,000 automotive, truck, and equipment dealerships and service companies across the US. Her DOT Hazardous Materials course won the National Excellence in Training Award from the Automotive Training Managers Council in 2013.

We’ll get some great information from them in the form of an interview, then we can all discuss the pros & cons, and crowd-source some suggestions during the chat on Aug. 7th. I hope this will help us all take the initiative to help our audiences start applying their new knowledge & skills right away.


Q1: What kinds of decision-making practice activities have you been able to incorporate in your elearning projects?

Fiona: I’ve designed a lot of content for health and social care professionals, especially in the area of communication skills and policy compliance. I have designed scenarios that present a typical patient interaction and then ask the question – what would you do or say next? That would be what I call a level 1 decision making scenario.

Higher level scenarios are more immersive and instead of leading a leaner down a pre-planned path, they include random events that differ each time you ‘run’ the scenario. I’ve used these higher level scenarios with nurses and pharmacists in areas such as medication management – reducing errors and also for dealing with patient complaints.

Laura: I spent three years building elearning for the automotive industry, particularly in the areas of environmental safety and compliance. There was a lot of regulatory information that had to be included and I was constantly challenged to find ways to make it relevant and interactive. One of the ways I did that was to inject scenarios into the training. Keep in mind that many of them were on the smaller side. In other words, I didn’t build a course around one big scenario with a million branching options (although that would’ve been so cool!). My content simply didn’t lend itself to that. Instead, I used smaller scenarios and sprinkled them in where they had the most impact.

Q2: What kind of planning steps did you take before beginning to write the activities?

Fiona: You must talk to real people who do the jobs. Observing people making the real decisions is the gold standard – but it is often difficult to get the opportunity to do this. You need to be careful who you chose as the Subject Matter Expert is. Often SMEs are senior ‘expert’ people who are very far removed from day-to-day practice. To help people practice real decisions you must talk to the people who make the everyday decisions.  I also like to structure conversations with SMEs into what I call a ‘DIF’ analysis:

  1. Difficult – what if anything, do you find difficult about this decision
  2. Important – what is most important about getting this decision right/wrong?
  3. Frequent – what frequently comes up, e.g. common myths/misunderstandings, good practice?

Often competent practitioners won’t be aware of how they make good decisions. They are unconsciously competent; so it is the job of the ID to turn this tacit knowledge into explicit learning. Once that learning has been made explicit, you can more easily share that with others.

There is also a difference between formal and informal practice. There may be formal rules in place about how someone does their job – but many competent practitioners create shortcuts as they gain experience. Being able to identify these ‘tips and tricks’ is very useful learning in itself.

Finally, I would also advise talking to people at various levels of experience. For example, talking to a novice in the area will help you see the challenges first hand, rather than relying on the recall of someone more senior who may gloss over these challenges.

Laura: Research! Obviously, reviewing content from SMEs and talking to SMEs is critical but, like Fiona said, talking to people on the frontlines — or who at least aren’t far removed from the frontlines — really helps build context for understanding the challenges that employees face in doing their jobs. Sometimes that access can be hard to get; it was for me. If that’s the case, use everything you can to tease it out. Think of yourself like an investigative reporter. In my case, I had access to a robust database with thousands upon thousands of real-life examples that had been logged. I also had access to people who could elaborate on those examples to help fill in the gaps. I relied heavily on them and went back many times to ask additional questions.

Q3: How did you determine appropriate activities that would simulate the real-life application of your learning objectives?

Fiona: Again – much like the answer to Q2, observe decisions being made, find out how people actually make the decisions and base the activities on what actually happens in the workplace – not what SHOULD happen. Too often in elearning, we are forced to idealise and formalise the learning process, which then becomes so far removed from reality that it loses credibility with the target audience! You often see this in elearning content where the scenarios are so easy that you don’t actually need to complete the course to be good at them.

For example, when we designed a Medications Management programme, quite a few of our nurses said that one of the most difficult challenges they had was doing the ward round, handing out medication and being interrupted by patients or family members. They said they needed to concentrate and focus on making sure they gave out the correct medication – often a complex range of drugs for patients with very different medical needs. Another source of concern was worrying about patients who found medications hard to swallow and not having enough time to spend with them to help them. Together, we came up with guidelines about how to resolve these challenges and built in a scenario challenge around this.

Laura: The scenarios that I wrote were usually an outgrowth of the content development process. In other words, I didn’t approach a course with a specific scenario in mind. I generated them organically as I pulled the content together. It becomes apparent in talking to SMEs and frontline employees and in reviewing existing content where the gaps in understanding and practice are. Those gaps were usually the places I chose to insert scenarios because they illustrated the performance issue and allowed employees to think through things by answering the questions. In some cases with my content, there were right and wrong answers (remember, a lot of it was compliance based), but there were also usually shades of gray — and it was in those areas that I was able to challenge employees through scenarios to think about their actions and the ramifications of them.

Q4: How did you evaluate the effectiveness of your activities?

Fiona: I normally ‘dry run’ the decision plan with a selection of the target audience in a focus group setting. It is important to have a range of different people with different levels of experience. Role-playing the scenario, trying it on for size, works well to see if it is a realistic enough representation of the actual day-to-day-job. I normally use simple post-it notes to visualise the decision and focus on:

  1. Decisions – what is the actual decision to be made?
  2. Knowledge/Skill – what knowledge or skills do you need to make the decision?
  3. Actions – what specific actions do learners take to make the decision?
  4. Consequences – what are the results of each action, for both good and poor decisions?

Laura: I ran the activities by stakeholders and SMEs, as well as a core group of what I’ll call advisors for lack of a better word. (They were internal employees who interfaced directly with the external employees I built training for.) If they responded by effectively saying, “Oh, that really made me think about things differently,” or, “That really caught my attention,” then I knew I had hit the mark. If they didn’t, or if they were confused by what to do or how to respond, then I knew the scenarios needed more work. I know that’s vague, but there’s really no set recipe for scenario building; it’s very context specific. I also evaluated the activities by looking at actual evaluation responses from employees who took the course once it was deployed.

Q5: What made certain activities more effective/impactful than others?

Fiona: The more realistic the decision and scenario – the closer it is to the learner’s actual normal workplace activities, the better. Not only does the decision need to be realistic, but so does the consequence. We don’t want to use phrases like “Well done, that is correct” – rather, we need to show actually what happens in the workplace.

We have a challenge in elearning in that we usually have to design for a very generic audience. That means we lose the nuance and subtlety that actually drives high-performance. If you look at what drives and helps people to perform at a high level, it is mainly about understanding the subtlety of communication that goes on around you. It is also about reacting to unexpected happenings – like covering for a co-worker or working when you are understaffed.  We need to make sure we build in this nuance and realism.  To do this well, we perhaps need to have different types of scenarios to suit different types of people in our target audience. As learning designers we just can’t go on accepting a once-size fits all approach to our learners.

Also – a by-product to this analysis is that you need to be open to the fact that not all challenges that you uncover will be solved by training. For example, for our nurses, we identified that adding a simple “Drugs round in progress” notice to the drugs trolley, helped to reduce the interruptions staff faced. Identifying these possible environment or process problems is a great benefit of doing good decision making analysis. If you explain this to your client upfront, it can also be a great motivator for them to really engage with you.

Laura: Fiona makes some very good points here. I think including real consequences to real situations, and writing them in the parlance your target audience speaks, is key. If you fabricate your scenarios they won’t be authentic and people will dismiss them. They also have to be contextually bound. In other words, you may see a great idea for a scenario somewhere and think, “I’ll put that in my course!” But if you don’t mold it for your audience/content and their specific performance needs, it won’t be a great scenario for what you’re building. I think sometimes the scenarios that are most impactful are those that address gray areas — the places where employees are a little uncomfortable or uncertain — and the places where the biggest performance gaps are.

Q6: Please share your top tips/tricks for crafting decision-making practice activities.

Fiona: I think I have covered most of these in answering the questions above, but to summarise:

  1. Talk to real learners of different levels of experience.
  2. Be aware of the formal way of doing something versus the informal way.
  3. Help your SMEs make their decision making practice more explicit by asking good questions.
  4. Have a range of scenarios to suit different types of people in your target audience.
  5. Dry run your scenario plan with representatives of the target audience and adjust accordingly.
  6. Find out why people are making common mistakes e.g. is it a process or environment problem rather than a training problem?

Laura: Fiona’s tips are great. The only thing I’d add is be sure to craft your scenarios in the language your target audience speaks so they sound authentic.

Thank you to Fiona and Laura for sharing their insights. What about you? If you have some experiences and insights to share, or just want to hear what others may have to say, please join us Thursday, August 7th for #chat2lrn at 8am PDT, 11am EDT, 4pm BST.