The Trouble with Tribbles: Traditional Smile Sheets. Lovable? Or Exponentially Dangerous?

Will bookSmile sheets, happy sheets, reaction forms, response forms, learner evaluations, level 1’s. The same thing, different names.

In a 2016 ATD survey, 88% of respondents reported that their organizations used smile sheets, yet only 44% said their learning measurement efforts were supporting their organization’s learning goals.

In two meta-analyses—studies of many scientific studies—traditional smile sheets have been found to be virtually uncorrelated with learning results, with correlations r = .09. That’s like correlating the daily number of my footsteps with the number of Patti Shank’s social-media posts. Not highly related, with the slight negative correlation being due to me being riveted by Patti’s brilliance, which keeps me from walking away from my computer screen!

Smile sheets are ubiquitous, but they are clearly not effective—in their current form—for giving us feedback about the success or weaknesses of our learning interventions. And, aren’t we, as learning professionals, sort of charged with ensuring that what we’re doing is working? Shouldn’t we get good feedback and make improvements?

Maybe we should just throw them out… On the other hand, there are many reasons besides getting good feedback to use smile sheets. From my recently published book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, I offer the following list, which I borrowed and modified from measurement expert Rob Brinkerhoff:

  1. Red-flagging training programs that are not sufficiently effective.
  2. Gathering ideas for ongoing updates and revision of a learning program.
  3. Judging strengths and weaknesses of a pilot program to enable revision.
  4. Providing instructors with feedback to aid their development.
  5. Helping learners reflect on and reinforce what they learned.
  6. Helping learners determine what (if anything) they plan to do with their learning.
  7. Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the training and/or the instructors.
  8. Upholding the spirit of common courtesy by giving learners a chance for feedback.
  9. Enabling learner frustrations to be vented—to limit damage from negative back-channel communications.

In the book, I focus on the first four—the ones related to getting good feedback. I wrote the book because I think we can create better smile sheets. Not perfect smile sheets! There’s no such thing as a perfect measurement tool, and in the complex world of learning, this is doubly true. But organizations will still use smile sheets, so if we can make them better, we should. Also, as the list above shows, there are other reasons to use smile sheets.

To create better smile sheets—better in enabling feedback—we have two imperatives. First, we have to ask questions that give us information related to learning. Second, we have to ensure that our questions give us results that are more actionable. When a course is rated using a traditional smile sheet at a 4.1, it causes two HUGE problems. First, it enables bias. There is no clear standard for whether a 4.1, 4.3, etc. is acceptable or not. So, we evaluate the number based on our biases. Second, these numeric responses also create paralysis within our organizations. Because we don’t know what a 4.1 means, we stick with the status quo, which too often means we stick with learning interventions that are not as effective as they might be.

For further reading before our chat, here is an article that describes what improved smile sheet questions might look like:


What can L+D learn from product management?

Today’s post is written by Holly MacDonald, #chat2lrn crew member and Chief Spark at Spark + Co located on an island off the coast of BC in Western Canada. Holly is an instructional designer, consultant, serial dog walker and a self-confessed whale nerd. Find her on Twitter @sparkandco.

According to wikipedia:

Product management is an organizational lifecycle function within a company dealing with the planning, forecasting, and production, or marketing of a product or products at all stages of the product lifecycle.

In L+D, we often focus on what goes into our instructional product (content), but less about WHO uses it, WHY they use it, WHEN they use it etc. We tend to think of our work in terms of “projects” not products, but what if we changed our perspective?

What if we developed instructional products? What lessons could we learn from product management?


Product managers are guided by the following principles:

  • Products have a limited life and thus every product has a life cycle.
  • Product sales pass through distinct stages, each posing different challenges, opportunities, and problems to the seller.
  • Products require different marketing, financing, manufacturing, purchasing, and human resource strategies in each life cycle stage.

Lessons for L+D

We could develop principles for our instructional products.

How do we develop instructional products to make maintenance or sustainment easier? Do we even consider that? Do we start a “project” thinking about it’s lifespan and how things might be different on launch than 2 years down the road? For our audience and for ourselves? Do we consider product roadmaps?


Product planning involves relentless focus on the customer – using tools like Customer Discovery – the product manager is always thinking about their customers and how to deliver their product to their customer segments. They often use techniques like the “Fuzzy Front End” – which is the conceptual idea stage of the product. Some also use the “Minimum Viable Product” methodology to test their design.

Lessons for L+D

This is analogous to our analysis phase, however do we ensure that we define our customer on every instructional product? Do we truly define the problem that our instructional product will solve? Are we focused on our customers? Do we understand that our customers and users are not the same?  Do we do an FFE? Could we adopt a Minimum Viable Product methodology?


Product managers know their competitors – who are yours? Who vies for your customer’s attention? They also scan the competitive landscape to determine what influences are happening: political, economic, social, and technological.

Lessons for L+D

What is going on in your “market” that you need to keep tabs on? Do you do any forecasting around external forces? Do we anticipate what our business/client is going to need in the future? Are we prepared to provide that?


Product managers of course spend a lot of time on producing their product. They use techniques like “design thinking”. Consider all of the things that are designed: teapots, cars, solar panels, chainsaws, electric cars, stand up desks (and a bazillion other things). Take the lowly door. Even doors can be designed in a way that’s right or wrong. A door that isn’t designed well is a “Norman Door”:

Lessons for L+D

Do we approach design in the same way? Do we look at the overall process of design from all angles: Can we resist the pressure to just jump in and start building? Do you have “safeguards” in place so you don’t build a “Norman Course”.


There’s been variations on the “Marketing Mix“, or the “P’s” of marketing – product, price, promotion, place, (and in some instances, 5 P’s, adding profit) and more variation for service businesses (adding physical evidence, people and process to the mix) and even more for more “digital products” for decades. However you think about it, product managers use a model for marketing their products.

Lessons for L+D

Could we use some of  the “P’s” for our instructional products or adapt them to instructional products?

We’d love to know what you think. What CAN we in L+D learn from product management? Come and join us on April 7th to share your ideas, insights, questions, challenges and concerns.

Device Agnostic Learning Design: Are You a Believer?

Today’s post is written by Andrea May, #chat2lrn crew member and Vice President of Instructional Design Services at Dashe & Thomson in Minneapolis, MN. Andrea is an instructional designer, project manager, wife, mother, Girl Scout troop leader and theater artist. Find her on Twitter @andreamay1.

device-explosionHow many different electronic devices do you interact with on a daily basis? My guess is that most L&D professionals will say at least two, perhaps a computer and a smart phone. Some will add several more to that total. Tablets, smart TVs, gaming systems, smart watches, e-readers, and many other devices have made their way into our daily lives. I’ve even seen wi-fi enabled refrigerators that can hunt for coupons and notify you when you are running out of milk.

Now consider the two or three devices you use most often. For many of us, those three devices are a computer/laptop, a smart phone, and an iPad/tablet. We use each device for different purposes based on its strengths and weaknesses. Your smart phone is ideal for dashing off quick texts, providing turn by turn directions to your destination, getting answers to questions while on the go, and even for making an occasional phone call. But the screen size is less than ideal for doing actual work or taking a course online. Tablets provide a nice hybrid for some tasks, but they don’t fit well in your pocket and it is still challenging to use it for the tasks that would seem best suited for a PC with its full keyboard and mouse and much larger screen.

But what if more content was designed in such a way as to be device agnostic? eLearning courses, for example, that work just as well on a smart phone as they do on a PC, could set users free to interact with that content exactly when and how they choose. The user could choose a device based on their preferences rather than on the needs of the content. Exciting stuff!

The trends are heading in this direction. So what do we, as L&D professionals, need to consider in order to design content that is essentially device blind? Join us this Thursday, March 24th 09.00 PDT/12.00 EDT /16.00 GMT on Twitter for #chat2lrn and share your ideas on enabling device agnostic learning content.


What skills do learners need to learn?

This week’s post is written by Lesley Price (@lesleywprice).  Lesley is a co-founder of the #chat2lrn crew and now, although supposedly ‘semi-retired’, she works part-time for Learn Appeal  and continues to love challenging and being challenged! 

Those of us who work in learning tend to focus on our own skills and recognise that the skill set we require is changing. We know that even delivering high quality engaging face-to-face training or e-learning will not necessarily improve either performance of those we want to support or have a positive business impact.

Those we support very often focus on what skills they will need to be better in their roles or on the practical skills they need to acquire. Management ask us to deliver all kinds of training.  The list is endless and varied – leadership, project management, health and safety, compliance, performance support and so it goes on.

But how often do we think about the skills that are needed to learn? Do we ever actually spend time helping those we support to develop learning skills or do we assume that because everyone has been through the education system that they know how to learn?    We also know that some learners are more ‘successful’ than others in terms of academic ability or in their ability to learn practical skills.  But what makes them ‘better’?  Is it some inherent part of their character or are some people just better at learning that others?

There have been rapid advances in the use of technology to support learning. Not just in terms of the way it is used in the workplace, but also that if people want to know about almost anything, they can probably find it somewhere on the internet.   However there is also a lot of incorrect information out there which is at best ill-informed and at worst just a hoax. Do those that we support have the skills to be able to tell the difference?  Do they need to learn another skill set?

So my question is, what skills do you need to have to be a successful learner, can they be developed and if they can, how can we as learning professionals help learners learn?

Join in using the hashtag #chat2lrn and discuss these and other questions on 10 March, 2016, 08.00 PST/11.00 EST /16.00 GMT.

The Learning Technology Ecosystem – are we there yet?

Today’s post comes from Fiona Quigley (@fionaquigs), chat2lrn crew member and
Director of Learning Innovation for Logicearth Learning Services.


The term learning technology ecosystem has been around for a few years now but I’m not seeing much practical application of it. It could be one of those newish ideas in our industry – we have to talk about it for a few years before we decide we can actually embrace it successfully (hello social learning and mobile learning!)

But after giving this some more thought, I now think the time is right for us to be thinking about a set of integrated tools, content and processes that should work seamlessly together to support organisational learning. eLearning (content) has been around long enough now for us to both want and expect more. And as for the LMS, well that debate will run on and on.

What I do know for sure, is that the trend to have the LMS more and more hidden, is a very real one. It seems that while other technologies have developed – curation tools, user generated content tools, enterprise social learning tools, not to mention the myriad of learning and productivity apps now available, the good ‘ole LMS, even with a shiny new talent management badge on it, remains quite traditional and dare I say ‘locked down’.

The new open systems – LRS, xAPI and LTI

If you are anything like me, you’ll probably have heard of LRS and xAPI, but you might struggle with LTI? This term was new to me until a few weeks ago; it stands for Learning Tools Interoperability. So think of third party rich content platforms like Khan Academy, TEDEd, Code Academy etc. These tools can be bolted together to allow single sign-on for all your staff. And no, this isn’t about formalising what some people would call informal learning. And it isn’t about tracking everything your staff does either. It’s about providing easy to access resources in a central location to facilitate self-service learning in its truest sense.

Instead of relying on procuring a complex and expensive LMS that could go out of date quickly, we think in terms of flexible bolt-on technologies. When used together, the sum of the parts is greater than the whole. What better way to get to self-service learning than to encourage your staff to use best in class curation or social learning apps that are part of your total learning technology ecosystem rather than just accepting what ‘comes’ with the LMS. Imagine following the high performers in your organisation as they curate the best content paths and share their insights in a way that could never compare to ‘click-next’ eLearning.

So there you have it – some initial thoughts from me on what a learning technology ecosystem could give us. Join us Thursday, February 25th to share your input.

Virtual reality: Can it change how we learn?

This week’s post comes from #chat2lrn crew member Ross Garner. Ross is an Online Instructional Designer at GoodPractice and a member of the eLearning Network. You can reach him on Twitter @R0ssGarner

Virtual reality is back – and this time, it works

If you’ve spent any time on Twitter in the past couple of months, or have attended any Learning and Development conferences, you’ll be aware that the industry is abuzz with the news that virtual reality (VR) is about to go mainstream.

Forget the crummy graphics of the 1990s. For the first time, VR seems like it’s about to live up to it’s name. Realistic visuals and surround-sound audio are creating an immersive experience that can finally trick your brain into believing you are somewhere else.


Woman Using a Samsung VR Headset at SXSW. Image courtesy Nan Palmero on Flickr.

Facebook, Sony and HTC are all launching headsets later this year, and Google Cardboard has made it affordable to try VR in your home.

pMeanwhile, companies like Magic Leap are raising millions in investment as they develop sophisticated augmented reality (AR) devices that combine simulated graphics with the world around you. Think Minority Report, or this YouTube demo.

But what does this have to do with L&D?

To quote blogger, speaker and #Chat2Lrn friend Donald Clark:

“In my 30+ years in technology I have never experienced a heat so intense and shocking as that I got when I first tried the Oculus Rift.

“As a learning professional, lots of applications flooded my mind. But more importantly, and this IS important, I thought of learning theory.

“The big problems in learning are:

  • attention
  • emotion
  • doing
  • context
  • retention
  • transfer

“This technology tackles these head on. We may be on the threshold of delivering educational and training experiences that are compelling and super-efficient, in terms of these positive attributes in learning.

“There’s also a bonus – this is a cool, consumer device that young people love. 2016 is only the start. VR is not a gadget, it’s a medium and a great learning medium.”

NATTC NAS Pensacola

U.S. Navy personnel using a VR parachute training simulator. Image from Wikipedia.

So what’s next?

VR is already used to train the army, pilots and surgeons, but what applications can you think of for VR and AR?

Is this going to be a technology that L&D grabs and exploits? Or will the cost and difficulty of implementation leave us lagging behind the entertainment industry?

Join in using the hashtag #chat2lrn and discuss these and other questions on 11 February, 2016, 08.00 PST/11.00 EST /16.00 GMT.

Catching the Wave

This week’s post comes from Dr. Greg Ketchum. Dr. Greg is a former clinical psychologist-turned CEO and media workplace and career coach. He presides over an executive talent firm, providing coaching and recruiting for executives and Fortune 500 companies. A unique mix of psychology and coaching expertise gives Dr. Greg a great understanding of people and what it takes for career success. You can find him at 
TalentPlanet® |, and on Twitter @drgreg. You can also find him in his KPIX & KRON TV Workplace & Career Segments on YouTube and his Workplace and Career Talk Shows from CNET & XM Satellite Radio.

Catching the Wave:

How to Catch the Wave of Massive Change Happening in Enterprise Learning and Development


Summarized from webcast delivered with Andrew Bateman at Human Capital Institute July 14, 2015

View the PDF of Webcast slides here: Catch the Wave or Get Left Behind

Rapid Development of Learning Technology:

The rapid development of technology enabled learning is driving major changes in how learning happens inside the organization and in the role of the Learning Professional. This evolution is causing learning leaders to confront and examine a number of questions about the future direction of learning within their respective organizations. Here are some of the questions that we will examine today.

  1. Traditional Vision of the Role of Learning: The traditional view is that the learning
    function assesses the learning and skill development needs of the organization and
    then either creates or acquires the appropriate learning resources to fulfill those
    needs. In this view the business leaders and their business goals are the main
    customers of the learning function.
  2. Emergent Vision of the Role of Learning: The emergent view of learning is that your employees are your main customers and that the role of learning is to help them maximize their potential. It’s best summed up by these quotes from Bridge’s

    1. “Rather than shaping your employees into a particular mold that achieves an
      organizational goal, L&D now focuses on helping employees reach and discover their potential — it’s about growth and personal aspirations.”
    2. “L&D now focuses on helping employees with personal growth and aspiration.
      Developments in collaborative, blended, and social learning help make learning
      interesting, engaging, fun and continuous.”
  3. What is Your Vision? Given these two starkly different visions of learning what is
    your vision of the role that learning plays in your organization now and in the
    future? Without clarifying your vision of the role and goals of learning in your
    organization both now and in the future no amount of dazzling learning
    technology will help you. This is the bottom line.
  4. You’ve Got Your Vision and Goals: Now What? Once you’ve nailed down the
    vision and goals of learning how do you make those key decisions on what kinds of
    learning technology, content, and programs you’ll deploy to drive your goals?

    1. The Fundamentals of Learning: With all of the advances in learning technology it’s easy to lose sight of the enduring fundamentals of learning. That is, how does the best learning happen, what role does learning play in each of our lives, why do we seek learning, do we learn better alone or in interaction with others, etc. These questions aim to look at how we’ve evolved as creatures and what role learning has played in that evolution. If we didn’t learn that certain plants are poison, or that lightning can kill you, or other things like that we would have never survived as a species. From this perspective learning is what has kept us alive as individuals and a species and that suggests that the drive to learn is as strong as any other native drive such as for food or shelter. This is a rich and deep area to explore: How do we learn? Why do we learn?
      • Darwin quote……“I believe there exists, & I feel within me, an instinct for the truth, or knowledge or discovery, of something of the same nature as the instinct of virtue, & that our having such an instinct is reason enough for scientific researches without any practical results ever ensuing from them.” — Charles Darwin.
    2. Mix of Modalities: What mixture of technology, in-person, classroom, social,
      self-directed, crowd sourced, casual, on the job, and other learning modalities
      will you employ and for which groups and individuals?
    3. Laying the Ground Work: Before you bring in any new learning technology
      what ground work do you need to do to ensure that the solution you’ve chosen
      is right for your organization and that you’ve set yourself up for success?
    4. Realistic Expectations of Learning Technology: What can learning technology do and not do? What’s realistic to expect as to the benefits both to the individual and the organization from technology enabled learning? What role does human interaction still play as an essential element in learning?
    5. Leadership Development, Bench Strength, Succession Planning: If learning is increasingly becoming an end user, self-directed exercise how do you achieve your goals for these three areas?
  5. The Big Question Remains – Engagement: Given all we’ve discussed, one of the
    biggest questions remains as to how you get your employees to engage with any of
    your learning solutions whether they be technology, classroom, person-to-person,
    virtual, or group based?

    • Our View: Building an engaged audience for your learning solutions starts with
      having compelling content, and with understanding the natural role that
      learning plays for us as individual human beings. Without either those building
      an engaged learning audience just isn’t going to happen.

So, what do you all think? Do you agree with these steps as a process to “catch the wave” of change?

Join in and discuss these and other questions during #chat2lrn this week on 28 January, 2016 at 08.00 PST/11.00 EST /16.00 GMT. Hope to see you there!