Learning Measurement Means Getting into the Trenches

This weeks chat2lrn is about measurement and we are delighted to also include a ‘guest’ blog post from Kelly Meeker.

Measuring learning is and always has been a controversial and divisive issue.  Senior operational managers are used to hard targets and whilst learning professionals need and want some feedback on the impact of their efforts, the “smileys” approach and even end of intervention testing are now widely accepted to be of little use in assessing the real impact of learning.  However, measurement is a critical aspect of every professional’s work and if we don’t measure, how do we know whether the learning intervention has had any impact and delivered bottom line business benefits?

Kirkpatrick’s model was devised for face to face training, and some would argue that it is now outdated. It is also fraught with difficulty in its higher levels as performance improvement is rarely the result of a single identifiable intervention.  ROI is also a very contentious issue as to calculate an accurate ROI of learning from formal provision and prove direct cause and effect, all other workplace variables would have to stay the same.  It would require a ‘control’ group as well as an ‘experimental’ group, i.e. one group receives the formal learning provision (the ‘experimental’ group) and the other does not (the ‘control’ group). This is very often the model that is used during ‘pilot’ programmes, which if successful are then rolled out to a wider audience.

However, as we move so strongly towards a 70:20:10 model and recognise that most learning actually takes place ‘on-the-job’, does it mean that pilot programmes and establishing control groups are really only be suitable for formal learning interventions and if so, is it possible to measure informal learning?

It may be far better to look for measurements rooted in the day to day of the workflow.  What tends to work well is when line managers can clearly say that decisions made or actions performed would not have happened before the intervention. The key is therefore in choosing the metrics and choosing them well.  If you’re going to devote the time and energy to a learning programme, is it truly solving a problem that is important to your business?

Kelly Meeker aka @opensesame has this to say on the subject and suggests that we need to get into the trenches!!

Measurement is a challenge for learning and development professionals. Too often measuring learning outcomes falls into the pattern of sharing anecdotal evidence or only measuring production: “we’ve provided X resources” or “we’ve distributed Y widgets”.

Subconsciously, perhaps, developers like this kind of measurement because it measures only the outcomes that they can strictly control – what they do and make, day in, day out. What really matters for an organization, of course, isn’t measuring the number of courses the learning department produced, but measuring changed behaviors or outcomes.

This means L&D folks have to take a risk, and start measuring their own productivity by external factors. A successful learning initiative is measured by the change in behavior, situation or outcomes of the organization.

So what’s the challenge? First, identifying those desired outcomes – this can be harder than it sounds – and then identifying the incremental steps along the way to the desired end state. Second, assigning specific qualitative and quantitative values to both the baseline and the end state. This is probably just as hard as it sounds.

Theory of Change and Learning Measurement

The Theory of Change model is used by nonprofits and social change organizations to plan and target their programs. It also offers a helpful model for planning and measuring learning and development. This model supports productive change by forcing the developer to articulate a theory of change, or a model by which the desired outcomes can be reached.

The first step is beginning with baseline data that measures the current status or situation. The next step is to identify desired end outcomes – and the final and most powerful step is to create a model describing how your initiative will change that situation, and how. This puts huge goals into incremental, achievable steps – making the process simpler to understand and simpler to measure.

This, of course, is needs assessment. But it’s needs assessment with an open mind – that interests itself in more than just the traditional realm of L&D – and has a basis in data. Of course reaching agreement on all phases of this process requires group decision making, and that can be the biggest challenge of all. As Joitske Hulsebosch describes in this post on “Benchlearning”, it’s key to have an open mind, open discussion and avoid defensiveness on all sides.

The theory of change, once articulated, provides the metrics of your success. You will know you have succeeded in generating positive change once you can demonstrate the uptick in the metrics you planned to address.

Data’s Role in Decision Making

In summary, it’s essential to shift your focus from “What can I produce?” to “What can I change?” And those changes should be based on thoughtful analysis of the organization’s needs.

That means getting out of your office and into the trenches of your organization. Doing ride-alongs, observations and “undercover L&D professional” days. Be curious about what your organization does – and you’ll soon know where the gaps are. That’s the really valuable challenge for any knowledge worker.

Kelly Meeker is the Community Manager at OpenSesame, the elearning content marketplace, where she creates, curates and shares with the learning and development community. Find her on her blog at www.OpenSesame.com/blog, on Twitter (@OpenSesame) or at kelly.meeker@opensesame.com.

Finally, a question?

Beyond what point in time after an intervention can improvement or application be identified and measured?  For example, the airline pilot who learns an emergency drill in basic training but whose skill is only evident way down the line when something happens.

The transcript is now available for the chat……just look under transcripts and summaries.  Also Kelly curated the content using Storify you can find her summary in our Links and Resources section.

Are you supporting performance?

All organisations are measured by their performance.  Measures of success vary, but all successfully performing organisations have business strategies in place that allow them to survive and sometimes even grow during tough economic conditions.  As employees and businesses begin to leave the comfort of “things known” and venture or are forced out into the untrodden lands of uncertainty and unfamiliarity, their need for support increases and  our ability to deliver appropriate support is increasingly challenged.

In our now constantly and ever more rapidly changing world, the recognition that learning is ubiquitous is now widely accepted, as is the recognition that it cannot be managed. So what is the place of a learning expert or a learning function in this new world? What does support mean and how can it be provided?

Learning requires to be inextricably embedded in the work stream. At its simplest people need to have the support tools or “sidekicks” as they are described by Allison Rossett  and Bob Mosher to help them perform tasks. Such tools need to be incredibly practical and accessible. It is often forgotten that learners are in complex situations back on the job and regardless of the quality of the training, application is always more difficult when in the work stream and away from the protected environment of the learning intervention.

Supporting peoples performance in the workplace can and should happen using the whole span of technology and learning theory that is now so easily available to us.  Whether it is a simple checklist of codes at the self-checkout in a fruit and veg shop or a pilot’s complex pre-flight procedure, or maybe a simulation for a surgeon to practice a rarely used operation before treating a patient, the tool needs to be directly applicable and be focused on successful performance of the task and the achievement of desired results. Access to a co-worker skilled and experienced in the task or to an expert coach is another form of performance support.

At an organisation level, recognition of the predominance of informal and social learning needs to be given prominence and supported accordingly. All of us who work in learning need to concentrate fully on understanding the business sufficiently so that we can apply our knowledge to performance and improvement of results, whether at an individual, team or organisation level.

It goes without saying that human capital is an organisation’s greatest and most fundamental resource, therefore for an organisation to perform at its best, its members of staff need to be performing to the best of their ability.

The more we can support people out of an understanding of their own and their organisation’s needs, the more likely it is that a beginner will become an expert, a follower will become a leader, and stagnation will transform to innovation.

So how well are we doing? In 2010 Capita asked senior business leaders in the UKs largest firms how learning and development was contributing to the organisations ability to perform.   The outcome of the research is worrying with only 18% of leaders saying that L & D strategy aligns to the overall business strategy.  There may be many reasons for this, but we have to address them. A recent unpublished American survey revealed the horrifying statistic that only 15% of executives would recommend L&D to their colleagues as a resource in improving business performance. We are a very long way from delivering the added value to the people and organisations who employ us, both in their perception and in our aspiration. If we fail, we will become isolated and increasingly irrelevant overheads that organisations will no longer afford.  Succeed, and we become central to the future.

#chat2lrn on Thursday 16 February will seek to explore what performance support means in practice, and how as learning professionals we can position and skill ourselves to provide that support…..hope you can join the conversation!

To help you get ready for our next exciting session, have a look at the following posts………….