Learning Measurement Means Getting into the Trenches

This weeks chat2lrn is about measurement and we are delighted to also include a ‘guest’ blog post from Kelly Meeker.

Measuring learning is and always has been a controversial and divisive issue.  Senior operational managers are used to hard targets and whilst learning professionals need and want some feedback on the impact of their efforts, the “smileys” approach and even end of intervention testing are now widely accepted to be of little use in assessing the real impact of learning.  However, measurement is a critical aspect of every professional’s work and if we don’t measure, how do we know whether the learning intervention has had any impact and delivered bottom line business benefits?

Kirkpatrick’s model was devised for face to face training, and some would argue that it is now outdated. It is also fraught with difficulty in its higher levels as performance improvement is rarely the result of a single identifiable intervention.  ROI is also a very contentious issue as to calculate an accurate ROI of learning from formal provision and prove direct cause and effect, all other workplace variables would have to stay the same.  It would require a ‘control’ group as well as an ‘experimental’ group, i.e. one group receives the formal learning provision (the ‘experimental’ group) and the other does not (the ‘control’ group). This is very often the model that is used during ‘pilot’ programmes, which if successful are then rolled out to a wider audience.

However, as we move so strongly towards a 70:20:10 model and recognise that most learning actually takes place ‘on-the-job’, does it mean that pilot programmes and establishing control groups are really only be suitable for formal learning interventions and if so, is it possible to measure informal learning?

It may be far better to look for measurements rooted in the day to day of the workflow.  What tends to work well is when line managers can clearly say that decisions made or actions performed would not have happened before the intervention. The key is therefore in choosing the metrics and choosing them well.  If you’re going to devote the time and energy to a learning programme, is it truly solving a problem that is important to your business?

Kelly Meeker aka @opensesame has this to say on the subject and suggests that we need to get into the trenches!!

Measurement is a challenge for learning and development professionals. Too often measuring learning outcomes falls into the pattern of sharing anecdotal evidence or only measuring production: “we’ve provided X resources” or “we’ve distributed Y widgets”.

Subconsciously, perhaps, developers like this kind of measurement because it measures only the outcomes that they can strictly control – what they do and make, day in, day out. What really matters for an organization, of course, isn’t measuring the number of courses the learning department produced, but measuring changed behaviors or outcomes.

This means L&D folks have to take a risk, and start measuring their own productivity by external factors. A successful learning initiative is measured by the change in behavior, situation or outcomes of the organization.

So what’s the challenge? First, identifying those desired outcomes – this can be harder than it sounds – and then identifying the incremental steps along the way to the desired end state. Second, assigning specific qualitative and quantitative values to both the baseline and the end state. This is probably just as hard as it sounds.

Theory of Change and Learning Measurement

The Theory of Change model is used by nonprofits and social change organizations to plan and target their programs. It also offers a helpful model for planning and measuring learning and development. This model supports productive change by forcing the developer to articulate a theory of change, or a model by which the desired outcomes can be reached.

The first step is beginning with baseline data that measures the current status or situation. The next step is to identify desired end outcomes – and the final and most powerful step is to create a model describing how your initiative will change that situation, and how. This puts huge goals into incremental, achievable steps – making the process simpler to understand and simpler to measure.

This, of course, is needs assessment. But it’s needs assessment with an open mind – that interests itself in more than just the traditional realm of L&D – and has a basis in data. Of course reaching agreement on all phases of this process requires group decision making, and that can be the biggest challenge of all. As Joitske Hulsebosch describes in this post on “Benchlearning”, it’s key to have an open mind, open discussion and avoid defensiveness on all sides.

The theory of change, once articulated, provides the metrics of your success. You will know you have succeeded in generating positive change once you can demonstrate the uptick in the metrics you planned to address.

Data’s Role in Decision Making

In summary, it’s essential to shift your focus from “What can I produce?” to “What can I change?” And those changes should be based on thoughtful analysis of the organization’s needs.

That means getting out of your office and into the trenches of your organization. Doing ride-alongs, observations and “undercover L&D professional” days. Be curious about what your organization does – and you’ll soon know where the gaps are. That’s the really valuable challenge for any knowledge worker.

Kelly Meeker is the Community Manager at OpenSesame, the elearning content marketplace, where she creates, curates and shares with the learning and development community. Find her on her blog at www.OpenSesame.com/blog, on Twitter (@OpenSesame) or at kelly.meeker@opensesame.com.

Finally, a question?

Beyond what point in time after an intervention can improvement or application be identified and measured?  For example, the airline pilot who learns an emergency drill in basic training but whose skill is only evident way down the line when something happens.

The transcript is now available for the chat……just look under transcripts and summaries.  Also Kelly curated the content using Storify you can find her summary in our Links and Resources section.

6 thoughts on “Learning Measurement Means Getting into the Trenches

  1. Some interesting stuff in there, I have a few comments. Firstly organisational theory and Heisenberg suggest that you change what you measure. The law of unintended consequences applies here. As a MIS manager a few years ago, I provided statistics on staff student ratios, there was no need to tell department managers that this was important, simply providing a list departments and their associated courses with SSR’s in rank order in meetings was all that was needed.

    I agree that what you can’t measure you will struggle to improve, and its easer to measure activity and interactions (inputs) than outputs, harder but not impossible. Softer methods such as surveys, dip tests, opinion are all ways of estimating advocacy, the whole tipping point thing suggests that cultural and organisational change is driven through advocacy.

    While not everything important can be measured, you can always measure its influence and impact.

    Off to get my yardstick.

    • Thanks for your observations Richard. It was a very interesting debate which definitely had two sides! The one thing that everybody seemed to agree on, is that simply counting the number of courses, the number of people who attended courses and whether they liked them or not has little or no value and is certainly no indication of impact.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s