Size doesn’t matter …

The #Chat2lrn Crew are delighted to welcome a post from Owen Ferguson (@owenferguson) Product Development Director at GoodPractice. Owen gave a presentation on this topic at #LearningLive last month: Data – The Final Frontier.  You can see Owen’s presentation here.

Big Data is definitely one of the buzz terms of the last couple of years. Marketeers are excited by it and the enterprise IT giants are salivating over the opportunities to make money from it. But, it might not be a good idea to jump onto the Big Data bandwagon just yet for a number of reasons.

In general, there’s so much we can do with the normal data sets we already collect. However, until we learn how to get to grips with these smaller data sets, we’ve not really got the competence to truly handle Big Data.

Getting involved in Big Data right now, for most learning departments, is like attempting a copy of Michelangelo’s David as your first sculpture project.

There’s also a case to be made that the tools to handle Big Data aren’t really mature enough to take full advantage of it yet. Just look at where Gartner place Big Data on their Hype Cycle for Emerging Technologies and draw your own conclusions.

Gartner's Hype Cycle for Emerging Technologies

Gartner’s Hype Cycle for Emerging Technologies

That is not to say we can safely ignore Big Data. It’s in the post, and it’s going to transform many aspects of how business works. Not being prepared for Big Data when it starts to get truly useful is to condemn ourselves to obscurity.

So we need to get ready for Big Data, but we don’t do that by starting with massive, complex data sets. We get ready for it by improving how we work with small data – and small data isn’t that small.

Its easy to overlook how vast the quantities and varieties of data we’re talking about when discussing Big Data – mind boggling amounts. Petabytes. Zettabytes. Working with that volume of data makes a lot of sense – in fact it’s necessary – if you’re working in the realms of economic research, public policy or large scale web companies. But I don’t think we need to worry about that too much yet – most of the data we need to work with can be handled by a PC with basic software.

Microsoft Excel has a limit of 1,000,000 rows by 17,000 columns. That’s 17 billion data points!

Admittedly, you do need a powerful machine to carry out calculations with a data set that size, but you can run dozens of websites visited by hundreds of thousands of users on a single server. It doesn’t require cloud computing.

My team and I regularly carry out analysis on a user base of over 250,000 unique users, armed with Microsoft Excel and some basic statistical techniques. For some more of the very large data sets, I might use a more powerful computer (I nick one of our developers for an hour) – but one that I can sit down and touch. We hardly ever need to go to the cloud for our data crunching needs. The difficult part of the analysis isn’t the tools, it’s not the techniques (those have been around for over a century), it’s defining the right questions, and gathering the right data.

Mastering ‘small’ data as a profession is essential if we’re to really take advantage of ‘big data’.

How we do that is the significant challenge we face today. The truth is most people working in L&D don’t like data. Not really. Not the way that epidemiologists or baseball fans love data.

You never hear the phrase, “let’s carry out a linear regression to check that” in L&D departments. You don’t hear disagreements about the p-value or correlation coefficient calculation for some finding. You don’t even hear very much discussion about possible approaches to data collection and analysis.

That’s not to knock L&D – our profession is in the company of a lot of people. It’s also an exciting opportunity, because if we can get better at working with data, we can make an even greater contribution to people’s working lives and performance.

One place we can start is to learn the lessons other professions have learned about working effectively with data. Here’s Tim Harford with some advice to get you thinking:

You may also be interested in reading more about Big Data in a White Paper written for the Learning and Performance Institute by John Mattox, Director of Research at KnowledgeAdvisors – Big Data – Why it Matters to Learning and Talent.

Hope you can join us to chat about Big Data:  08.00PDT/11.00EDT/16.00BST on Thursday 15 October, 2013

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s