The Big Hole In The Big Data Talent Pool

Big data is a hot topic in the world of tech – for all sorts of reasons.

Largely, there is much excitement continuing to brew surrounding the subject. Not least because of some of the online tools that are now available, which mean that even SMEs can access at least some of the benefits that big data offers.

By this I’m primarily talking about things like marketing insights that can be gained from analytics tools like Google Analytics, Canopy Labs and Tranzlogic. These are fantastic tools that all online businesses should be using, for the insights that they give can provide a business with over consumer behaviour can be invaluable when IT comes to making predictions and planning future marketing strategies.

However, the reality is, that even if you’re fully equipped with every analytics tool going, if you’re not trained and qualified in the art of data science then there is only so much use that having such data will be.

To make the most of big data, the study, the science and the analysis of IT really needs to be your full time job – or rather, your business needs to employing a full time big data professional to analyse all the data that your company gathers for you.

The Shortage

Yes, the shortage. And I’m of course not talking the shortage of data – for big data is big. It’s bigger than big. It’s huge. Enormous. Perhaps even infinite.

No, I’m talking about the shortage of big data scientists that currently exist, and the general shortage of skills and skilled professionals in businesses around the globe. Indeed, although the analytics tools like those mentioned above are relatively simple to use and are very good for harvesting data – when IT comes to cooking up some meaningful insights and predictions with that data, there is a very large and conspicuous hole in the skills set.

The fact is that there is now so much data available to be analysed, that there simply aren’t enough people as yet who are qualified to make sense of IT all.

ACI.info produce some very interesting stats and a revealing infographic from DOMO about the amount of data that currently floats around the web waves on a daily basis.

Here are the key findings and the infographic:

Every Minute:

• Facebook users share nearly 2.5 million pieces of content.
• Twitter users tweet nearly 300,000 times.
• Instagram users post nearly 220,000 new photos.
• YouTube users upload 72 hours of new video content.
• Apple users download nearly 50,000 apps.
• Email users send over 200 million messages.
• Amazon generates over $80,000 in online sales.

According to IBM, 2.5 quintillion bytes of data are created every single day and there just simply isn’t the manpower to cope with IT all. In fact, in the US alone, we are looking at a skills shortage of between 140,000 and 190,000 big data professionals – so, if you fancy a career change, big data is hiring.

Data Scientists

There are other worrying reports from some very respected sources around the world. For example, Forbes notes that:

“there is still a significant shortage of skilled professionals who can truly be called Data Scientists who can evaluate business needs and impact, write the algorithms and program platforms such as Hadoop.”

And Gartner weighs in with:

“the need for data scientists [is] growing at about 3x those for statisticians and BI analysts, and an anticipated 100,000+ person analytic talent shortage through 2020.”

Indeed, such people with such coveted skills are in high demand, due to their unique skill sets in being able to unlock valuable pieces of business intelligence from a very large pool of unstructured data to give companies competitive insights over their rivals.

The data is used to determine everything from customer behavior to providing analysis on how the company should function in the future if IT wants to achieve sustained growth (and pay the salaries of the data scientists that make IT possible).

A report from The Guardian in February cites another factor that is contributing to the skills’ shortage – the fact that companies aren’t just after a single, talented individual, but a whole team of them.

“British Airways and other major companies recognize that getting the best from their data scientists, however, requires more than just hiring smart people and setting them loose to analyze data.

“They believe that data scientists are more effective and bring more value to the business when they work within teams. Innovation has usually been found to occur within team environments where there are multiple skills, rather than because someone working in isolation has a brilliant idea, as often portrayed in TV dramas.”

The Future?

Starting salaries for data scientists have gone north of $200,000, according to Bloomberg.com. If you want just one for your company – that’s the price, let alone a whole squadron of these “superheroes”.

But, thankfully, universities are stepping up. From the Bloomberg report:

“MIT, where graduate students in physics, astronomy, and biology are fielding offers from outside their chosen fields, is in the process of setting up a dedicated data-science institute. Marilyn Wilson, the university’s associate director for career development, says the center will begin enrolling graduate degree candidates in 2016.

“In the U.K., the University of Warwick introduced a three-year undergraduate data-science program last year, which David Firth, the program’s mastermind, says may well be the first of its kind. “Big Business was complaining about the lack of people,” he says. “Finance is a major employer, but also large-scale insurers, large online commercial retailers, high-tech startups, and government, which has huge data sets.””

Big data has arrived, and the world has decided that we need it. Furthermore, IT has decided that we need big data scientists to analyze IT all. No doubt one day, some such scientist will develop the software that will put an end, largely, to data science as a profession like we know (and pay for) IT now. But until then, the shortage remains, and while students currently study hard to gain the understanding that they will need to literally walk into full-time, high-earning employment, the big hole in big data continues to expand – and IT could be decades before we actually have the skills, software and manpower to catch up with it.