Suite 1, 25 Westerton Road, Glasgow, G68 0FF


November 13, 2012

Big Data Analytics

“Big Data” may sound like a character from “Star Trek: The Next Generation” after a particularly large lunch, but, in fact, it is a new and rapidly expanding business market.


Big data is defined as “datasets whose size is beyond the ability of typical database software tools to capture, store, manage and analyse” (MacKinsey Global Institute). Large corporations and public sector organisations holding data on millions of consumers are seeking new ways to generate profitable management information from that data efficiently – that is Big Data Analytics.


Leading businesses have always known that they are competing on how well they understand their data, and the information age means that large corporations and public bodies are increasingly having to handle massive amounts of consumer data. They are seeking new ways to manipulate and use this data to provide goods and services efficiently and generate new business. The ability to generate useful management information from such vast quantities of data is rapidly becoming a source of competitive advantage and process efficiency for organisations.


Estimates vary, but the market for Big Data Analytics – hardware, software and expertise – is expected to grow rapidly in the next 5 years to around $50 billion globally (Source: Wikibon). Although the big players are currently hardware and software providers such as IBM, HP, Intel and Fujitsu, there is a growing market for expertise to develop profitable management information from the data. This offers opportunities for small businesses, specialist “boutique” operators and new start-ups.


Governments, public bodies, financial services, telecommunications utilities, healthcare and manufacturing are the main consumers of Big Data Analytic services. Big Data Analytics offers these organisations competitive advantage in terms of consumer intelligence, new product development, supply chain management, risk and fraud management, and process improvement. However, the expertise in analysing such large amounts of data to produce useful management information to drive improvement is in short supply. Some sources suggest that whole economies will grow or decline based on their ability to provide a workforce with such IT and analytical skills.


The main challenge facing organisations aiming to fully exploit their data is the lack of skills and expertise – not just IT skills, but the management skills needed to interrogate, interpret and deploy the data effectively. The high cost of the investment required to fully utilise Big Data, and the lack of agreed standards in this area, are also barriers. Public concerns about data security and privacy also have a significant impact on the market.


However, the investment required need not be prohibitive. Several research studies have shown that cross matching location data with key Google or Twitter search terms, with weather forecasts or other publicly available data providers retailers with stores sales forecasts that are just as accurate as MI derived from traditional statistical analysis and, crucially, available in real-time, reacting to real events.


Similarly, data on Google searches relating to flu or other communicable diseases, cross referenced to location data, is now routinely used to forecast short-term demand for GP and emergency services.


Logistics firms have improved their delivery performance by combining data on transport availability, with weather forecasts and traffic conditions.


Many of the skill sets required for Big Data Analytics are not taught in traditional statistics courses. The skills required include cleaning, organising and manipulating large data sets. In addition, visualisation tools and techniques, and the ability to devise and test hypotheses rapidly and flexibility, are vital. Individuals who combine these skills with good business understanding are rare and, therefore, expensive.


To make Big Data work, an organisation needs to be flexible and open to new ideas. Large organisations typically have many sources of data, but they are held in different locations, in different (often incompatible) databases, and managed by different departments. Linking data across these various sources is a primary requirement, and major challenge. It may be as much a challenge for corporate culture as for IT.


As important as talented and flexible IT specialists, organisations need the skills and imagination to build models to analyse the data in new ways and test the outcomes. A culture where different propositions can be modelled and tested without “blame” for getting it wrong is essential.


Also vital is to make the tools and techniques of Big Data Analytics simple and understandable enough to be deployed in the workplace. Involving local managers and teams in the data is vital for responsive and timely decision making. This will require training and continuous development.


Finally, organisations need to be aware of aspects of corporate culture that might work against the move to Big Data. Departmental silos, internal politics and organisational policies are likely to be barriers. In one sales-based organisation providing flexible and accessible data on product and market profitability saw no improvement in performance. This turned out to be because incentives were solely based on sales targets with staff assessed on sales volumes rather than profitability. A major change to reward and recognition were required to realise the benefits of Big Data.


Big Data Analytics offers organisations tremendous opportunities for cost reduction, process improvement, and new product development. The ability to devise and test hypotheses to analyse the vast amounts of data that large organisations hold, and to develop timely and effective management information to drive improvement is a skill area in significant demand. Big Data Analytics is a huge market opportunity for UK businesses.