Why big data means big business for online retailers

The explosive growth of online – and now omni-channel retailing – has resulted in a tremendous increase in transactional and shopper data. Retailers see opportunities to use this data not only to better personalise the shopping experience, but also to influence merchandising decisions on their sites, and in the case of multi-channel retailers, at store level.

Retailers that are harnessing the power of big data – companies such as Boots, Marks & Spencer and John Lewis – are seeing their efforts pay off with higher customer spending and improved retention rates.

While retailers are increasingly using data to improve the customer experience, there is still a long way to go before we can say the industry is truly data-driven. Due to the complexity of accessing, integrating and analysing data, as well as the time and cost of implementing data management systems, there is a bottleneck between retailers and this valuable asset. In a study by CEB, it was found that only 11% of retailers look at data to make decisions – let alone use data to drive real-time decision-making.

Here are five changes that retailers can undergo to start driving real value from their data:

1. Put data at the centre of your business culture

To thrive in today's business world, companies must adopt not just the technologies and talent to manage big data, but also the organisational culture. A data-driven company is a more meritocratic company, allowing numbers and results (instead of someone's rank or tenure) to provide the baseline for decision-making.

Companies such as Amazon and Google thrive on the principle that nobody's idea is above or below being tested. This type of flat hierarchical structure empowers everyone in the organisation to generate and test ideas. By using data to not only deliver an exceptional user experience, but also to drive incremental change, organisations will reduce time and costs spent on IT while also encouraging innovation.

2. Think long and hard about which data will tell you what matters to your customer

Not all data is created equally. What sources of data should you acquire, and how can you get them? Can you ask your customers nicely for it? If so, what should you provide in return for data? You need to be asking yourselves all of these questions.

Securing the most meaningful data upfront will enable you to truly personalise your shoppers' experience. Being transparent with what information is collected, and providing options for how any such data are used, offers customers a sense of control and ownership in exchange for what they've shared.

Once you determine which data matters most for your business, consider and test the available data platforms to understand how you can best analyse and act upon your acquired data. Traditional relational databases may be prohibitively expensive and less suited to the vast majority of unstructured customer data that is generated today – increasingly, cloud-based, open-source, non-relational database systems are enabling in-house analytics teams to perform ad hoc queries on complex data sets to quickly monetise customer data.

3. Integrate your data for a single view of the customer

Customers don't think in channels – they expect a consistent experience from a retailer, whether they choose to shop in-store, online or by mobile. All retailers, whether multi-channel or not, need to develop a single view of the customer in order to provide this consistency and drive the execution of personalised, relevant marketing campaigns.

Getting this right can be extremely difficult, but becomes much easier when using open-source tools that can more easily integrate unstructured data sets from online transactions, in-store loyalty card activity and a myriad other sources.

Once a single view has been created, it is possible to tie together the execution of a message between multiple channels and target customers, allowing for personalised campaigns such as banner adverts, emails and website views. Personalisation brings better and more relevant retail experiences for consumers and helps retailers to accelerate shopping decisions through more relevant product exposure.

4. Test, test and test again

Data-driven companies such as Amazon and Google never stop testing. Retailers should adopt a robust experimental platform and replace costly IT project expenditures with small, low-cost and easy-to-execute tests. This is only possible if data can be easily accessed for ad hoc queries by the analytics team. By performing constant and ongoing tests of business process design, pricing, offer management, user interface design and marketing programmes, solutions can be improved incrementally and with a higher success rate.

Most retailers can only allocate the budget and resources for an average of three major IT projects each year, yet some projects that may not necessarily have the strongest ROI – such as social tool development – make their way on to the roadmap. Testing these innovations before committing to a major implementation will provide an accurate forecast of the results and help retailers steer clear of frivolous IT expenditures.

Amazon has a virtually unlimited budget, yet they test every move they make before implementing it across the site. Take a page from their playbook and add rigorous testing to your decision-making process.

5. Broaden the dataset

Consumer purchasing decisions are influenced by a variety of external factors, including weather, location and special events. Weather, census and calendar data are just a few samples of data sets that can all be integrated into data management systems so that it becomes possible to more accurately predict customers' needs. By using this data, retailers have the opportunity to be creative when delivering the most relevant product recommendations and promotions to customers. One example of this is highlighted in the results of a study conducted by Lovehoney, which showed that levels of shopping activity were directly proportionate to the level of customer alcohol consumption. This allowed them to target customers between the hours of 4pm and 10pm, at times when they were most relaxed with the purchasing process.

The ability to identify, harness and (crucially) monetise relevant data can create a powerful advantage for retailers, yet many companies are only just starting to come to terms with how to turn data into meaningful customer insight. To become a data-driven organisation, retailers must adopt a culture of innovation, embrace open-source technologies and put data at the heart of every decision they make. Big data is only going to get bigger so there's no better time than the present for retailers to start treating data as their most cherished asset.

David Selinger is the founder and CEO of RichRelevance

To get more articles like this sent direct to your inbox, sign up for free membership of the Guardian Media Network. This content is brought to you by Guardian Professional.