Analysing Big Data can help businesses run a lot more efficiently. The main challenge businesses face today is getting the best value from the data they have. They are beginning to explore where they fit within the Big Data cycle, and how to leverage data to turn it into actionable information.

In this post I examine a typical Big Data cycle, highlighting examples of how businesses are investing in Big Data and the direction this is taking.

The typical Big Data cycle

Big Data means different things to different people. It's basically defined as, data that's too complex to be processed on one machine. This can be due to various factors like the volume of it, the speed at which it’s generated and processed, inconsistency, and the variety of its formats and sources.

Big Data isn’t new, but traditional processing applications weren’t adequate to process these large and complex data sets. Newer technologies allow us to handle large scale data well, and carry out more detailed analysis to reveal patterns and trends.

To produce something meaningful from data, businesses need to manipulate it. This may involve having to parse and validate it, sorting it, or combining multiple sets of data before interpreting and visualising it.

The above diagram is a simplified view of a typical Big Data cycle. It shows each stage in the cycle and the leading technologies that could be used. For this cycle to work, the culture of Big Data needs to be embedded in the organisation’s business strategy.

Big Data at the core of the business

Enhancing customer experience and informing business decisions

Big Data analytics can be used to enhance the customer’s experience. Within retail, businesses can collect data about consumer behaviour in store and feed it back to the retailers and manufacturers. Knowing who users are allows businesses to better cater for them. They can use Big Data analytics to optimise promotions, implement targeted marketing, enhance the use of shelf space in the store, or improve demand forecasting.

Recently, insurance firm Admiral announced that it wanted to analyse the Facebook accounts of first time car owners to price their insurance. The project was prevented by Facebook, who wished to protect the privacy of its users. This controversy aside, we can look at this as an example of the type of insight businesses are trying to gain. Admiral’s firstcarquote was to be optional and would have offered discounts to users deemed safe drivers, based on its analysis of the potential clients’ Facebook posts. It analysed linguistic characteristics such as excessive use of exclamation marks. It looked for personality traits that it thought were linked to safe driving and used that information to inform its pricing strategy.

Big Data analytics can help inform business decisions. For example, when merchant bank Close Brothers wanted to find optimal locations for expansion, it used statistical and geospatial modules to gain insight into where its customers live in relation to its branches. This allowed the firm to identify target locations for potential expansion.

Sustaining competitive advantage

Big Data can give a business an edge over its competitors. Gracenote has the largest database of all music collections, with more than 28 million songs. Its online services handle a trillion queries a month. Leveraging its database, it built music recognition software MusicID, which it uses to identify files and deliver track metadata, and relevant content. MusicID retains audio fingerprints - unique digital identifiers for audio files that can then be used in queries. It identifies files by combining the audio fingerprint with its matching text. Users can view information like the artist’s page, biography, older releases and most popular songs. Gracenote’s database gives it an advantage over competitors like Shazam.

Enhancing security

Advances in data science and real time analytics can drive major improvements in fraud detection and enhance security. This type of problem is time sensitive and requires immediate action, and while it used to take days to detect problems, Big Data analytics can detect anomalies in real time, so they can be addressed straight away.

For example, banks can monitor and track the digital behaviour of users to prevent fraud. They can monitor how fast users generally type in their username and password, the geographic location from which they log in to their accounts, or frequent commands they use. A change in their behaviour can be quickly flagged. Cognitive systems can be used to prevent security threats at an unprecedented speed.

Transport network Uber recently partnered with Microsoft using its cognitive service Real-Time ID to verify the identity of the driver. The driver takes a selfie that should match the picture associated with his or her account, to prevent fraudulent use. It instantly compares the photo sent by the driver to the one in the database and blocks the account if there’s no match.

Predictive analytics and enhanced performance

Big Data can be used to improve the efficiency of an application and report issues in real time. While reporting on the data is important, it’s only the first step. One of the greatest benefits to organisations of investing in Big Data is the ability to accurately predict what’s likely to happen. This is only possible with a large volume of data. This is how Shazam, for example, is able to predict the next number one song. Users have uploaded 15 million songs to be identified. This makes it possible to know what users are listening to at any given time, and to identify the tracks with the biggest spikes in the number of listeners.

Using Big Data analytics to anticipate results is particularly useful for predictive maintenance. Data is used to learn about the current condition of a product, and predict when maintenance is needed.

Using Microsoft's Azure IoT suite, propulsion solutions provider, Rolls-Royce, is collecting and analysing airline operations data to ensure reliable operation, improve performance, reduce maintenance costs, reduce fuel consumption and detect technical issues before they even happen. Previously the company only analysed engine data. The engine monitoring system uses sensors to gather real time information about the health and performance of the engines. For instance, it collected data about the amount of fuel needed to make a set amount of power.

Now, because it has the technology to process and analyse bigger sets of data, it receives data streams from both the engines and aircraft, and the aircraft monitoring system enables more information to be captured. For example, it can detect unusual aircraft speed and collect technical logs and black box data. This analysis then feeds into the product design, manufacturing and support.

Similarly, GE Aviation integrates flight data and analytics to optimise fuel consumption. By measuring how something currently functions, analysing inefficiencies, optimising performance and predicting failures, it turns the engine from a product into a service, allows repairs to be planned for scheduled downtimes.

The future of Big Data

Big Data is useful for understanding how users interact with a product and detecting anomalies in real time, but its true potential is in its ability to predict what’s going to happen, rather than reacting to an event. It’s gradually moving from simple data reporting to more advanced analytics, with predictive modeling and machine learning.

While there’s a lot of value in Big Data projects, one of the main challenges that businesses will need to resolve is privacy protection for their users. We’ve already seen an example of that issue with Admiral’s firstcarquote project, and some insurers even charge higher premiums for customers thought to be less likely to switch providers based on data analysis. There has always been a struggle to balance the use of data to drive business value, and respecting user privacy. This is set to become even more prominent now that data is constantly generated by everything around us and we’re better able to process it.