Overflowing with information

An alumna of my seminar visited me the other day so we went to a restaurant near the campus for lunch. When I looked at the menu, it contained about 100 selections with different flavors and toppings. It took me a few minutes to decide on what to order while my eyes darted over the pages. Soon my ex-student mumbled, “Waiter, anything is fine, you decide!”

We often hear people complaining that they can’t make up their mind because there is too much information. Many of the manufacturers or retailers have developed a wide array of products and services to satisfy consumers' needs. As a result, there are a massive number of products and services distributed in the market, which is causing an overflow of information. Often times such a large amount of information makes selections too complicated for consumers.

The Nomura Research Institute released a result ^[1] from a survey of 10,000 people, showing that the proportion of people bothered by too much product information greatly exceeds that of those who feel there is a lack of such information. In this way, it can be said that the era of excessive information has arrived.

In the face of this situation, suppliers such as EC (Electronic Commerce) and retailers are being demanded by consumers to narrow down and provide information that is more beneficial.

Expanding available data

An increase in information is not entirely problematic. In the retail industry, since the introduction of Seven Eleven’s POS (point of sales) Register and Yodobashi Camera’s Gold Point Card (FSP; frequent shoppers program) in the 1980s, we have been able to collect detailed purchase history data and customer data. The representative of these is POS data and ID-POS data. By analyzing these data, it is possible to create marketing strategy and conduct CRM (Customer Relation Management) ^[2]. Also, in EC, all data in the buying process such as item search, comparisons and purchase on the internet, is recorded as access logs. Everyday affairs and stores you visit are casually posted on SNS, and these various types of behavioral data and lifestyle records have become diversified and recorded in increasing frequency. In recent years, these kinds of data has been called “big data,” and become a hot topic in the information industry besides manufacturers and retailers. Big data is not just large volumes. It also has characteristics where it can be used real-time or as a combination of data from multiple information sources. For example, web-viewing information of a certain category can be combined with the purchase history at stores in that category.

The range of usable data has expanded from structured data stored in conventional relational databases to sensor data obtained from GPS and other sensors, and to unstructured data such as posts and images uploaded onto social media sites (see figure). According to EMC, it is predicted that, in ten years from 2010 the worldwide information distribution quantity will swell 200 times more than that of today ^[3].

Extracting beneficial information

How to record and accumulate data is also an issue, but the existence of data alone is not useful for anything. We need to extract some useful information for management and marketing from the data. In other words, aggregating and analyzing data is necessary. Statistical theory has a long history in data analysis, and since then, various analysis methods have appeared. However, the importance of statistical theory has not waned. There has been a focus on operations research as a technique representing probability theory and optimization, as well as a focus on computer science including machine learning and data mining to analyze large scale data.

In regard to how to use and apply the analyzed information, the representative of that in the field of marketing are probably recommendations in EC sites. This is a system where each customer’s purchase history and behavioral data are analyzed, and products that they want to buy are displayed and promoted. This means that the “narrowing down of too many selections”, which I introduced in the beginning, is done by the website. This is a realization of communication, where in the past a sales staff at the market would look at a customer’s face and say, “I recommend this to you today.” Instead, the computer automatically makes a judgment based on analyzed data and combines it with each individual customer’s needs. In addition, cost reduction initiatives are taken, such as improving the accuracy of forecasting by reducing inventory and adopting flexible price changes to fit the changing demands.

It is not only essential to deep-dive into the research of every method and theory in the academic research field, but also along with that, knowledge and experience to judge the right situations and method of analysis are also becoming necessary. Data scientists, who are now in the spotlight as business data analysts, are required to obtain such additional judgment skills on top of high-level analytical techniques.

Therefore, it does not mean that you should only obtain data analysis skills, but it is necessary to understand what characteristics a business has and what kind of values it is providing to society in order to apply them in real business. Moreover, it is also necessary to correctly define what problems currently exist. The results gained from the analysis need to be digested and linked to the next action step. It is necessary to be able to oversee and drive the process as a whole.

Expectations for the future

The current mainstream of data analysis in the marketing field is recording the active behavior of customers such as purchase history and access log data in websites. In big data, the previously mentioned sensor data which is automatically acquired is also gaining attention. For example, NTT docomo is proceeding with the usage of mobile spatial statistics, which is a collection of location information obtained at regular intervals from mobile phones. Of course, we must prioritize privacy, but it is hoped that this kind of data will be utilized in disaster prevention and community development as well as area marketing.

In recent years, open data, which enables us to share usable data, has been put in the spotlight. Also, in December 2013, the government opened the trial version of a data catalogue site ^[4]. As of February 2014, roughly 10,000 items of data possessed by the government can be accessed. However these data are not detailed. Consequently it is hoped that this trial of providing data to the whole population will expand in the future.

Activities for data utilization are not limited to marketing, even in social science. For example, in energy consumption issues, it will be possible to further optimize energy supply and consumption by measuring energy consumption in real-time, instead of measuring it monthly.

You may think that the term “big data” came about as a totally new field. However as I have stated so far, it has fitted in perfectly with the evolution of several research areas such as computer science and statistics and begun to attract attention while creating the synergy effect. Because this evolution is guaranteed to progress in the future, I believe that research into how data can be utilized widely in society will become even more valuable from now on.