Comments on Big Data Market Forecast 2012-2017 and Big Data Adoption Barriers

Wikibon just came out with a forecast (Big Data Vendor Revenue and Market Forecast) which underlines my last post: The hype is over, big data is getting real.


“In the enterprise space in particular, the combination of a better understanding of the use cases for Big Data and more mature product and service offerings resulted in a significant percentage of Big Data early adopters graduating from small, proof-of-concept projects to large-scale, production-level deployments.”

It also talks about the adoption barriers. These revolve around three major themes:

  1. Lack of Data Scientists
  2. Moving to higher levels of maturity as an analytic enterprise
  3. Lack of application development tools and services

It’s not a suprise that all these difficulties still persist as we’re still in an early phase of adoption from an innovation perspective. Over time all the adoption barriers mentioned there will be overcome. However, I do not believe we will get there by focusing on these barriers per se. Let’s re-frame it this way: In the early days of the automobile, every driver needed to be its own mechanic. In the early days of the PC, the early adopters were extremely knowledgeable about everything – they even built their systems by sticking together the components (as I did, too ;-)). This kind of capability is analogous to what is expected from a data scientist: He’s a Jack of all trades with a scientific foundation in Math, Statistics, Computer Science, programming with a diverse set of tools and languages and specific insights into the topic at hand. Over time we will not need that many Data Scientists of that profile, as technology will mature and the market will consolidate.

Till then, two options for the enterprise: sit and wait…. until others took care of making big data adoptioin more accessible or palpatable – OR: relentlessly focus on some kind of business scenario where going beyond the data that was analysed so far will expand the analytic capabilities. Pick the solution or technology to make it work now, but do not expect to define your big data standards NOW and for ever. It may well be that you will have to enlarge or change the technology foundation in 2-3 years from now. Till then you’ll have earned some early benefits and you’ll have developed a staff with far more experience to build on for the next phase in your big data journey.

Concluding remark: If you go through the above mentioned adoption barriers, it is obvious that the focus is on big data – per se. That focus is wrong. The focus has to be on business opportunities  that can be exploited by advancing our analytic capabilities. Technology considerations are an afterthought. This helped the early adopters to move from a big data pilot to large scale implementations.



Big Data has moved beyond the hype

Moving beyond the hype means that the dust starts to settle down and one starts to see clearer. We definitely moved beyond the stage where one might doubt if big data is a new trend or a fad. There’s too many proofpoints that have shown the transformational potential of analyzing huge amounts of data that could have neither been done manually nor with traditional data warehousing or business intelligence means. Let me just pick two examples today that affect society at large:

  • Obama’s last presidential campaign leveraged a team of 100 data scientists to mine every information about supporters and potential voters it could get hold of and combined that insight with “social business” by smartly leveraging the campaign’s friends on Facebook. Those working on that campaign said they think they were four years ahead of the Republicans. And as if to confirm how important big data has become for everyone who needs to understand his supporters or “clients”, we can now read about the Republicans catching up, see e.g. here: WANTED: A Silicon Valley Tech Genius To Save The GOP, Gracy Wyler,

Many companies are still in the early stages of exploring how to use big data. Keeping in mind that Obama’s campaign was the second round after similiar experiences in his first campaign it becomes clear that big data will indeed be a key differentiator to compete. Those who don’t give up now (because they might have fallen in the trough of disillusionment, see Svetlana Sicular’s blog from January 2013: will enjoy that competitive edge in the near future.

Big Data Adoption – Skepticism is not a bad sign

I’m just back from IBM’s Analyst Insight (see: – a meeting with analysts where IBM shares key insights about what we see happening in the marketplace, how IBM is positioned and where it’s heading.

A key question that came up from several analysts is what are the “real” use cases, what is it that clients pay for. And indeed there’s a number of Big Data projects already under way and we have a number of compelling stories to share.

However, it seems there’s still a lot of skepticism in the market and also among some analysts. Which goes back to the fact that a majority of companies still doesn’t see the value and asks for proof points.
In fact this is what we would expect with any technology driven innovation: First, there’s the innovators and then the early adopters that explore how the new technology could create value for them. Together, these two segements account for about 15% of the whole market. The rest, i.e. the majority, is waiting to see the proof points and asks these skeptical questions. Which means a majority of skeptics is not really a negative signal about neither the trend nor the market. It’s just natural in the early adoption phase.  

What makes this phase exciting and interesting though, is that the early adopters are looking for fundamental change and big transformations. And there we see indeed a couple of good stories unfolding. More on that in a subsequent blog.

Big Data Adoption – Does It Depend on How You Define Big Data?

Today, I’d like to comment on (not summarize) a new big data study*, that came out on October 16th.

As opposed to the view that analysts and vendors paint, this study sheds some light on the current level of understanding and use of the much hyped notion of “Big Data” among business and IT professionals.

It turns out that

  • for more than 50% of respondents big data starts already at 1 Terabyte,
  • the most frequently named characteristic associated with big data was “a greater scope of information” (18%) while “large volumes of data” came in with only 10%, social media data with 7%,
  • more than half of the respondents reported internal data as the primary source of big data in their organization

which tells me that many have a very pragmatic view on big data, going back to the fact that data volumes are simply growing and with it the challenges to manage and analyze it in a timely fashion.

The study also reveals how many respondents already do have proper big data technologies already in pilot:

  • a columnar database (~ 14%)
  • Hadoop or NoSQL engines (~12% each)

The key challenge for all these pilots is to find a compelling business case. Which might be due to the fact that these pilots are mostly launched within IT and not within the business.

There’s still a lot of exiting work to be done to drive innovations that exploit the huge potential of Big Data – “the new oil” as Clive Humby coined it. Explorative work that starts with the challenging business questions, then identifies what resources are needed to answer these, and only then narrows down on the data resources needed. Which may be accessible already or which have yet to be acquired and collected. If that data is massive volume machine generated- or social media data, then most likely new analytic capabilities, true big data capabilities, are needed.

*) The IBM Institute of Business Value and the Oxford University just published a study of how big data is used today. The study surveyed 1144 business and IT professionals in 95 countries. It can be downloaded here:

Understanding Each Customer as an Individual – Why?

To understand the customer has always been important. But until very recently no company did more than segment their customers. No customer has been treated as a segment of one. Obviously there’s room for improvement if you have a million of customers but only 8 segments that you distinguish. So you may try to go to 20, 30 or maybe even 500. But why would it make sense to understand each and every one of those as an individual?

Obviously there’s no way to have 1 million different products or services to offer so that you could provide an exclusive service. So there will always be some sort of segmentation. But if you consider the factor of time – then you get the point: You need to understand what this individual most cares about NOW. Hence you have to have a much better understanding of your customers to serve them really well.

Each company has to rethink its Marketing Strategy. Social Media and Mobile change the way customers perceive, share, respond and act. Does this mean every company now needs to spy what their customers do on Facebook or Twitter? Certainly not. There has to be a transparent opt-in and a give and take of benefits. Information sharing for convenience, a nice experience or even financial rewards.

Now, how does big data come into the picture? To really extract meaningful insights from all that data – be it messages in social media, be it patterns of behaviour that can be inferred from the usage of a mobile phone – this all means challenges that have to be dealt with with a new kind of technologies – big data technologies.

I will come back to that in a future post. And I will try to keep my writing meaningful to a broad audience – not the IT professionals only.

Getting Started

Just returned from the CMO CIO Leadership Exchange in Paris. Had a lot of good and thought-provocing discussions there. Time for me to finally get started with a blog on what I care about in business. Which has always been innovation, strategy and transformation. Big Data is my current focus and I will share with you what I learn from my clients, the research I’m doing and I’m looking forward to learn from you and your responses, too. Would be great to get a dialogue started…

One final word: I do work for IBM. I’m the Big Data Leader for Global Business Services. But what I’m posting here is my point of view and does not necessarily represent IBM’s opinion or view.