Wikibon just came out with a forecast (Big Data Vendor Revenue and Market Forecast) which underlines my last post: The hype is over, big data is getting real.
“In the enterprise space in particular, the combination of a better understanding of the use cases for Big Data and more mature product and service offerings resulted in a significant percentage of Big Data early adopters graduating from small, proof-of-concept projects to large-scale, production-level deployments.”
It also talks about the adoption barriers. These revolve around three major themes:
- Lack of Data Scientists
- Moving to higher levels of maturity as an analytic enterprise
- Lack of application development tools and services
It’s not a suprise that all these difficulties still persist as we’re still in an early phase of adoption from an innovation perspective. Over time all the adoption barriers mentioned there will be overcome. However, I do not believe we will get there by focusing on these barriers per se. Let’s re-frame it this way: In the early days of the automobile, every driver needed to be its own mechanic. In the early days of the PC, the early adopters were extremely knowledgeable about everything – they even built their systems by sticking together the components (as I did, too ;-)). This kind of capability is analogous to what is expected from a data scientist: He’s a Jack of all trades with a scientific foundation in Math, Statistics, Computer Science, programming with a diverse set of tools and languages and specific insights into the topic at hand. Over time we will not need that many Data Scientists of that profile, as technology will mature and the market will consolidate.
Till then, two options for the enterprise: sit and wait…. until others took care of making big data adoptioin more accessible or palpatable – OR: relentlessly focus on some kind of business scenario where going beyond the data that was analysed so far will expand the analytic capabilities. Pick the solution or technology to make it work now, but do not expect to define your big data standards NOW and for ever. It may well be that you will have to enlarge or change the technology foundation in 2-3 years from now. Till then you’ll have earned some early benefits and you’ll have developed a staff with far more experience to build on for the next phase in your big data journey.
Concluding remark: If you go through the above mentioned adoption barriers, it is obvious that the focus is on big data – per se. That focus is wrong. The focus has to be on business opportunities that can be exploited by advancing our analytic capabilities. Technology considerations are an afterthought. This helped the early adopters to move from a big data pilot to large scale implementations.