Are BI/ETL vendors ready for "data kitchens"? Because users are

If you've been in the BI/ETL industry for several years you may remember that many years ago BI/ETL vendors actively promoted the concept of so called "BI standardization". Gartner, Forrester and other market analysts also talked about it -- organizations should stop having "zoo parks of systems" and standardize on one platform. At that time even the big BI vendors were only transitioning from a single-tool client-server architecture to a multiple-tool web-based one and many hoped that once they complete the transition organizations would be able to cover their data analysis needs with a comprehensive product set (platform) from one vendor. These expectations were driven by high cost and complexity of the analytical systems at that time, so standardizing on one platform would facilitate building in-house expertise, lower maintenance costs, and simplify support and administration.

However, the reality turned out to be more complex. As a matter of fact it became clear that no vendor can offer really comprehensive product suite that would satisfy data analysis hunger of various types of users. The more users became involved into data analysis the more diverse and sophisticated needs they developed.

It seems to me that organizations are increasingly becoming ready to embrace the concept of "data kitchen" where users have a choice from many tools so that they can choose whether to use a "spoon", "fork", or "knife" for a job, rather than having just a "spoon" for all cases. However, the problem is that the vendors are not ready -- they still want customers to buy their expensive cumbersome enterprise platforms.

So what would be the difference between a "data kitchen tool" and an "old-school tool":

Click to zoom

I guess the table above is self-explanatory. I would only make a couple notes:

Usability was long ignored but now it's the king. First, because data analysis is difficult, therefore I believe that software vendors should go the extra mile to design well thought out, clean and polished UI. Enterprise software should be smarter and simpler, even at a cost of removing some functionality (look at some popular mobile apps). Second, when you have many tools in your "kitchen" you can't afford spending a lot of time figuring out how to use each of them. A single tool may not require too much attention. Selfish ones don't survive in a team. Hence the necessity of open data formats and APIs. Open metadata is required for throughout data governance -- a must-have for a "data kitchen".

Another note is about price. Cost structure per user will change. If previously an organization could spend $5,000 for one license for one user, one should not expect that because of the "data kitchen" organizations will start buying 10 tools for the same $5K each spending in total $50K per user. Instead they will be looking to offer a user 10 tools for $500 each. I believe those software vendors that resist the change and keep prices high will be eventually squeezed out of the market.

You can check your favorite software against the table above. Some products are better suited to find a place in a "data kitchen", some are not. In my opinion Tableau is a good example of well thought out and polished user experience aimed for self-service use. I wish only they opened TDE and/or adopted some open format for data exchange. I hope EasyMorph can become another good example of a tool that is perfectly suitable for the "data kitchen" concept. We're living in an interesting time after all -- the BI/ETL market stagnated for long time, but now the pendulum has swung in the opposite direction and we can observe many interesting products coming to the market.

Isn't it great?

A few slightly pessimistic comments on Gartner's BI Platforms report 2010

Gartner's annual BI reports always create a lot of buzz, when issued. The most discussed (and perhaps, the most contradictory) part of these reports surely is the "quadrants" with vendors positioned inside.

Ironically, these quadrants definitely are not the most interesting and valuable part of the report. Really, I don't know what valuable knowledge can be obtained from them. The latest-greatest BI? Come on. In BI rollouts every organization has it's own set of goals, criteria, preferences and restrictions that need proper analysis and specific solution. How would you apply that quadrants there? In my opinion, the quadrants are just too abstract to be practically useful.

Gartner fulfills tremendous task of identifying market trends, surveying numerous users and analyzing complex product portfolios, and they do it very professionally. I find the sections Market Overview and Vendors Strength and Cautions very compelling and worth in-depth reading. Can't say I agree with everything there, but no doubt, it's a good job done by smart people.

A few comments on Gartner's Market Overview this year:

• "Data discovery platform momentum accentuates the need for a portfolio approach". Last year Gartner mentioned that idea of "BI-standardization" has actually failed -- more and more customers intentionally use 2-3 BI-platforms to cover needs of business users better. This year the trend continues. As of today, there is no single BI-platform that can effectively cover needs of a large enterprise, solely.
• "Data discovery" is a new hot thing this year -- did you notice this? Do you remember the hype about "pervasive BI" a few years ago? Did it make any revolution? Well, no. Will data discovery do it? Who knows. However, it's obvious that BI industry still can't overcome it's biggest problem -- low adoption rates among non-technical users. What concerns me is that this year there is absolutely nobody in the Visionary quadrant -- it means that there will be no any significant innovations in the next 3-5 years. No really fresh ideas on BI market. Well OK, QlikView has shown very interesting and innovative approach to BI. But their recent new versions (10, 9, etc.) look more like boring updates rather than new breakthrough.
• "Shift from measurement to analysis, forecasting and optimization". This is what BI is for -- to drive businesses in the right direction. However, I don't believe in hype about predictive analytics. This is just another wrong attempt to overcome the above-mentioned biggest problem of BI. Building predictive models is not a simple task and it requires from user to have skills in statistics at least to understand what's under the hood. Tools will not magically compensate lack of these skills. No magic.
• "Mobile BI" -- another one hot thing in BI. Will BI revolution come from this side? Who knows. What's good about it -- is that this trend is inline with global, fundamental shift in lifestyle which is happening now. What's not good about it is that common hype about tablets (iPads, etc.) inflates hype about mobile BI. Hype about tablets will fade as more and more people make distinction between toys and tools. But today this hype prevents us from clear understanding the value mobile BI can give. Really, is it a toy or a tool?
• Unfortunately, the second current fundamental shift in lifestyle -- socialization -- didn't get proper reflection in BI industry. And this is disappointing, because this is clearly the way to better productivity caused by better knowledge exchange. And knowledge is what business intelligence used to be proud of. I believe, social BI, if designed properly, can have much higher chances to bring valuable innovation to what is called "management information systems". Who's going to do this?

Links:
[1] - Gartner's Magic Quadrant for Business Intelligence Platforms (27-Jan-2011)
[2] - Paul Graham's essay about tablets