What's missing in Google Analytics charts

My two recent posts about Google Analytics charts. These posts might be useful if you're looking for ways how to share your Google Analytics stats with others (this can be done using Explainum):

1. What's missing in Google Analytics charts
2. How to add Google Analytics chart to your website or blog without Google

Here is an example of widget which was created using Explainum. The widget is interactive -- try clicking comments, selecting rectangular areas on chart to get related comments, etc. For more features read Explainum How To/FAQ.

Busting 5 myths about QlikView

You've probably noticed how well QlikTech's marketing machine is working -- bold statements, slightly ecstatic customer stories, provocative (or simply not well-thought?) assertions, impressive growth figures, etc. All this is intended to create aura of "magic thing" which works extremely well for brand awareness as people like to tell each other magic stories since beginning of humankind.

QlikView is for sure an interesting tool, which, as any other BI platform, has its own area of applicability. The goal of this post is to examine some popular myths about QlikView and help those who are choosing BI platform and considering QlikView. At the end of the day, the less is disappointment from unrealized expectations -- the higher is satisfaction.

Myth1 #1: QlikView is extremely fast
This is true -- QV is indeed very fast. Its in-memory engine, which stores all data in RAM indexed and compressed, eliminates slow disk I/O operations and therefore all selections and filters are processed extremely fast. Not to forget to mention high utilization of multi-core CPU architectures and not so widely known QlikView's feature -- precompilation of selections.

At the same time, as data volume becomes larger the response time increases proportionally. However, QlikView can't scale horizontally and split processing of a query to several nodes (thus reducing query time inversely to number of nodes). At the same time, relational analytic DMBSs with horizontally scalable MPP architecture can achieve similar to QlikView response times on relatively large data sets ( >1bln. of rows ). One of my customers has got similar response times for QlikView on 64GB server and two-node Vertica cluster (32GB each node) for 200GB data set.

Myth #2: Rollout time for QV is weeks not months
This is also true. Due to absence of sophisticated metadata layer and use of automatically pre-joined tables (so called associative model) developing not complex analytic applications is fast and easy. 1-2 weeks from requirements specification to first working prototype is not a nonsense. Good set of charts with very flexible expressions engine also make life easier.

However this has its back side of the moon. Primitive metadata layer means that you wouldn't be able to operate with hundreds of measures and dimensions in one model. At the same time QlikView combines cumbersome and overloaded forms for object properties with surprising inability to easily adjust basic visual settings like background color or font size of a table header row. For me QlikView is the first BI tool where I can spend more time adjusting visual appearance of a report than building a data model for it. I wonder -- is it really necessary to have a separate checkbox (!) for rainbow-colored borders (see the screenshot below) when it's not possible to easily adjust the above-mentioned basic visual settings?

Myth #3: With QlikView you don't need a Data Warehouse

Well, if you don't need data warehouse with QlikView then most probably you don't need it in any case. In another words -- for the majority of cases this is not true. First, QlikView by design deals with star-schema models only. Second, volume of data stored in QlikView is strictly limited by RAM capacity. Even with typical compression rate 1:3 this may very soon become an obstacle. Third, despite QlikView has its own ETL engine its data cleansing possibilities are very basic.  So in case of need for good data quality processing a dedicated ETL/DQ tool will be necessary. Forth, the mentioned above lack of good metadata model limits capabilities for metadata management, lineage and impact analysis.

At the same time, good fit for star-schemes, fast response time and quick prototyping make QlikView very attractive choice for data marts built over a corporate data warehouses. Not to forget QV's wonderful capability to merge Excel spreadsheets and text files with data from RDBMSs in a few mouse clicks which makes enhancing DWH data with Excel data very easy for non-technical users -- an important use case. (UPDATE 23/6/2015 - with my new ETL tool EasyMorph you can do it even easier).

Myth #4: QlikView is an enterprise BI platform

Briefly -- no, it's not. While QlikView is a very advanced dashboarding tool with some nice query & analysis capabilities it wouldn't cover the needs for heavy reporting, ad hoc analysis across hundreds of measures and dimensions, balanced scorecarding, data mining etc. which a large enterprise might have. Lack of single, easy manageable and scalable security model also limits areas of applicability for QlikView. Currently, in order to manage user access rights one should set them up in 4 (!) places -- load script, document properties, visual object properties and QlikView Publisher. While it's more or less bearable for 50-100 users, I guess you wouldn't want to go through all of this for a few thousands of users.

Myth #5: QlikView is inexpensive

As of price-list, QlikTech offers a pricing model which is rather attractive for small businesses when compared to offerings from its larger rivals -- IBM, Oracle or SAP. However, when it comes to larger deployments, the difference in cost per user becomes smaller and sometimes not in favor of QlikTech.

According to Gartner:

"QlikView is increasingly seen as expensive — almost a third of its customers surveyed (31.4% vs. 26.1% in the whole sample) see this as its main barrier to wider use. Its pricing model often does not sit well with larger deployments to more users, nor does the investment in RAM required to support the increasing numbers of concurrent users." (Gartner's MQ for BI platforms 2011)

At this point you might think that QlikView isn't worth its money. Well, while QlikView is definitely not what QlikTech's marketing propaganda is trying to tell us, it's still worth consideration.  Short time-to-production, good capabilities for data manipulations by non-technical users, very interactive point-and-click user interface and high-speed query processing -- all of these can definitely make QlikView a good fit for some people in your organization.

Read also: Really, is QlikView a BI tool?

Explainum

In May, 2010 I've started a project which seemed like an adventure -- I decided to build web-service for creating charts with comments linked to data regions. I did it because I think that BI and data visualization industry have missed one important point -- they do not deal with human interpretation of visual data. In another words, existing BI and data viz tools just get data, draw picture and stop there. All findings from that data, as well as thoughts, questions, conclusions and forecasts about it remain outside of a system. While people generally used to it, nevertheless it produces some inconveniences. Here are some of them:

1. It's not an easy task to find reasons for a KPI change in a certain period -- neither popular data visualization tools nor search engines are note capable to do this simple thing which is obviously needed in the business world.
2. Knowledge of influencing factors behind KPI trends is spread across emails, documents, IM messages (this makes p.1 even less easy).
3. As majority of influencing factors are rather qualitative than quantitative, so they remain out of decision-support systems (hmm... why are they called decision-support systems in this case?).

Thus, I've decided to make a tool that will deal with this problem. It's called Explainum. It is not in production yet -- we're just about to launch closed beta-testing. If you want to take part in it -- feel free to register.

Some of its features:

• Trend charts, which can automatically update data set with new data every day from various data sources -- CSV files, stock market data, currency exchange rates or web-services like Google Analytics
• Users can create/read comments for selected data regions. In order to find comments related to a certain time period users should simply select a rectangular area on a chart
• Charts can be embedded into 3rd party web-pages as interactive widgets

Read "What is Explainum?" for more detailed descriptions of the idea behind Explainum or see sample widgets. Here is screenshot of chart made using Explainum -- just to give you an idea how it looks like (clickable). As you can see -- commented areas are highlighted and chart has list of comments attached.

Teradata acquires Aster Data: Final switch to a new generation of analytical engines

Following HP's acquisition of Vertica, Teradata decided to buy Aster Data. The deal is a noticeable milestone -- now all major vendors of DWH platforms have switched to a new generation of analytical engines:

• Teradata will have Aster Data soon
• IBM has Netezza
• Oracle has Exadata
• Microsoft has SQL Server Parallel Database, going to have columnar storage in Denali
• SAP has Sybase IQ, Sybase MPP, Explorer Accelerated

Not to forget emerging players on DWH market:

• HP with brilliant Vertica
• EMC with Greenplum

The new generation  features (in various combinations) Massively Parallel Processing (MPP), columnar storages, hardware SQL acceleration, MapReduce, advanced in-database analytical functions.

Era of row-based SMP databases for analytical workloads on large datasets has gone. Don't miss the train.

3 reasons why you should use wiki for BI deployment

As I mentioned in one of my previous posts, BI vendors do not pay much attention to collaboration around BI deployments. However, this doesn't mean that large BI deployments don't have difficulties (or at least inconveniences) with collaboration and knowledge exchange. Some of these problems can be resolved using wikis:

First, and the most often problem -- lack of documentation convenient for both business users and technical team. Traditionally, project documentation (like specifications, glossaries, scopes of works, etc.) is done using Word/Excel documents. This results in network folders filled with tens or even hundreds of documents without any browsing possible. No need to say, that average business user will never look there. As BI-platforms doesn't have any good collaboration capabilities (excluding may be IBM), so user doesn't have many choices if he/she needs to know the logic behind certain indicator in a report -- only to ask somebody from tech team. In case of 5 users and 10 reports -- this is not a problem. But if you have 2'000 users and 10'000 reports -- this would be a problem. Often users, having such obstacles, just don't want to dig into details. And then we have low user adoption. So, reason #1 -- you need wiki to have the documentation searchable, manageable, consistent and convenient for use by business users first of all.

Reason #2 is that users usually don't have good How-To manuals. BI vendors usually do rather good manuals for developers, but in majority of cases they do not produce good illustrated manuals for business users. How to drill data, how to join data from two data sources, how to make ad hoc queries with subqueries, etc. etc. -- for all of these how-tos users need simple easy-to-understand illustrated manuals. And, what is important -- these manuals have to be easily extensible to target specific problems, if they occur.

And, finally, reason #3 is that large BI deployments typically have several BI tools. All of them have more or less decent portals for their own content, but none of them can hold BI content from other BI platform. Cognos knows nothing about BusinessObjects reports or QlikView application, Oracle BIEE knows nothing about Tableau and so on. At the same time a user might need access to several BI suites to perform his/her daily tasks. This is why you may want to have single subject-oriented (not tool-oriented) portal that would have links to various BI-content, or even have BI content embedded just in wiki pages. The latter may require some web-development works, as not all BI tools allow easy content embedding, and setting up integrated security, but it can also lead to much better convenience and user adoption.

There are a lot of wiki engines available -- free and commercial, easy and complex, etc. Wikimatrix can make the task of choice a bit simpler. I found DokuWiki to be perfect tool for majority of cases.

CV as dashboard

A few weeks ago I read on HackerNews a story about one designer, who made his/her CV as infographic artwork. This idea inspired me to create my own resume in a form of QlikView dashboard. So, I spent 3 hours to create QlikView application, that visualize my experience, education and some skills. As it appeared rather good to me, I decided to test the idea by publishing it on LinkedIn in QlikView group (link).
The response has surpassed my expectations. I've got a lot of very good and encouraging comments from people all around the globe. I've also got some mails from recruiters and potential employers, but this time without any result (not having US/EU work permit is a huge obstacle). Two weeks in a row LinkedIn had been indicating me as top influencing person in the group. The CV was also published as Creative CV #19 on www.globalrecruitingroundtable.com.

While it was almost a joke, I think the idea has some reason behind it. A random resume looks like typical poorly designed BI report -- too much text, small font, and sometimes, too much formatting. At the same time, every recruiter needs only some key facts and figures from a resume in order to make a decision -- whether to continue with it or throw it into waste basket. Clean and accurate visualization of these key facts and figures could make their life easier.

(click to enlarge)

A few slightly pessimistic comments on Gartner's BI Platforms report 2010

Gartner's annual BI reports always create a lot of buzz, when issued. The most discussed (and perhaps, the most contradictory) part of these reports surely is the "quadrants" with vendors positioned inside.

Ironically, these quadrants definitely are not the most interesting and valuable part of the report. Really, I don't know what valuable knowledge can be obtained from them. The latest-greatest BI? Come on. In BI rollouts every organization has it's own set of goals, criteria, preferences and restrictions that need proper analysis and specific solution. How would you apply that quadrants there? In my opinion, the quadrants are just too abstract to be practically useful.

Gartner fulfills tremendous task of identifying market trends, surveying numerous users and analyzing complex product portfolios, and they do it very professionally. I find the sections Market Overview and Vendors Strength and Cautions very compelling and worth in-depth reading. Can't say I agree with everything there, but no doubt, it's a good job done by smart people.

A few comments on Gartner's Market Overview this year:

• "Data discovery platform momentum accentuates the need for a portfolio approach". Last year Gartner mentioned that idea of "BI-standardization" has actually failed -- more and more customers intentionally use 2-3 BI-platforms to cover needs of business users better. This year the trend continues. As of today, there is no single BI-platform that can effectively cover needs of a large enterprise, solely.
• "Data discovery" is a new hot thing this year -- did you notice this? Do you remember the hype about "pervasive BI" a few years ago? Did it make any revolution? Well, no. Will data discovery do it? Who knows. However, it's obvious that BI industry still can't overcome it's biggest problem -- low adoption rates among non-technical users. What concerns me is that this year there is absolutely nobody in the Visionary quadrant -- it means that there will be no any significant innovations in the next 3-5 years. No really fresh ideas on BI market. Well OK, QlikView has shown very interesting and innovative approach to BI. But their recent new versions (10, 9, etc.) look more like boring updates rather than new breakthrough.
• "Shift from measurement to analysis, forecasting and optimization". This is what BI is for -- to drive businesses in the right direction. However, I don't believe in hype about predictive analytics. This is just another wrong attempt to overcome the above-mentioned biggest problem of BI. Building predictive models is not a simple task and it requires from user to have skills in statistics at least to understand what's under the hood. Tools will not magically compensate lack of these skills. No magic.
• "Mobile BI" -- another one hot thing in BI. Will BI revolution come from this side? Who knows. What's good about it -- is that this trend is inline with global, fundamental shift in lifestyle which is happening now. What's not good about it is that common hype about tablets (iPads, etc.) inflates hype about mobile BI. Hype about tablets will fade as more and more people make distinction between toys and tools. But today this hype prevents us from clear understanding the value mobile BI can give. Really, is it a toy or a tool?
• Unfortunately, the second current fundamental shift in lifestyle -- socialization -- didn't get proper reflection in BI industry. And this is disappointing, because this is clearly the way to better productivity caused by better knowledge exchange. And knowledge is what business intelligence used to be proud of. I believe, social BI, if designed properly, can have much higher chances to bring valuable innovation to what is called "management information systems". Who's going to do this?

Links:
[1] - Gartner's Magic Quadrant for Business Intelligence Platforms (27-Jan-2011)
[2] - Paul Graham's essay about tablets