For those who monitor QlikTech's stock price

Many visitors of this blog are very loyal to QlikView. For those of them who are interested in QlikTech's share price trends -- here is embeddable widget that updates automatically every day. The widget is interactive -- try clicking comment, scroll, zoom, click dots or select rectangular areas. I create comments to it from time to time. Feel free to grab the chart and embed anywhere you like and make your own comments (you need to have account on Explainum for this).


This is the HTML code you should use if you want to embed this widget into your web-page. Width and height of the widget can be adjusted by according parameters.

<script type="text/javascript">
//<![CDATA[
explainum_chart_id = 38;
explainum_widget_width = 500;
explainum_widget_height = 350;
//]]></script>
<script type="text/javascript" src="http://explainum.com/scripts/loader.js">
</script>

How to build good dashboard. Part 2: Usage scenario

Once you've decided to build a dashboard you need to plan it. Planing a dashboard actually means two tasks: defining usage scenario and zoning.

To understand usage scenario you'll want to have answers to these questions:

• Who will use the dasboard? What are their roles in the organization?
• What are business goals of each user or each particular group of users? What kind of tasks are they trying to accomplish?
• What information is of primary importance for the user? What is of secondary importance? What is nice to have but not very important?
• What kind of troubles do users need to identify?

Let's take a closer look at these questions.

User roles
In majority of cases there are 3 main groups of users: regular users, analysts and management users.

Regular users usually work with relatively narrow subject area and always need detailed information about it. They do not do much of ad hoc analysis and usually have pretty straightforward workflow. In terms of dashboard content they usually need balanced mix of gadgets, charts, tables with a few filters.

Analysts work a lot with detailed information in ad hoc manner. Usually dashboards are not the best main tool for them -- they need powerful query & analysis applications. However, dashboards can be good for quick identifying of problems and starting point of analysis. Analyst dashboards usually contain a lot of tables, charts with actual numbers and many filters.

Management users often track set of key performance indicators. They also constantly check actual numbers versus planned/estimated numbers. Besides KPIs they usually want to know best (or worst) products/customers/dealers/etc. So typical management dashboard has a lot of gadgets and charts with actuals and estimates and a few tables with lists of top products, customers, etc. Management users rarely work with detailed data so it's better not to use large tables for them.

Goals and Tasks
Make sure that dashboards have obvious and distinct indicators that show progress towards reaching strategic and/or operational goals. These goals can be either rarely changed -- "static" goals like annual targets or operational benchmarks, or "dynamic" goals like goals of short-time projects, marketing campaigns, reorganizations etc. When planning a dashboard keep groups of "static" and "dynamic" goals separately.

Priority and Importance
Prioritization is a key to building good dashboard. When designing a dashboard we need to deal with 2 key limitations: a) limited screen space, and b) limited human ability to read and prioritize information from many visual objects. And keep in mind that b) is more significant than a) -- Moore's low doesn't work for humans (unfortunately). Hence, it is very important to make more important information more eye-catching, easy to read and understand. However, we can't have 500 or 100 very important things in our dashboard -- average human can't track more than 20 important indicators and even 20 is a lot. Therefore, we need to prioritize carefully and plan working space of a dashboard accordingly. The less important is information -- the more clicks/actions/time should be necessary to get it.

Identification of Troubles
Make sure that you understand what are major problems/troubles that users want to identify with the dashboard. Try to make list of them -- it shouldn't be very long. Then make sure that every trouble will be obviously indicated in the dashboard -- either with different color or shape, special gadget, eye-catching flag or alert message that is invisible in normal conditions.

In Part 3 I will talk about zoning.

How to build good dashboard. Part 1: Dashboards vs reports

For majority of BI developers Business Intelligence have roots in building static reports which mostly contain tables with aggregated data from SQL queries. While static reporting still remains as a significant portion of data visualization in a large organization, dashboards continue to gain popularity, especially with wide adoption of tools like QlikView which simply doesn't have any other form of data visualization except dashboards (don't tell me about reports in QlikView -- they're barely usable).

However, building a dashboard requires different approach than creating a table-based report, because the way users work with dashboards noticeably differs from the way users work with reports:

1. Reports usually have multi-page content, dashboards are single-page (dashboards can have several sheets or tabs but usually they can be considered more or less independent as they need to answer different questions)
2. Dashboards are intended to give answers at first sight, at the same time reports can contain a lot of detailed data that might require more thorough analysis
3. Reports are often designed to be printed while dashboards are designed for screens; despite printing dashboards is rather common practice, I think this is done because of lack of social features in BI suites
4. Dashboards are interactive; reports, despite often having filters and drill-down capabilities, are more static by nature
5. Dashboards tend to be similar to applications, reports tend to look like a document.

Generally, I consider dashboards as a more progressive way of data visualization than static reports (excluding cases when basic documents such as invoices or regulatory reporting are needed) because visual representation of numerical data is better than textual one, especially when we need to catch deviations from a pattern, which is very common case in business data analysis. Many years BI vendors had been diminishing role of dashboards -- luckily in recent 2-3 years they changed their mind and greatly developed their offerings. SAP, Oracle, IBM, Microsoft -- all of them now offer dashboarding tools, but none of them is so advanced as QlikView is.

In Part 2 I will talk about planning a dashboard.

What's missing in Google Analytics charts

My two recent posts about Google Analytics charts. These posts might be useful if you're looking for ways how to share your Google Analytics stats with others (this can be done using Explainum):

1. What's missing in Google Analytics charts
2. How to add Google Analytics chart to your website or blog without Google

Here is an example of widget which was created using Explainum. The widget is interactive -- try clicking comments, selecting rectangular areas on chart to get related comments, etc. For more features read Explainum How To/FAQ.

Busting 5 myths about QlikView

You've probably noticed how well QlikTech's marketing machine is working -- bold statements, slightly ecstatic customer stories, provocative (or simply not well-thought?) assertions, impressive growth figures, etc. All this is intended to create aura of "magic thing" which works extremely well for brand awareness as people like to tell each other magic stories since beginning of humankind.

QlikView is for sure an interesting tool, which, as any other BI platform, has its own area of applicability. The goal of this post is to examine some popular myths about QlikView and help those who are choosing BI platform and considering QlikView. At the end of the day, the less is disappointment from unrealized expectations -- the higher is satisfaction.

Myth1 #1: QlikView is extremely fast
This is true -- QV is indeed very fast. Its in-memory engine, which stores all data in RAM indexed and compressed, eliminates slow disk I/O operations and therefore all selections and filters are processed extremely fast. Not to forget to mention high utilization of multi-core CPU architectures and not so widely known QlikView's feature -- precompilation of selections.

At the same time, as data volume becomes larger the response time increases proportionally. However, QlikView can't scale horizontally and split processing of a query to several nodes (thus reducing query time inversely to number of nodes). At the same time, relational analytic DMBSs with horizontally scalable MPP architecture can achieve similar to QlikView response times on relatively large data sets ( >1bln. of rows ). One of my customers has got similar response times for QlikView on 64GB server and two-node Vertica cluster (32GB each node) for 200GB data set.

Myth #2: Rollout time for QV is weeks not months
This is also true. Due to absence of sophisticated metadata layer and use of automatically pre-joined tables (so called associative model) developing not complex analytic applications is fast and easy. 1-2 weeks from requirements specification to first working prototype is not a nonsense. Good set of charts with very flexible expressions engine also make life easier.

However this has its back side of the moon. Primitive metadata layer means that you wouldn't be able to operate with hundreds of measures and dimensions in one model. At the same time QlikView combines cumbersome and overloaded forms for object properties with surprising inability to easily adjust basic visual settings like background color or font size of a table header row. For me QlikView is the first BI tool where I can spend more time adjusting visual appearance of a report than building a data model for it. I wonder -- is it really necessary to have a separate checkbox (!) for rainbow-colored borders (see the screenshot below) when it's not possible to easily adjust the above-mentioned basic visual settings?

Myth #3: With QlikView you don't need a Data Warehouse

Well, if you don't need data warehouse with QlikView then most probably you don't need it in any case. In another words -- for the majority of cases this is not true. First, QlikView by design deals with star-schema models only. Second, volume of data stored in QlikView is strictly limited by RAM capacity. Even with typical compression rate 1:3 this may very soon become an obstacle. Third, despite QlikView has its own ETL engine its data cleansing possibilities are very basic.  So in case of need for good data quality processing a dedicated ETL/DQ tool will be necessary. Forth, the mentioned above lack of good metadata model limits capabilities for metadata management, lineage and impact analysis.

At the same time, good fit for star-schemes, fast response time and quick prototyping make QlikView very attractive choice for data marts built over a corporate data warehouses. Not to forget QV's wonderful capability to merge Excel spreadsheets and text files with data from RDBMSs in a few mouse clicks which makes enhancing DWH data with Excel data very easy for non-technical users -- an important use case. (UPDATE 23/6/2015 - with my new ETL tool EasyMorph you can do it even easier).

Myth #4: QlikView is an enterprise BI platform

Briefly -- no, it's not. While QlikView is a very advanced dashboarding tool with some nice query & analysis capabilities it wouldn't cover the needs for heavy reporting, ad hoc analysis across hundreds of measures and dimensions, balanced scorecarding, data mining etc. which a large enterprise might have. Lack of single, easy manageable and scalable security model also limits areas of applicability for QlikView. Currently, in order to manage user access rights one should set them up in 4 (!) places -- load script, document properties, visual object properties and QlikView Publisher. While it's more or less bearable for 50-100 users, I guess you wouldn't want to go through all of this for a few thousands of users.

Myth #5: QlikView is inexpensive

As of price-list, QlikTech offers a pricing model which is rather attractive for small businesses when compared to offerings from its larger rivals -- IBM, Oracle or SAP. However, when it comes to larger deployments, the difference in cost per user becomes smaller and sometimes not in favor of QlikTech.

According to Gartner:

"QlikView is increasingly seen as expensive — almost a third of its customers surveyed (31.4% vs. 26.1% in the whole sample) see this as its main barrier to wider use. Its pricing model often does not sit well with larger deployments to more users, nor does the investment in RAM required to support the increasing numbers of concurrent users." (Gartner's MQ for BI platforms 2011)

At this point you might think that QlikView isn't worth its money. Well, while QlikView is definitely not what QlikTech's marketing propaganda is trying to tell us, it's still worth consideration.  Short time-to-production, good capabilities for data manipulations by non-technical users, very interactive point-and-click user interface and high-speed query processing -- all of these can definitely make QlikView a good fit for some people in your organization.

Read also: Really, is QlikView a BI tool?

Explainum

In May, 2010 I've started a project which seemed like an adventure -- I decided to build web-service for creating charts with comments linked to data regions. I did it because I think that BI and data visualization industry have missed one important point -- they do not deal with human interpretation of visual data. In another words, existing BI and data viz tools just get data, draw picture and stop there. All findings from that data, as well as thoughts, questions, conclusions and forecasts about it remain outside of a system. While people generally used to it, nevertheless it produces some inconveniences. Here are some of them:

1. It's not an easy task to find reasons for a KPI change in a certain period -- neither popular data visualization tools nor search engines are note capable to do this simple thing which is obviously needed in the business world.
2. Knowledge of influencing factors behind KPI trends is spread across emails, documents, IM messages (this makes p.1 even less easy).
3. As majority of influencing factors are rather qualitative than quantitative, so they remain out of decision-support systems (hmm... why are they called decision-support systems in this case?).

Thus, I've decided to make a tool that will deal with this problem. It's called Explainum. It is not in production yet -- we're just about to launch closed beta-testing. If you want to take part in it -- feel free to register.

Some of its features:

• Trend charts, which can automatically update data set with new data every day from various data sources -- CSV files, stock market data, currency exchange rates or web-services like Google Analytics
• Users can create/read comments for selected data regions. In order to find comments related to a certain time period users should simply select a rectangular area on a chart
• Charts can be embedded into 3rd party web-pages as interactive widgets

Read "What is Explainum?" for more detailed descriptions of the idea behind Explainum or see sample widgets. Here is screenshot of chart made using Explainum -- just to give you an idea how it looks like (clickable). As you can see -- commented areas are highlighted and chart has list of comments attached.

Teradata acquires Aster Data: Final switch to a new generation of analytical engines

Following HP's acquisition of Vertica, Teradata decided to buy Aster Data. The deal is a noticeable milestone -- now all major vendors of DWH platforms have switched to a new generation of analytical engines:

• Teradata will have Aster Data soon
• IBM has Netezza
• Oracle has Exadata
• Microsoft has SQL Server Parallel Database, going to have columnar storage in Denali
• SAP has Sybase IQ, Sybase MPP, Explorer Accelerated

Not to forget emerging players on DWH market:

• HP with brilliant Vertica
• EMC with Greenplum

The new generation  features (in various combinations) Massively Parallel Processing (MPP), columnar storages, hardware SQL acceleration, MapReduce, advanced in-database analytical functions.

Era of row-based SMP databases for analytical workloads on large datasets has gone. Don't miss the train.