My speech at IDC BI Roadshow

I did opening speech at IDC BI Roadshow in Kiev a week ago. Here is my presentation with some remarks.

• SQL was initially designed as tool for business analysts as well, but it quickly became too complex so it was clear that special tools are needed
• In 90s BI wasn't like a platform but rather like a set of tools for data retrieval and visualization
• Lots of M&A on BI market in 2000s led to formation of Enterprise BI platforms
• Low user adoption is the main BI problem which is still not resolved; complexity and traditional pricing models are among main obstacles
• World goes mobile and social, so should BI
• Traditional OLTP databases for analytical workload? No more
• BI standardization has failed -- there is no perfect BI suite under the moon
• There were many attempts to overcome low user adoption (Pervasive BI, Operational BI, etc.) but all of them failed. Let's see if Data Discovery will succeed or it is just a new hot buzzword
• Finally BI vendors started to pay attention to good data visualization, no more silly 3D bar charts. Small BI vendors lead the race here.
• Business users need data manipulation capabilities, whether IT like it or not
• Collaborative and social features in BI suites are greatly underestimated and underdeveloped by vendors, however they might have the key to dramatically higher user adoption
• Data warehouses will not disappear, despite hasty statements of some BI vendors, because DWH is not about databases, it's about abstraction layer that transforms data models of many diverse transactional systems into single business data model

Using configuration files for QlikView applications

Often, development of QlikView applications requires (as any other IT project) several environments -- e.g. development, UAT, production, backup, etc. Usually, it means that QlikView application should connect to different databases, read/write QVD files to/from different directories, have different sets of users, etc. depending on environment. Changing or commenting/uncommenting parts of loading script every time is not so convenient way of doing things. Much more efficient approach is using custom configurations files. The general idea is to have different config files but the same QV app in each environment and read environment-specific settings from these config files into QV application. In this case it is possible just to copy application from one place to another place, without changing it. Also it helps to not mess up with version control, as you always have one version of application.

A configuration file can be a simple txt file, for example like this one:

Description=TEST environment
DBName=db_test
DBUser=db_user
XPassword=xxXpppAasssWwwOoorRrddDd
QVDPath='C:\QVD'


Create it in Notepad, name it, say, myapp.cfg and put to the same directory with your QlikView application.

In the application load parameters from configuration file. Sample script:

//Load configuration settings from config file which should be in the same directory as application

//Get application path
LET vAppFullPath = DocumentPath();
LET vAppName = DocumentName();
LET vAppPath = left('$(vAppFullPath)',index('$(vAppFullPath)','$(vAppName)')-2);

//Load config table
Config:
LOAD @1 as Parameter,
     @2 as Value
FROM $(vAppPath)\myapp.cfg (txt, codepage is 1252, no labels, delimiter is '=', msq);

//Assign variables
LET QVDPath = lookup('Value','Parameter','QVDPath','Config');
LET vDBName = lookup('Value','Parameter',DBName','Config');
LET vDBUser = lookup('Value','Parameter',DBUser','Config');
LET vXPassword = lookup('Value','Parameter','XPassword','Config');

Drop table Config;


Then you can use variables vDBName, vDBUser and vXPassword in database connection strings and vQBDPath in paths to QVD files.

Parameter Description can be used just for, well, description of a configuration file to distinguish one environment settings from another.

Tags, as we know them, are flawed

Using tags on web sites and in applications has become a common practice. You can find them everywhere -- in blogs (like this one), news and media sites, internet shops, discussion boards, Q&A sites and in many other places, including enterprise collaboration tools. Sometimes they're really helpful, but in the majority of cases it's just chaotic mess of words. Concept of tags was intended to facilitate the search for related content or to categorize things. However, actually it doesn't work or it works poorly. Why is it so?

The question is important for social BI tools as well, as they often feature tagging.

Too much freedom
Initially, idea of tags is very reasonable, as it employs associative thinking, which is natural for humans. I believe, it was supposed that users will choose right tags because they're interested in keeping information organized. While it can work for one person or a small team, it doesn't work for large groups, if only they don't have special and strict tag moderation policy. And usually, they don't. Rare exclusion -- Stackoverflow, they have very efficient system of tags, but they had to develop a whole self-sustaining social policy to achieve this. In other words, if you offer tagging to users, then you should either teach them how to use them right, or do not offer it at all. Well, SEO people like tags, but this is not a valid reason -- applications are made for people and not for search bots, at the end of the day.

Tag means nothing specific
Another cause of misuse is that tag doesn't have any specific meaning. There could be customers, products, blog posts or articles -- these are understandable entities. But what is tag? Is it a keyword? Then why this word is considered to be key? Why not other?

Word "tag" itself doesn't provide any direction for use. Actually, it just means "some random word subconsciously generated by user's mind in attempt to establish some memorizable associations with the item".

I can't clearly formulate what tag means, and I'm sure that vast majority of people also can't. So how can we use something in the system design, if we don't know exactly what it means?

Tags are unreliable
If there is no any common understanding what tags mean and how they have to be used, it is no wonder why tags do not do well what they were intended for -- obtaining related and relevant content. If users do not get expected result -- they won't use the feature. And users usually consider contextual search much more reliable than tags, even if it produces a lot of irrelevant information. So if tagging is not a reliable tool, why should users bother using it?

I don't know how tagging can be improved. But sooner or later, it has to, because of at least two reasons:

A) Theoretically, associative navigation should produce much better results than contextual search, if done right.
B) Amount of information, generated by society, grows exponentially, especially in social systems. Therefore, problem of signal/noise ratio will become more and more actual.

Video: annotations in Yellowfin BI

Example of annotations in BI tool. Technically well done, but social features and semantic value are rather basic. However, many BI tools lack even this.

BI Review gets imitators

It seems that my blog BI Review (which you are reading now) made some noise in the BI world as now I'm getting imitators who are not ashamed to pick exactly the same name for their domain which has been registered a few weeks ago and pretends to publish reviews about business intelligence. Moreover, that guys must be so shy, that they cloaked domain owner's name using anonymizing service from GoDaddy.

I've launched this blog only a few months ago with a lot of skepticism because English is not my native language and I never blogged in English, and because I live in small, retarded and corrupted country from "the rest of the world". Now every month more than 1000 readers find something to read here, I've got a lot of interesting connections and now I'm getting imitators -- I couldn't wish more when I just started :)

BI Review will always remain here, at this web address: http://bi-review.blogspot.com

Annotations in BI tools: why are they important?

I believe that the most important part of collaboration in Business Intelligence tools is capability to annotate data down to database row level. Why is that important?

Any data present in transactional and then in analytical systems usually reflects real-life events in business environment. "Primitive" events like purchase order, customer call, consumed content or service, etc. usually are well structured and explicitly represented in data models. However, there are much more important events and influencing factors at macro level that are not that obvious -- e.g.reasons of loss of sales, increased customer churn, government acts, competitor's actions or weather cataclysms. Such influencing factors and events usually don't get accounted in IT systems and do not exist in data models. However, their impact can usually be observed in key performance indicators change (actually, this is what KPIs are for). It means that KPI trends contain encoded information about influencing factors in specific time period. Role of good analyst or manager is to decode knowledge about influencing factors from data trends in current context. And then use this knowledge (often in a collaborative manner) for making right decisions. Therefore, KPI data itself doesn't represent big value -- it's just bits and bytes that mean something. But important knowledge extracted from this data -- does.

That's why any decent data analysis and data visualization tool should have capability of data annotations -- i.e. explicit explanations or comments made to specific subset of data. Here are a few considerations how it should be done, in my opinion:

Social Business Intelligence: Things start changing

I wrote earlier that BI vendors underestimated need for collaboration capabilities and social functions in their BI platforms. However, it looks like things start changing -- there are more and more news about upcoming social features and even full-scale collaboration platforms from BI vendors. Here are some of them:


Most interesting and innovative product comes from TIBCO (the producer of in-memory BI tool Spotfire with advanced analytical capabilities) with trendy name Tibbr (http://tibbr.com). Tibbr is actually a full-scale collaboration platform which plays in the same category as Yammer, but offers much more interesting approach, which definitely deserves separate review.


Not that innovative, but still solid and comprehensive social feature set was released by IBM a year ago, when they integrated IBM Cognos and Lotus Connections -- powerful IBM's collaboration platform. Capability of annotating data existed in Cognos Planning since ages, now it has reincarnated by means of Lotus Connections for the whole Cognos product family.


It's good to see that QlikTech also makes some steps towards developing collaboration features as well, as they're going to implement annotations in the upcoming 11th version of QlikView. Not that revolutionary as Tibbr but still better then nothing.

Small SaaS BI startup Lyza which made strong point on collaboration around data since early days seems also going to make next iteration of its web-based BI tool -- teaser on their web site speaks about this.