Using configuration files for QlikView applications

Often, development of QlikView applications requires (as any other IT project) several environments -- e.g. development, UAT, production, backup, etc. Usually, it means that QlikView application should connect to different databases, read/write QVD files to/from different directories, have different sets of users, etc. depending on environment. Changing or commenting/uncommenting parts of loading script every time is not so convenient way of doing things. Much more efficient approach is using custom configurations files. The general idea is to have different config files but the same QV app in each environment and read environment-specific settings from these config files into QV application. In this case it is possible just to copy application from one place to another place, without changing it. Also it helps to not mess up with version control, as you always have one version of application.

A configuration file can be a simple txt file, for example like this one:

Description=TEST environment
DBName=db_test
DBUser=db_user
XPassword=xxXpppAasssWwwOoorRrddDd
QVDPath='C:\QVD'


Create it in Notepad, name it, say, myapp.cfg and put to the same directory with your QlikView application.

In the application load parameters from configuration file. Sample script:

//Load configuration settings from config file which should be in the same directory as application

//Get application path
LET vAppFullPath = DocumentPath();
LET vAppName = DocumentName();
LET vAppPath = left('$(vAppFullPath)',index('$(vAppFullPath)','$(vAppName)')-2);

//Load config table
Config:
LOAD @1 as Parameter,
     @2 as Value
FROM $(vAppPath)\myapp.cfg (txt, codepage is 1252, no labels, delimiter is '=', msq);

//Assign variables
LET QVDPath = lookup('Value','Parameter','QVDPath','Config');
LET vDBName = lookup('Value','Parameter',DBName','Config');
LET vDBUser = lookup('Value','Parameter',DBUser','Config');
LET vXPassword = lookup('Value','Parameter','XPassword','Config');

Drop table Config;


Then you can use variables vDBName, vDBUser and vXPassword in database connection strings and vQBDPath in paths to QVD files.

Parameter Description can be used just for, well, description of a configuration file to distinguish one environment settings from another.

Tags, as we know them, are flawed

Using tags on web sites and in applications has become a common practice. You can find them everywhere -- in blogs (like this one), news and media sites, internet shops, discussion boards, Q&A sites and in many other places, including enterprise collaboration tools. Sometimes they're really helpful, but in the majority of cases it's just chaotic mess of words. Concept of tags was intended to facilitate the search for related content or to categorize things. However, actually it doesn't work or it works poorly. Why is it so?

The question is important for social BI tools as well, as they often feature tagging.

Too much freedom
Initially, idea of tags is very reasonable, as it employs associative thinking, which is natural for humans. I believe, it was supposed that users will choose right tags because they're interested in keeping information organized. While it can work for one person or a small team, it doesn't work for large groups, if only they don't have special and strict tag moderation policy. And usually, they don't. Rare exclusion -- Stackoverflow, they have very efficient system of tags, but they had to develop a whole self-sustaining social policy to achieve this. In other words, if you offer tagging to users, then you should either teach them how to use them right, or do not offer it at all. Well, SEO people like tags, but this is not a valid reason -- applications are made for people and not for search bots, at the end of the day.

Tag means nothing specific
Another cause of misuse is that tag doesn't have any specific meaning. There could be customers, products, blog posts or articles -- these are understandable entities. But what is tag? Is it a keyword? Then why this word is considered to be key? Why not other?

Word "tag" itself doesn't provide any direction for use. Actually, it just means "some random word subconsciously generated by user's mind in attempt to establish some memorizable associations with the item".

I can't clearly formulate what tag means, and I'm sure that vast majority of people also can't. So how can we use something in the system design, if we don't know exactly what it means?

Tags are unreliable
If there is no any common understanding what tags mean and how they have to be used, it is no wonder why tags do not do well what they were intended for -- obtaining related and relevant content. If users do not get expected result -- they won't use the feature. And users usually consider contextual search much more reliable than tags, even if it produces a lot of irrelevant information. So if tagging is not a reliable tool, why should users bother using it?

I don't know how tagging can be improved. But sooner or later, it has to, because of at least two reasons:

A) Theoretically, associative navigation should produce much better results than contextual search, if done right.
B) Amount of information, generated by society, grows exponentially, especially in social systems. Therefore, problem of signal/noise ratio will become more and more actual.

Video: annotations in Yellowfin BI

Example of annotations in BI tool. Technically well done, but social features and semantic value are rather basic. However, many BI tools lack even this.

BI Review gets imitators

It seems that my blog BI Review (which you are reading now) made some noise in the BI world as now I'm getting imitators who are not ashamed to pick exactly the same name for their domain which has been registered a few weeks ago and pretends to publish reviews about business intelligence. Moreover, that guys must be so shy, that they cloaked domain owner's name using anonymizing service from GoDaddy.

I've launched this blog only a few months ago with a lot of skepticism because English is not my native language and I never blogged in English, and because I live in small, retarded and corrupted country from "the rest of the world". Now every month more than 1000 readers find something to read here, I've got a lot of interesting connections and now I'm getting imitators -- I couldn't wish more when I just started :)

BI Review will always remain here, at this web address: http://bi-review.blogspot.com

Annotations in BI tools: why are they important?

I believe that the most important part of collaboration in Business Intelligence tools is capability to annotate data down to database row level. Why is that important?

Any data present in transactional and then in analytical systems usually reflects real-life events in business environment. "Primitive" events like purchase order, customer call, consumed content or service, etc. usually are well structured and explicitly represented in data models. However, there are much more important events and influencing factors at macro level that are not that obvious -- e.g.reasons of loss of sales, increased customer churn, government acts, competitor's actions or weather cataclysms. Such influencing factors and events usually don't get accounted in IT systems and do not exist in data models. However, their impact can usually be observed in key performance indicators change (actually, this is what KPIs are for). It means that KPI trends contain encoded information about influencing factors in specific time period. Role of good analyst or manager is to decode knowledge about influencing factors from data trends in current context. And then use this knowledge (often in a collaborative manner) for making right decisions. Therefore, KPI data itself doesn't represent big value -- it's just bits and bytes that mean something. But important knowledge extracted from this data -- does.

That's why any decent data analysis and data visualization tool should have capability of data annotations -- i.e. explicit explanations or comments made to specific subset of data. Here are a few considerations how it should be done, in my opinion:

Social Business Intelligence: Things start changing

I wrote earlier that BI vendors underestimated need for collaboration capabilities and social functions in their BI platforms. However, it looks like things start changing -- there are more and more news about upcoming social features and even full-scale collaboration platforms from BI vendors. Here are some of them:


Most interesting and innovative product comes from TIBCO (the producer of in-memory BI tool Spotfire with advanced analytical capabilities) with trendy name Tibbr (http://tibbr.com). Tibbr is actually a full-scale collaboration platform which plays in the same category as Yammer, but offers much more interesting approach, which definitely deserves separate review.


Not that innovative, but still solid and comprehensive social feature set was released by IBM a year ago, when they integrated IBM Cognos and Lotus Connections -- powerful IBM's collaboration platform. Capability of annotating data existed in Cognos Planning since ages, now it has reincarnated by means of Lotus Connections for the whole Cognos product family.


It's good to see that QlikTech also makes some steps towards developing collaboration features as well, as they're going to implement annotations in the upcoming 11th version of QlikView. Not that revolutionary as Tibbr but still better then nothing.

Small SaaS BI startup Lyza which made strong point on collaboration around data since early days seems also going to make next iteration of its web-based BI tool -- teaser on their web site speaks about this.

Explainum Feeds can use domain tags now

Small update to Explainum Feeds -- now you can use tags in extension properties, which are defined in properties' field Domain Tags. Now, when comments are made, they are attached to tags as well as to selection in QlikView fields. With the help of tags it is possible to merge or separate comment streams. For example imagine you have two applications (with the same extension token): one application with extension tagged "sales" and another one application where extension has tag "finance". Comments made in one application will not be displayed in the other, even if context fields and selections are identical. However if you create 3rd extension box and define in it's properties tags "sales, finance", it will display all comments from 1st and 2nd extensions according to context fields and selections.

Besides separation by business departments, tags can also be used for separation of comments by languages, regions, etc.

If you don't need this feature just leave default value in this field ("all").

If you downloaded extension in last 2 weeks -- no update needed. Extension script is updated automatically.

A few philosophical thoughts about QlikView extensions

This post represents some afterthoughts appeared after my attempt to create extension for annotating QlikView applications -- Explainum Feeds.

If you have read some of my previous posts then you probably noticed that I'm not among the most passionate QlikView fans. I worked long time with BusinessObjects and Cognos. I've seen some other BI tools -- Microstrategy, Tableau, Oracle BI to name a few, and my long-term enthusiasm about BI made me read numerous posts and articles about products and trends in BI industry (which looks stagnating now). So I've seen various approaches to solving the same tasks and not all of them were bad.

Without any doubts, QlikTech has offered interesting approach for data analysis -- I'm talking about so-called "associative data model". This is really fresh thinking (if we can call "fresh" an idea which went live 15 years ago) in rather narrow-minded BI world which seems long ago forgot what decision-support system means (if ever knew). I believe that company which once were able to create something original, sooner or later should be able to produce something really game-changing again. It looks like since inventing the associative in-memory engine long time ago, QlikTech made nothing innovative of the same degree again. Their old-school desktop app with cumbersome, overloaded properties forms looks outdated compared to industry peers who did much better job in terms of going web and polishing ergonomics. Recent QlikTech's attempt to create and use metadata have produced something raw and basic, which is years behind of what other have done. Security system still is a cry. Etc, etc.

But one thing has really big potential -- the extensions, which appeared a year ago in QV10. Extensions can become the next QlikTech's disrupting innovation in BI world. If not overlooked and done right, surely.

For those who don't know -- QlikView extensions is a framework based on Javascript API that allows creating custom data visualization objects in QlikView applications -- custom charts, maps, various gadgets and widgets, etc. In my previous post you can find some useful links and notes that can give more technical insight.

Extensions can do for QlikView what neither R&D investment nor sales staff can -- rapid transformation into ubiquitous application platform. It is known that QlikTech had long time rather small R&D team (this is one of reasons why QlikView's UI is not so modern). But extensions can in short term involve into product development many smart people with bright ideas which could never appear in walls of R&D labs. Who would have used iPhones if only Apple made applications for them? What would have happened to Twitter and Facebook if they didn't open API? Mega-popular Farmville and Angry Birds have been created by external developers, not by Zuck's engineers.

Of course, building developers ecosystem is not an easy task. Developers need well-documented, reliable and powerful API. They need clear and rewarding cooperation model. Both users and developers need convenient marketplace that would allow users to browse, install and update extensions and allow developers to find users, get feedback and may be earn some money.

Today QlikTech is still far from all of this. When we made Explainum Feeds -- many people have shown interest to it because many need annotations in their apps. But almost all of them lost interest when they knew that it works only in slow and inconvenient WebView mode and last releases are not supported because they have bug in the extensions' Javascript API. Also, building extension wasn't an easy task in the beginning because of lack of good API documentation and very limited feature set of Javascript API itself (because too long time QlikTech made bet on Visual Basic, as we know).

I'm sure QlikTech knows about all these problems. I'm sure they also think that extensions have big potential (and they're going to offer us something new in QV11). I hope they understand that this time in order to make outstanding innovation they need not only technical performance but organizational as well (by the way, QlikTech completely ignored my request for trial keys for QV Server which we needed to test the extension -- not the best way to support extension developers).

I hope they would be able to make something game-changing once again. Because not many revolutionary innovations are happening in BI industry today. Not too many to neglect them.

Building extensions in QlikView: some hints & tips

Here are some useful hints & tips that I learned while working over Explainum Feeds -- our extension for creating context-dependent comments in QlikView applications.

First of all, I would like to thank Stephen Redmond for his very useful Beginners Guide to QlikView Extension Objects (part 1, part 2, part 3, part 4) -- a mustread for everyone who starts developing extensions.

If you have done with that guide then these hints might be useful for you as well:

• Namings
• Debugging the extension
• Getting custom properties
• Accessing other objects
• Using external CSS stylesheets