Data Preparation is the new Business Intelligence

What's now happening with Data Preparation bears a strong resemblance with the state of Business Intelligence in the late 90s. There is a number of similarities between the data prep market now and BI back then:

• Emerging technology
• Designed for the convenience of business users (which is very atypical for enterprise software, btw)
• Introduces a degree of independence from IT departments (which also means a lower burden for the latter)
• Low general awareness of this type of tool in the market
• General underestimation of the proposed value ("why can't we just use Excel for this")
• A rare "System-to-human" kind of enterprise application. Not "Human-to-system" (front-end) or "system-to-system" (back-end) software commonly found in the enterprise.
• Designed and suited for mass user adoption, rather than for use by 1-2 people in a department or even entire company.
• From a licensing perspective the total cost of ownership is within $40-100 per month per user.

Having this analogy, it may be possible to predict certain future trends for the Data Preparation industry:

• User adoption will broaden slower than anticipated by Data Prep vendors. It will probably take 10-15 years to reach the "late adopter" stage and market saturation.
• User adoption will be wider than first customers (early adopters) envision it.
• There will be attempts for "Data Prep standardization" across organization but they will fail, just as BI standardization failed.
• Enterprise administration/governance features will become necessary sooner rather than later.
• Authoring will shift from desktop clients to web-browsers.
• Expensive software (>$100/mo/user) will be squeezed out from the market by competition.
• There will be a wave of consolidation when major Data Prep companies are acquired by big enterprise vendors.

For me, a confirmation of this similarity is that most of multi-user installations of EasyMorph tend to grow over time doubling in about 2 years, on average. We've recently seen a customer that requested 30 user licenses just for one business line. This tells me that although data prep tools are close to ETL systems from a technical standpoint, from a user adoption perspective they clearly resemble Business Intelligence and Data Discovery applications.

Why enterprise software is switching to subscription-based pricing

Business Intelligence and other enterprise software vendors are switching to subscription pricing en masse. Microsoft probably championed the shift when they introduced Office 365 and Power BI. Tableau recently announced switching to a subscription-based pricing model. At the time of writing this article the monthly fee for Tableau Desktop Professional was $70. Qlik and some other BI vendors have introduced subscriptions as well.

While software vendors are apparently pushing the trend, the customers sometimes have mixed feelings about it. On one hand, the significant reduction of upfront licensing costs makes rolling out new software deployments faster and with less risk -- you can start with purchasing only a few licenses and see how it goes. In the worst case, you just cancel the subscription instead of turning expensive licenses into shelfware.

On the other hand, in the long run subscription based pricing appears to be more expensive. In the previous pricing model, Tableau Desktop Professional cost $1999 paid once. I don't remember what was the maintenance fee, but for the industry the typical rate is 20-25% per year. If we assume 25% maintenance, in 5 years the total cost of ownership would be $3998 per user, while under the new subscription model the cost will be $4200. In a 10 year term the difference becomes even more significant - $6500 vs $8400 per user, at least on paper.

Is switching to subscription-based pricing just a marketing gimmick to squeeze more money out of customers? I don't think so, and here is why:

First of all, in a highly competitive market vendors can't squeeze more money from customers simply because a) competition won't miss a chance to undercut pricing, and b) the amount of money (market size) remains the same, no matter what pricing model is applied.

If so, why the change? As someone who runs a company that also employs a subscription based model, I believe the answer is sustainability.

The problem with the one-time pricing model is that it came from the times of industrialization, when the economy was based on physical, tangible goods. It's not the only business pattern that has been inherited from that epoch. The 8-hour workday from 9 to 5, and the need to commute to the workplace every workday have also originated from those times, because you know it was kind of problematic to do industrial-scale cast iron melting working remotely from home. Everybody needed to be at the factory and work hard with their hands.

Producing physical, tangible goods required lots of materials and some labor too. In the cost structure, the part of materials was typically much bigger than the cost of labor. Therefore, one-time pricing in such economy was logical because the cost was mostly driven by materials.

In the modern, post-industrialization economy some of the old models don't work well anymore. For many knowledge-based professions, such as software development, working fixed hours 9 to 5 or commuting to the office every day is becoming increasingly irrelevant, if not counter-productive.

A similar thing is happening with pricing models too. Labor by its nature is subscription-based, because an employee isn't get paid a lump sum of money upfront and then is expected to work forever without additional pay. Instead, s/he is paid a salary which is basically a monthly or weekly subscription to the worker's services.

In software development cost, labor comprises the biggest share. Therefore the expense structure for a company is predominantly of subscription nature. At the same time, having revenue structure that is based on one-time payments introduces financial instability and risk that needed to be offset by higher pricing and/or more conservative product development strategy.

With that in mind, switching to subscription model totally makes sense for software vendors as it allows to offset subscription-based expenses with subscription-based revenue and achieve better financial sustainability for the company. It also works well for the customers, as the vendors can now be less conservative in R&D spending which means that users will receive better products sooner.

UPDATE 2/2/2019
Switching to subscription-based pricing has an interesting ethical aspect as well. One-time pricing creates for enterprise sales people a [wrong] incentive that is not aligned with the interests of customers. Under the traditional pricing model the sellers quickly lose interest in customer success after the sale is closed. For them, the sale IS the success, while for the customer the success is what is supposed to only happen AFTER the sale. A subscription-based pricing model aligns the interests of sellers and customers as the sellers now have more "skin in the game" which is attractive to customers.

Unplugged: What I learned about people and technology while writing QViewer

This post is not about BI technology or vendors or data analysis. "Unplugged" is a new kind of articles in this blog. It's my personal observations with a bit of philosophical contemplation, if you will. Today I'm writing about what I've learned while developing and selling QViewer -- a side project started as a "quick and dirty" tool for personal use which then became an interesting business experiment that taught me a few new things about people and technology:

You can make and sell things as your side project. It's doable. I remember that somewhat awkward feeling when I received first ever payment for software that I made myself. It was very unusual. I had experience of selling enterprise software with a 6-digit price tag, but that was someone else's business. Getting your own first sale in a $50 range was no less exciting if not more.

People in general are good. Once you start selling software you interact with people all over the world. And it turns out that people are generally good around the globe. I was surprised how many very grateful and positive people there is. Probably it's the most unexpected and gratifying outcome of the whole project.

Some people cheat with licenses. Despite the fact that the cheapest QViewer license costs less than a dinner for two, and, unlike the dinner, is acquired forever -- they still cheat. I understand it's a part of the human nature -- feeling frustration and pity when someone steals from you and at the same time enjoying the benefits of stealing from someone else even if it's just pocket money. People are complicated animals. So I'm not saying anything about the people that cheat. I'm deeply content that the majority is honest. The humanity definitely has a chance to survive :)

Some people are strange. No need to deal with them. After all, doing business is a two-way street. I remember one person demanded a sales presentation, a webex demonstration and sending him a detailed commercial proposal for QViewer because he and "his guys" wanted to evaluate whether it's worth spending couple hundred dollars. I replied that I'm ready to answer any specific questions, and offered to try the free QViewer to get an idea about the product. I've never heard from him again.

95% of technical support time is spent on 5% of customers. Some people are just like that -- they don't read instructions, forget things, don't check spam folders before complaining that their key didn't arrive, can't figure out what instance of QViewer they're launching, etc. It's OK, they just need more help. After all, adults are just grown up kids.

User recommendations is the best advertisement. So far I've spent exactly $0 for advertising QViewer. Yet, it's quite popular, mostly because of user recommendations. For me it was a good example of what it looks like when you made something useful. If people recommend it to each other -- you're on the right path.

1 out 10 orders never paid. Spontaneous decisions, no problem.

Payment reminders work. Sometimes, your invoice sent to a customer may be buried in his/her email box under a pile of unread messages. Sending a friendly reminder once might help. Just once, that's enough for those who are really looking to buy.

Even small side projects can be extremely good for career opportunities. Needless to say, mentioning QViewer in my CV helped me tremendously in finding new employers (when I looked for them). I would argue that the salary increase it enabled has earned me more than selling QViewer licenses alone.

Developer tools are amazing nowadays. I wrote my first program in 1986. It was in BASIC on a military-grade DEC-type computer. In 90s I wrote programs in C++, Pascal and Assembly. Between 1998 and 2011 I didn't write a single line of code (except some Excel macros). Boy, how things have changed since then. When I started writing QViewer in 2012 I was totally fascinated with the capabilities of Visual Studio and C#. Later I fell in love with F# but that's a different story. And thanks God we have StackOverflow. Writing software has never been easier.

Obfuscate and protect your software. Sooner or later someone will try to disassemble your software for a purpose that might be disappointing for you. There is no absolute protection, but raising the barrier can significantly complicate the task. Once I interviewed a developer for EasyMorph. Trying to impress me, the guy told me that he also wrote a QVD viewer. However, after not answering a few questions about the QVD format he quickly admitted that he just disassembled and repacked some components of QViewer. I learned a lesson that day.

Writing and selling software changed my perception of the software industry. I understood what it takes to create it. I stopped using any pirated programs. Now I use only licensed software, even if it's rather expensive (I'm looking at you, Microsoft), and I always donate when a program is available for free but donations are accepted.