July 12, 2018

Data Preparation is the new Business Intelligence

What's now happening with Data Preparation bears a strong resemblance with the state of Business Intelligence in the late 90s. There is a number of similarities between the data prep market now and BI back then:

  • Emerging technology
  • Designed for the convenience of business users (which is very atypical for enterprise software, btw)
  • Introduces a degree of independence from IT departments (which also means a lower burden for the latter)
  • Low general awareness of this type of tool in the market
  • General underestimation of the proposed value ("why can't we just use Excel for this")
  • A rare "System-to-human" kind of enterprise application. Not "Human-to-system" (front-end) or "system-to-system" (back-end) software commonly found in the enterprise.
  • Designed and suited for mass user adoption, rather than for use by 1-2 people in a department or even entire company.
  • From a licensing perspective the total cost of ownership is within $40-100 per month per user.
Having this analogy, it may be possible to predict certain future trends for the Data Preparation industry:
  • User adoption will broaden slower than anticipated by Data Prep vendors. It will probably take 10-15 years to reach the "late adopter" stage and market saturation.
  • User adoption will be wider than first customers (early adopters) envision it.
  • There will be attempts for "Data Prep standardization" across organization but they will fail, just as BI standardization failed.
  • Enterprise administration/governance features will become necessary sooner rather than later.
  • Authoring will shift from desktop clients to web-browsers.
  • Expensive software (>$100/mo/user) will be squeezed out from the market by competition.
  • There will be a wave of consolidation when major Data Prep companies are acquired by big enterprise vendors.
For me, a confirmation of this similarity is that most of multi-user installations of EasyMorph tend to grow over time doubling in about 2 years, on average. We've recently seen a customer that requested 30 user licenses just for one business line. This tells me that although data prep tools are close to ETL systems from a technical standpoint, from a user adoption perspective they clearly resemble Business Intelligence and Data Discovery applications.