September 14, 2018

How I came up with the idea of EasyMorph

One of the most frequent questions I'm asked is "How did you come up with the idea of EasyMorph?".

Throughout my career of Business Intelligence consultant I could observe how non-technical people work with [structured] data. Even 15 years ago, their setup was pretty much the same as it can be commonly found these days: an IT team generates various extracts from transactional and accounting systems, which then are loaded and processed in Excel by business users without a technical background.

All in all, it worked rather efficiently. The IT part of this setup was apparently automated as much as possible, while the business part remained entirely manual with the exception of a few occasional VBA macros here and there. While this approach had some drawbacks, it was assumed a reasonable compromise between those who are proficient in data manipulation but don't have an actual need for it, and those who need it but have limited technical skills.

However, eventually I started noticing that the complexity of calculations is growing. Loading a text extract and calculating a few simple metrics from it in Excel is one thing. But loading several files, fixing data quality issues, applying various business rules and calculating complex metrics is another thing. Complexity (as well as probability of human errors) increases particularly sharply when there is a need for merging (joining) two or more tables, because Excel doesn't do joins natively. Joins need to be emulated which is a not so trivial task in Excel, especially when you deal with datasets of varying length and potential problems with uniqueness of primary keys.

I guess, this growth of complexity can be explained by two simultaneous trends:

1) Slow explosion of data sources as more and more processes became automated. As software keeps "eating the world", more data collected, more metrics tracked, and more things monitored.

2) Centralized IT-led initiatives such as data warehouses couldn't keep up with this explosion. The task of consolidation of data coming from various sources has been shifting to a great extent from IT to business users, effectively becoming decentralized instead of centralized.

For me, the turning point was a project in a large North American bank (which later would become our 1st customer). In that project, I had to deal with an Excel report that was produced on a daily basis with 3 text extracts merged, various business rules applied and multiple metrics  calculated. Due to the complexity of calculations, the VBA macro that did all the processing was very obscure and had more than 1000 lines. The financial analyst who created it had moved to another country and was no longer accessible. There was no documentation on it. It took me a few weeks to reverse-engineer the business logic behind the script. A few errors were found that caused some metrics to calculate incorrectly.

There is no point in blaming the author of the report for the errors, obscurity, and the lack of documentation. After all, finance analysts are not supposed to do high quality coding/scripting. Financial analysts are supposed to do financial analysis, not software development.

For me, it became clear that there should be a better way of doing things. And then EasyMorph was born.