Post by Bob Corr
Today’s crowded BI marketplace is packed with vendors jostling for a stall to vend their wares. Some have been around for more than a quarter of a century and some are relative infants. To the uninitiated, the companies that make up this mix can, at first glance, all seem to be offering a different version of the same solution; the same engine with different bodywork. Some BI products have gained traction and have grown rapidly – maybe too fast for their own good, like an infant that shoots up but hasn’t yet filled out and matured enough to leverage his height advantage.
Shiny New Things
Some of the newer players made their entrance noisily trumpeting their power, speed, data volume capabilities, superior data visualisations etc. Decision makers being presented with the razzmatazz of such a product’s user interface can be wowed into opting for the shiny new thing without considering the realities of how it actually performs as a mature enterprise-wide solution. Too late into the deployment do they discover not the promised insights from their data, but “cracks” beginning to appear in the product’s ability to integrate seamlessly all the data from disparate sources that makes up their business.
It is generally accepted that a successful BI deployment requires an ETL layer where data is consolidated from disparate sources and “fed” to the BI product. ETL stands for Extract, Transform and Load and the main purpose of the ETL layer can be defined as follows:
- Extraction of data from legacy or other sources which are usually heterogeneous.
- Data Transformation – for example transforming data that is optimised for transaction processing into data that is optimised for analysis and reporting.
- Synchronisation and cleansing of the data plus the ability to apply complex business rules.
- Loading of the data into a data mart or data warehouse or other data model.
But let’s consider the scenario above, where a company has opted for the “shiny new thing”. Once the initial integration is rolled out with, maybe, one set of clean data, the first flurry of excited users see how much power they have to create their own insights, and they soon develop an appetite for more. And crave more. They soon realise that in order to get a complete view of what is going on in their businesses, they need to add in those key spreadsheets with budgets or that market data that doesn’t reside in the ERP system, so they start demanding more information from different sources.
Two Options Leading where?
So how do BI providers, whose products fail to cope with synchronising and joining data, solve this issue? There seem to be two paths they can go by – but in this case unfortunately neither of them lead to BI’s Stairway to Heaven.
- Some providers develop their own ETL software “after the fact” and bolt it on to their products OR
- Partner with a third-party provider of back-end ETL software to plug the gaps and cover up their product’s shortcomings.
The problem with option 1 is that if the ETL layer is not developed in conjunction with all the other components of the offering prior to general release then there is inevitably a huge risk of dissatisfaction within the user community – post initial deployment. It will most likely kick in as soon as the users request more data from disparate sources. The provider will often supply promises of rapid development of their ETL layer, but in all probability, what usually happens when trying to restructure a product after the fact is that there will be considerable delays in delivery. These will deepen disappointments at the user level as well as adding to frustrations of the BI developer.
This approach may also mean that there is a long waiting period before the ETL layer is truly scalable and feature rich enough to be classed as enterprise wide. In the intervening period the developers may have to use their own initiative to create “plasters” to “glue” together what is only perceived to be an enterprise wide deployment.
For this type of BI provider, option 2 might therefore seem like the best path – partnering with a known, trusted ETL provider. Initially in this scenario, the requirement for the BI product to satisfy the data integration needs of an organisation, especially in a complex business area, will be satisfied.
Ask Yourself These Questions
However, if you are considering purchasing a BI product with this type of collaborative approach to delivering an End-to-End Solution, there are a number of questions I recommend you to ask yourself:
- How is the third-party back end ETL integrated into the BI product?
- How is the additional cost passed on to the customer?
- How are issues resolved between the front-end and the back-end?
- How does this affect the provider of the BI software (most likely to be a partner of the BI developer)?
- What are the knock-on effects to the end user organisation?
- What happens if the ETL provider partners with another BI provider or goes into the BI marketplace with its own BI offering?
- What happens if the ETL provider company is purchased by a third-party organisation (especially if it is one that competes directly with the incumbent BI software)?
I am asking you to consider these questions as we are aware of at least a couple of competing BI vendors using the same back end software provider. As the cost of third-party dedicated ETL products is comparatively very expensive, I for one would be concerned about procuring a BI product from a vendor whose margin is compromised by having to pay a third party for their ETL layer. Why? Because it limits their scope for in-house development, to enhance the existing modules you were “wowed” by initially, to respond to user requests and keep on developing the product in line with current and future trends.
According to some industry reports ETL work can consume 60 to 80 percent of all BI projects. So if you are considering a BI deployment my advice is to choose a vendor that has:
- Developed their ETL layer in conjunction with the front end
- A proven track record over many years
- AND who continually develop their software to embrace new technologies when they emerge as well as constantly providing enhanced functionality to their user base.
The Good News
The good news is that The Diver Platform from Dimensional Insight (DI) falls into this category. The Diver Platform is an enterprise wide BI product that was initially released in 1989 and is constantly being updated and developed by DI.
DI develop and own their own software and do not include third-party software in their product. The product is multi-platform, scalable, robust and extremely feature rich. The Data Integrator module combined with their superior Production module provide end-user organisations the ability to rapidly develop and deploy powerful BI solutions in a fraction of the time that competing vendors would take.
The end user community of The Diver Platform are clearly fans of both the product and the support from the company, as DI consistently outscore their competitors in industry wide user surveys such as KLAS or Dresner’s Wisdom of Crowds survey.
See more on these here.
About Bob Corr
Bob Corr is Managing Director of DBI and set up DBI in Feb 2003. Prior to this Bob was MD of System Options Ltd. and has been involved with The Diver Solution and Diver Platform since 1994. Bob has been designing and implementing Diver applications since 1994, and is particularly adept at integrating Diver with new systems and engineering proof of concept applications. He has worked with many clients; mainly those with multi-national operations where BI applications are required to bring information together in one place. Bob has a proven wealth of business experience, delivering applications that permit companies to save time and money, and make better educated decisions that are critical to their success.
