Clearing up the Confusion
In the third blog post in this series, the DBI team attempt to clear up some common misunderstandings about the terminology, technology and concepts used when people are discussing analytics or considering the implementation of a Business Intelligence platform.
The technology landscape is constantly changing, and the vocabulary is expanding so fast it can be challenging to keep up – it’s no wonder that people get confused. It becomes important to ensure you know where some common areas of confusion arise when you are relying on analytics software or applications to fulfil all your expectations.
Not every option you’re faced with does what it says on the tin – or if it does, are you sure you understand what that is before you buy it? You might be half-way through painting your equivalent of the Forth Bridge before you realise the quality is poor, or the colour is all wrong and you’ll have to start again – before you even finished the first coat.
So many of the misconceptions around Business Intelligence arise because it’s a multi-faceted discipline which involves technology, data science and business knowledge. The best implementation teams know that all three need to be strong and working together to produce meaningful results.
1. Bob Corr (Managing Director DBI)
Difference Between BI Software Platform v BI Applications
Talking about software and applications, they can seem in some circumstances to be interchangeable. But generally when we talk about BI software or a BI Platform, we mean the underlying suite of programs that has been developed and packaged in a kit or bundle that can be installed on a server as a set of developer tools and a data model engine. These are to be used by application developers and to a certain extent, administrative users.
The engine can be very powerful or perform badly “under the hood”. The tools might have a lot or very few features, and the way you use them can be more automatic or manual depending on what engine you choose.
The software platform itself generally has no intrinsic value to an organsation until application developers create scripts and data models that extract, transform and load your data, to present it back to you in raw data models, reports, dashboards and indicators. The platform provides tools for each step of this process – again some are easy and some are more difficult to use.
By contrast, a BI application is a completed data transformation product that is the result of taking your data and transforming it into an accessible format that can be viewed by various different application client types, (e.g. web interface or desktop executable) often via a secure logon account. It can have defined reports and dashboards and allow for some flexibility for users to carry out self service “data discovery” – see point 3 below. It can also be packaged to work seamlessly for any organisation providing they can map their data inputs exactly to those required by the application.
2. John Spillane (Technical Director DBI)
Difference Between Data v Insights
There can be confusion when a business first starts to consider a BI implementation between the ability to get at the raw data and the insights that can be derived from it. Simply plugging in a data extraction tool to an ERP or other database and pulling out tables doesn’t enlighten anybody with information. The raw data is just the starting point and it can come out in varying standards of “cleanliness”, depending on how well your database is managed.
Presenting this raw data in an accessible format is not delivering insights on the data, it’s simply allowing sight of the data.
In order to create something of benefit that will give you a return on investment you will need to take several steps after the first extraction process.
- Compare the basic numbers delivered in the raw data output model with those you have in your ERP system to ensure that nothing is being missed and you have a baseline to start designing from with confidence.
- Work together with business users and BI developers ensure that any business rules are applied to make more sense of the data. This might involve scripting some new calculations that cannot be done in the ERP, or looking up some external mapping tables to groups data together in a more meaningful way. We often create new fields that are the results of some logic, or bucket larger volumes of data into bands like hours or periods instead of just dates.
- Design some reports that can compare data sets, either by time, period, or different groups, e.g. one supplier or customer compared to the whole group, using percentage changes or percentage of the whole.
- Using graphs and charts discover patterns or breaks in patterns in the data that can inform you of unpredicted changes in your business direction.
- Publish exceptions and KPI reports that allow you to take action on outliers to the expected, or to targets.
- Set up automated sharing of the insights with those who have the ability to take action on them.
3. Paul Duggan (Software Development Manager DBI)
Different Interpretations of Self-Service BI
Self-service BI is a term that means something different depending on who you talk to. So here are some differing explanations of what the term means, so you can ensure you know what to look for when you are considering self-service BI.
When I think about offering “Self-Service BI”, I know that we will have gone through some important steps before this can be achieved. We will have carried out the above-mentioned ETL steps and will have also delivered one or more “clean data” models that have been validated and have the critical business rules applied. There will be some standard reports, KPIs and dashboards available to users and these will be dynamically updated on a timed refresh.
There are usually some more advanced users in each customer site that want more autonomy in being able to take a cleaned data model and interrogate it in multiple ways, without any constraints of hierarchy or predefined reports. They can save their own reports and even publish them to the web platform if they like, creating their own pages. In short, it’s the ability to empower users to work on deriving new insights without constant interventions by developers, who have already done their job by delivering clean and validated data.
Sometimes people are offered a “Self-Service BI Tool” which involves an individual downloading some BI software and plugging it into their own data with the guidance of the software “Help” pages.
They might expect to create accurate reports without first fixing all the data issues that we generally solve during the ETL and validation phase of a project. This can be very frustrating and disappointing, as almost all BI projects require some sophistication in the cleaning of data, and a certain level of data literacy to get the best out of the tools provided.
Using these tools without any form of data cleansing or data governance can ultimately lead a business into more confusion as time is spent trying to understand why the reports don’t reflect reality. They might not be able to rely on reports derived from this type of self-service tool to make fully informed decisions.
In our experience, using the Diver Platform has enabled us to solve these issues so we can deliver clean data models, allowing our customers to carry out their self-service analytics with confidence.
4. Debbie Lonsdale (Senior BI Consultant DBI)
Software as a Service BI (SaaS) v In House BI
Sometimes we find that when we offer Software as a Service BI it’s not really clear what the difference is between SaaS and In-House BI.
In our offerings, when we talk about SaaS we generally mean that whichever of our BI applications you are interested in using, you will be provided with a secure, managed and hosted application which you can access in the cloud. For this you will pay monthly or annually for any development time; for the space that your data takes up; and for user licenses. We also provide all scripting services, report creation and dashboarding services, and these are included in the annual fees, for which we manage the full application on your behalf.
In addition it’s important to be aware that a SaaS application can be either a bespoke application which has been designed for a specific customer need, or it can be a package where the user interfaces are the same for multiple customers and their data needs to be tailored to work with the outputs that we deliver.
Although we can allow some autonomy within an SaaS application for users to carry out some self-service and data discovery, there is in most cases, no option for anyone who wants to use the full set of tools available in the platform’s ETL modules. It’s not the best fit for those companies who want to manage and develop their own applications using all the features of the full software platform and work with a variety of different data sets.
If this is the case, we usually recommend that the platform software is installed on your own in-house server (or your own cloud server), and you will usually purchase the software platform outright on a perpetual license. User licenses are also perpetual, although there are annual maintenance fees. Once installed your own developers can create as many applications as they want using all the tools. Often our customers require a hybrid collaboration with our developers helping with the design, and with training, at least initially.
5. Anthony Wright (Senior BI Consultant DBI)
Integrating Data from Disparate Sources
The scope of what this phrase can mean and how much work might be involved is often underestimated.
Integrating disparate data sources can mean simply combining small data sets from, for example, a few spreadsheets that just need to be mashed together in an automated way. It can mean including data held outside of the main ERP system when working on the transformation stage of an application development project. This can typically be budgetary information held on spreadsheets, or other lookup tables that you might use to reorganise the data structures for reporting purposes.
These are the run-of-the-mill type of data integration combinations we carry out in nearly every script we write – and we write a lot!
There are some more complex scenarios where the ability to combine multiple data sources into a single data model can save our customers from having to carry out costly data migrations if they want to upgrade to a new ERP system. Or, if they are happy with the existing one, they might want to enhance it by combining it with CRM data for example. This can be more challenging as the goal is to makes all the data appear to have come from the same source and appear to use the same field names and use the same logic.
It can often involve trying to map incompatible definitions from the various sources, e.g. what is a customer – is it the account that pays the bills or the account where product is delivered? Do they have the same codes? If not, then they need mapping to a common agreed code. What category is a product in – do they use the same categories in all sources? Is the supplier the default supplier or the one that provided the last delivery? And then there are the measures – do sales include delivery costs in all systems? There can be fields that are completely missing in one of the systems – and you can no longer rely on these in existing reports.
Sometimes it’s not just the data formats and meaning that are at odds. The accessibility of the data from newer systems, especially in cloud-based applications, is more challenging than older systems where an ODBC driver can be set up directly into the data. APIs are not always so friendly to a BI developer, and access might only be available at certain times or for certain tables. Sometimes the sources can be out of synch time-wise, and so only some of the data sets are available for the latest extracts across the whole set of data sources.
All of these questions have to be resolved or at least understood and managed as part of integration projects.
