On January 24, 1848, carpenter James Wilson Marshall found gold while building a sawmill in California. That date marked the beginning of the very first period of gold rush. The gold people sought had been in the same place for millions of years, but this was the first time that it had been dug up by humans and put to good use. For many organisations it is now 1848: there is a lot of data that has been available for some time, but it is only now becoming easier to make that data usable and to add value to it.
In many organisations, data resides in all kinds of systems that use different storage methods. Every system that is used, contains valuable data. However, that data is not necessarily immediately usable. A number of steps will have to be taken to get the required data from the various systems. Below I discuss the different steps that need to be performed and how this ultimately leads to a (more) data-driven organisation.
Step 1: Executive buy-in
Executive buy-in is needed to extract value from the systems and receive the correct data. In different departments of an organisation, actions need to be taken to acquire the right data. Without this executive buy-in, there will be too many roadblocks to create a feasible project.
Step 2: Collect data
This step happens naturally: the longer a company exists, the more unstructured information it collects in systems. For example: support emails, information about the products sold or financial information about the purchase of the products. Each system used has its own storage method, so correlating that information is difficult. Sometimes people are not even aware of the existence of certain information.
Step 3: Data analysis
In order to extract value from the data of step two, an analysis will first have to be performed. What are the most important issues within the business that can be answered with information from different systems? What data exists? What systems is it in? This analysis seems easier than it is: it requires subject matter experts as well as technically savvy people who understand the possibilities for data extraction.
Step 4: Extract Transform and Load Pipeline
After it is decided which data should go where, an Extract Transform and Load (ETL) pipeline can be created. In such a pipeline, data is extracted from the systems in a fully automated way (the Extract part of ETL). This data then often has to undergo corrections, additions and conversion (the Transform part of ETL). Finally, that data must then be stored in a structured way in the Datalake (the Load step of ETL). After these steps have been completed, it is time to unlock the data.
Step 5: Business Intelligence Dashboard
The information from the steps above will have to be put together to extract value from it. This can be done in the form of a business intelligence dashboard, which makes complex data understandable for users. This dashboard is ideal for stakeholders to get an overview of the performance of the organization at a glance and to make immediate adjustments where necessary. In this way, organisations can achieve better results in the long run.
Governance & Security
After these steps have been completed, it is necessary to maintain the project. There must be a person that is responsible for continuously maintaining the dashboard and the underlying ETL pipelines. Last, but certainly not least, it is very important to pay attention to security and compliance, so that the “gold” remains safe from the beginning of the journey to the end.
A transformation to a data-driven organisation is complex and requires careful planning and preparation. When this change is made within an organisation, the organisation benefits optimally from the data it collects and it is worth its weight in gold.