Why Do We Need a Data Warehouse?

A lot of times people question the value of data warehousing. Why do we spend 1 year building a data warehouse? We can’t wait that long. Let’s just install QlikView/Spotfire and feed the transaction system direct to it and we have a BI!

Absolutely! You can. You can buy BO, MicroStrategy, QlikView, Spotfire or any BI tool you like, then report straight from the transaction system. Or, if you fancy, you can create a cube first (SSAS, Cognos or Hyperion), then install appropriate client tool (Tableau, Strategy Companion, etc). Of course you can. And this is the best way to learn about Data Warehousing: by not doing it.

What you will experience is:

a)    Data quality issues
b)    Low confident level from users
c)    Quick turn around of report but data is unstable
d)    Issues with data consistency
e)    Issues with performance

The whole year spent on building a data warehouse essentially for providing a quality data source. A data warehouse has the following characteristics:

a)    Integrated
b)    Consistent
c)    Contains historical data
d)    Tested and verified
e)    Performant

A data warehouse integrates data from multiple sources correctly. This integration doesn’t happen overnight. A Business Analyst spent weeks analysing the sources and wrote down a specification of how the data should be integrated. A Data Architect looked at that spec and designed a performant star schema to host the data. An ETL Architect looked at the star schema design and wrote an ETL population spec. An ETL developer studied the ETL spec and built the workflows. And finally, a tester verified the data.

That takes months, but as a result, we have integrated, consistent, clean data source containing the correct and valid data. And it is performant. Your query doesn’t need to join 15 tables in a horrible way. All the data is in a centralised place, ready for you to query.

But, as I said, experience is the best teacher. Once you experience how difficult it is to build business intelligence without a solid data warehouse, then you will truly appreciate the value of a data warehouse.

PS. I have just posted another article with the same title (but different): link