As found in this icrunchdata News article: “The insatiable demand for data continues unabated. We want to gain deeper insights into market trends, customers, competitors and our business performance, but many companies are not making the progress they anticipated.”
“Why? Because most companies still don’t take a strategic approach to data integration. It’s laborious and time-consuming. It’s costly.”
Data integration is really nothing new. We’ve dealt with this problem since we had two or more data stores that needed to interoperate. However, lately, the need to leverage data in strategic new ways leads us to rethink the role of data integration, and the strategic value this technology brings.
As stated in the article, “Newer integration technologies that support data migration, app consolidation, data quality and profiling, and master and metadata management go beyond the traditional ETL functionality. These tools automate much of the cleansing, matching, error handling and performance monitoring – processes that IT teams often struggle with manually.”
The core challenge that enterprises face is the need to think more strategically about their data, and thus their data integration. Most enterprises struggle with data integration projects that eventually fail, or, worse, they succeed enough to drive some value, but they leverage cumbersome technology to deploy and operate, and provide no means for expansion or change.
So what’s the right way to approach data integration? It’s really a matter of understanding where you are holistically, in terms of where the data is currently held, where the data needs to be, and what state it needs to be in. If it sounds simple, it actually is. However, massive planning needs to go into understanding the source data so the integration flow can be properly designed, and the data delivered to the target data stores in the correct structure with the correct content.
Beyond the data integration planning is the selection of the right technology set. Think about all of the core requirements, and then match those requirements up with technologies that go the farthest toward solving the most problems. This means that we think long and hard about the fit, understanding that it’s difficult to back some of the technology out once it’s implemented.
Finally, and most important, is the testing and deployment. The data should be correctly produced, put into the right state for transport, correctly changed in structure and content, and ultimately delivered to the right data source in the right format.
Other issues to consider include security, governance, compliance, management, monitoring, and other items that may relate to your specific vertical market. Failure to address these issues means the data integration solution won’t have a long and productive life. Which is really the objective here.
So, what are you waiting for? If you found this article interesting, then you likely have the need to strategically leverage data within your enterprise or organization. The good news is that these are very solvable problems, and just require a bit of thinking and an understanding that any effort and money spent will come back in strategic value 10 fold. Time to get started.