Not big data but lots of small data

I am certain that my views in this area are informed and shaped by my early consulting experiences. I was fortunate enough to work with experienced professionals that knew how to acquire, integrate, and analyze a wide range of related but disparate datasets. They never worried about bringing together a couple of databases to create new information and analytics. They saw this activity as quite natural. As a consequence of spending so much time with them and seeing the incredibly powerful and insightful results, I too came to believe that integrating data together was an incredibly valuable activity that provided tools and insights that other people just did not see or even think of.

In the beginning, we would integrate datasets that were obviously related. We would integrate shipments information from the client company systems with the consumption data from grocery stores. Consumption data or scanner data from Nielsen and Information Resources...