Big Data: Information is Not Knowledge

Say that you have a log file of every contact to your website. Or that you are a cell phone company with a multi–billion record log file of every call made from every location to every number dialed using your network. You can accurately state that you have big data at your fingertips. But it is useless in this raw, usually unstructured form.

Adding context to data takes this information and creates actionable knowledge. There’s even a new term for this: thick data. That’s data dense with information that is useful and can be mined with available tools to create actionable strategies. Google analytics provides this information at no cost for websites and pages viewed. Google uses proprietary tools to do this quickly, with resultant information available within a day.

But once you find a similar need for other types of logged information, you find yourself in the need for a data scientist or data engineer, and the costs climb accordingly. Still, there is value in getting help from professionals who can provide real-time log management solutions from your observability data.

Such knowledge can be amazingly invasive in the wrong hands or if used without consideration for the result. For example: What if American Express offered credit card details about purchases by competitors to a big box retailer? Could Sam’s Club find which households were shopping at Target and even know the annual spend amount by family? Would that be invasive – or just a great targeted marketing resource? Would you be happy to receive coupons from Sam’s Club knowing that they found you and your shopping habits from your credit card company?

And yet, there are times when strategic marketing demands information that is readily available in files owned and controlled by you. There are tools and a new generation of experienced data engineers ready to unlock this treasure chest of big data and turn it into actionable knowledge. You’d be remiss to ignore the opportunity; one that even recent generations of management could not dream of gaining access.