Big Data’s Challenges

I was asked this week by a colleague, “What is your definition of Big Data”? Here’s how I responded.

The technology industry describes Big Data in technology terms such as Volume, Velocity and Variety of data or too much data to store and process in traditional data warehouse infrastructure. This might help make technologists aware of the challenges in data management around large, changing, and unstructured data sets. It helps technology vendors sell new capabilities to address these challenges, the media to rally CIOs to invest in them and build awareness around the challenges attracting talent to analyze them.

But that’s not a good business definition of Big Data.

Big Data for All Businesses

Every enterprise already collects lots of data, but under utilizes it for intelligence, insight, and decision making. Data exists in disparate databases that can’t easily be connected and analyzed holistically. Unstructured data in the form of documents, spreadsheets and presentations exist that are largely used for individual or departmental needs and rarely coalesced and analyzed. Enterprises that have deployed collaboration platforms have the opportunity to better leverage the networks and intelligence of its employees by analyzing relationships, contributions, and consumption on these platforms. Data collected in workflow solutions such as ERP and CRM are rarely merged and analyzed for intelligence. Email is certainly one of the largest repositories of unstructured data.

Bottom line is, all enterprises already have big data. By my definition, Big Data is not defined by its data management challenges, but by the organization’s capabilities in analyzing the data, deriving intelligence from it, and leveraging it to make forward looking decisions. It should also be defined by the organization’s capability in creating new data streams and aggregating them into its data warehouses.

Volume, variety, and velocity of data define the overall size and complexity of Big Data’s data management challenges and for some, the size requires a different architecture than traditional data warehouses. But the real Big Data challenge, and the challenge for all business regardless of size or complexity is in transforming to a data driven organization driven by analysis and insight of existing and new data streams.

See my previous posts on this subject including Big Data, Big CIO Opportunity, Big Data Needs to Scale, Big Data’s Managerial Challenges, and my Top Five Tools of Big Data Analytics.