How important is data quality to your Big Data strategy? I suspect that once you have business leaders asking good data driven questions and have the beginnings of big data tools implemented, data quality and governance will become a bigger business priority.

Microsoft has the most successful Big Data tool – it’s called Excel. Most business users will jump into Excel first to do any kind of data discovery, analytical analysis, or visualization. Along the way, they may blend in multiple data sources, create formulas, cut/paste data into multiple worksheets, format the worksheet for presentation purposes, create pivots, or save the file into a private repository. Many of these steps make auditing the analysis or reusing the results in larger contexts a challenge for business managers, data scientists or technologists.

Recently, I had the opportunity to ask a Microsoft representative some questions on their Microsoft Excel 2013 and Office 365 products to see if their new functionality helps or hinders data quality practices.

Excel 2013 – With great power comes great responsibility

Excel 2013 comes with some new functions that make data scientists more self sufficient. Tools that were formally part of Microsoft Access are now finding their ways into Excel including creating Excel Data models. Pro versions of Excel also include Power Query to easily discover and connect to data from public and corporate data sources. Excel 2013 also combines in Power Pivot to perform analysis and visualizations as well as some new data quality functions. Excel 2013 Power Query also comes with a set of tools to perform some basic data merging and quality steps.

So with all these new functions, what’s not to like? So much power in the hands of sophisticated data scientists who are able to do advanced analytics without IT’s help.

The issue is, that Excel always made it too easy for business users to create splintered and derivative data sets. Pull data from enterprise databases, integrate other data sources, complete the analysis, and present findings – but no where in this workflow is there an easy way to incorporate the data, formulas, and other rules back into a central repository to be shared with others. This is a big issue for companies that are trying to leverage Big Data to transform their business or to become a more data driven organization. It is why I asked developers to Please Stop Creating Microsoft Access Databases because the work to integrate or cleanup a siloed database can be considerable.

The Unofficial Microsoft Response

So I asked the Microsoft rep for some answers, “How can businesses implement basic governance when the most popular business analytics tool makes it so easy to create derivative data sources?” Here were his answers

• IT can revere engineer the Excel Worksheets – I stopped him quickly, almost laughing at this answer, because it’s completely not realistic. Maybe your business users create simple spreadsheets, but if there are complex formulas, pivot tables, or copy/pasted data then good luck. Also, most organizations have many more spreadsheet jockeys versus DBAs, so it’s highly unlikely that IT can keep up.
• Saved spreadsheets to Office 365 are discoverable – That’s a better answer. At least IT knows where files exist and how frequently they are used. In theory, better defined spreadsheets can get reused by others in the organization.
• It’s coming in Reporting Services – Well, I’m not sure what is coming…

The new capabilities in Excel, if used without some discipline pose new data quality challenges. So why isn’t Microsoft doing even more in their tools to insure analytics can be reproducible, audit-able and centralized? My sense is that is that this may not be in Microsoft’s best interests. Microsoft has many competitors at the database, BI, Big Data, reporting and visualization technology spaces, and less (no?) competition for analytic tools that compete directly with Excel. So why make it easier to move the analytics outside of this tool?