Why self-service is everything in BI and big data

When it comes to BI, it still largely operates like it did at a company I worked for in the ’90s. A businessperson comes up with an idea, it requires some data, and she’s forced to ask IT not only to provide the data but often the report as well.

Usually this is because the data is in a form that makes writing a SQL query too difficult for the average Excel user. Often the data resides on a system with limited access, and many times, it sits on multiple systems that have to be brought together.

To be fair, today’s tools have become much more approachable, and they’re better at hiding much of the SQL as you create charts. Whereas Crystal Reports was usually in the hand of a specialist, Tableau is often in the hands of a business user or analyst.

Nonetheless, IT must usually be involved in part of the process because the data is in too many places. Even if the data is in one place, the query that must be written is too complicated.

The trick is to create views that serve groups of reports, which lets the business user understand how to slice things without having to write complicated queries. This is a high-touch process: Technical people must sit with business users and learn all about how they work with data.

If you ask me, this state of affairs is not good enough. I mean, having business experts spelunk through data to theorize what they might do with it certainly is important. But what about common problems? Questions like “Should I hire a new salesperson?” or the various concerns in building a sales and marketing machine don’t vary so much from company to company.

Where is the product that pulls this stuff together? For the most part you find a set of specialized products for single issues because your combination of operational systems create a schema unique to your company, and data transformation effort is a real thing that must be done.

That data transformation effort, while laborious, is worthwhile. Getting the data to people in a way that they can use it is the first task on the path to making your company “data driven” or “work smarter” or “more competitive” or whatever you want to call it. From there, bringing in the expertise to show how some of those common business elements are done elsewhere may be required.

Making visualizations business users create — and the processes they create out of them — into computer systems is the next step. Finding ways to do that in real time as opposed to batch is the competitive edge you’ve been looking for. But if you want to achieve that, IT can’t be in the direct path to getting the data — at least, not if any of this is to happen cost effectively.

Self-service really is the nirvana everyone thinks it is. A recent AtScale survey about the uptake of Hadoop stated BI was the killer app — and self-service was not only the goal, but the key to getting value. It’s time to get to work on that.