One thing that has struck me recently is that most of the focus when discussing big data is upon the technologies involved. The consensus seems to be that the biggest challenge with big data is a technological one, yet I don’t believe this to be the case. Sure, there are challenges today for organizations using big data, but, I would like to submit to you that technology is not the biggest problem. In fact, technology may be one of the easiest problems to solve when it comes time to tame big data.
The fact is that there are tools and technologies out there that can handle virtually all of the big data needs of the vast majority of organizations. As of today, you can find products and solutions that do whatever you need to do with big data. Technology itself is not the problem.
Then, what are the issues? The real problems are with resource availability, skills, process change, politics, and culture. While the technologies to solve your problems may be out there just waiting for you to implement them, it isn’t quite that easy, is it? You have to get budget, you have to do an implementation, you have to get your people up to speed on how to use the tools, you have to get buy in from various stakeholders, and you have to push against a culture averse to change.
The technology is right there, but you are unable to effectively put it to work. It FEELS like a technology issue since technology is front and center. However, it is really the cultural, people, and political issues surrounding the technology that are the problem. Let me illustrate with an example.
A few weeks back, I was at the annual Teradata Partners conference having a really good discussion with someone from one of our larger customers. When I suggested to him that his struggle with big data isn’t a technology problem, he challenged me. He pointed out that the network protocols they have in place just can’t support the level of data flow required to handle the big data they need to utilize. He felt he had a very real technological issue.
I pushed back on him and asked if there weren’t widely available products that would solve his network bandwidth problem tomorrow if he were to implement them. He said that was true but that he couldn’t just go and implement them tomorrow. I then pointed out to him that he had just proven he didn’t have a technology problem. The technology exists. He knows what it is and how to put it in place. He just can’t get it in his hands.
His real problem is with a culture that doesn’t want to invest in upgrading the network at this time. His real problem is a bureaucratic budgeting process that takes a long time to navigate. His real problem is building a business case to prove the need and value of making the upgrade. All of these problems are very real for my client and are also very valid. But, they aren’t about technology. Brush aside corporate policies, give him cash, and give him purchase authority today, and his problem will be solved very quickly.
I suspect that as many readers consider the story above, they will realize that they don’t really have a technology problem either. Technology is the easy part. It exists and it can be put in place as soon as you’re ready. However, those pesky budgets, policies, skill gaps, and cultural realities get in the way of that happening.
As your organization proceeds with the process of incorporating big data, be sure to give appropriate attention to the non-technical hurdles that you’ll face along the way. Don’t mistake them for a technology problem and don’t underestimate them. These hurdles will often be much more difficult to clear than finding a set of tools and technologies that will do what you require.
A few things to keep top of mind:
Think through the challenges you’ll face in getting your organization to effectively use the big data tools you’ve pushed so hard to acquire.
Plan to help your organization embrace the business model and process changes that might be suggested by results that your new tools and analytics uncover within the new data sources.
The fact that the tools and technologies to tame big data are readily available today is a terrific thing and you need not lose sleep on that front. However, there may be a lot of sleep loss as you figure out how to get your organization to actually acquire, implement, and put the tools and technologies to use. The people, cultural, and political issues surrounding big data are very big and very real.