When explaining big data to executives, the 4 Vs paradigm of volume, velocity, variety and veracity (introduced by IBM) does a good job illustrating how to handle the exploding growth in information processing. But the very first question one often hears from business leaders is, “So what technology do we need to buy to make big data work for us?” How does one recommend a technology strategy in an increasingly complex landscape full of uncertainty?
In the past, enterprise data technology choices were rather simple. If you wanted a database, you would go to IBM, Microsoft, Oracle or Sybase. If you wanted a reporting and analytics engine you would go to Actuate, Business Objects or Hyperion. You could rarely go wrong by selecting one of the big enterprise vendors, as they differed from each other in only a handful of features and you, the enterprise, would end up selecting one or two platforms. Your selection often was guided by acquisition costs, compatibility with your existing technology investments and the relationship between your IT and the vendor’s sales team. Furthermore, that choice would stay with you for many years, and sometimes decades, because the rate of radical change in this space was rather moderate.
Big data has changed all that. If there was ever a “disruptive” technology, this is it. But our big data paradigm, the 4 Vs, can illustrate not only what is happening to data, it can help us make sense out of making big data technology investments.
1. Volume: Increase in number of choices
With open source as a major driving force behind big data technologies, the cost barrier to become a commercial provider in this space has dramatically decreased. Today it seems that anyone with basic technology understanding can download a database capable of handling trillions of transactions, for free. Or just open an account on Amazon’s cloud and start one up using only your credit card. This extremely low market-entry cost has encouraged a plethora of startups over the past few years, offering various advanced yet experimental solutions and offerings that are geared to make these platforms customer friendly, more functional and useful.
Big data has become so hot that all the enterprise vendors are integrating it into their own offerings to be able to keep up and retain their existing clients. So in addition to the proliferating startup market, you now get sales call from companies ranging from IBM to EMC, Microsoft, Oracle and SAP to steer you towards their big data solutions
Action: The key is to identify your short list rather quickly and bring those providers into your lab for test drives, which are often available through a 90 day evaluation license.
2. Variety: Increase in innovation choices
This new breed of technology vendors is augmenting the big data platforms with anything from smart-storage file systems to encryption, compression, security, management, operations, reporting, analytics and other complementary products. They are aimed at rounding up your solutions so that you will have to build less on your own and concentrate your efforts on delivering integrated intelligence that empowers your decisions. The private equity firms are spending a lot of their investors’ money seeding many startups in the hopes that a handful of them will return a good multiple on their investment. But until this happens, vendors and solutions are mushrooming. Inevitably, many will fold, some will be acquired and others will go public, just like any consolidation wave in technology. Until then, there seems to be numerous incredibly innovative solutions with very compelling value propositions.
Action: Come up with the minimal set of initial components needed for proving your solution. Start with the functional assessment aspects to get your business’s stakeholders attention early on.
3. Velocity: Increase in speed to market
Every morning when I read the news, I see even more vendors joining the big data play. At this pace, enterprises become very confused when it comes to making a stable, long-lasting decision on partnering. Enterprises are beginning to recognize that the booming tech universe surrounding big data technologies will rapidly morph, and they can’t afford to wait and see who makes it and who doesn’t. Under pressure from their business imperatives and outside market influences, they need the intelligence to deliver value now. In this case, time is money.
Action: In this new era of technology disruption, you have to think in short timeframes as you advance your evaluation and decision-making process. Decisions made today will be obsolete in 1.5–2 years, not the traditional 3–5. Investments need to be amortized in a shorter 12–18-month timeframe.
4. Veracity: Not all will survive, and that’s ok
When presented in the context of big data, Veracity is a term (introduced by IBM) to illustrate the uncertainty factor that data in large mass may present. It can help companies become more comfortable making decisions with less than perfect data, given their need to respond quickly to market conditions. When it comes to big data vendors, veracity plays a similar role. Long gone are the days where a firm would select and invest in a platform intending to stick with it for the next five years. At the industry’s current pace of change, enterprises must face the reality of what happens if they wait, forcing the current shopping cycle for platforms to compress. They are recognizing that some decisions will be made with a short-term benefit in mind while waiting for something that will be available in a year and that may be the ultimate solve-all whiz-bang. Making a partially good decision based on a partially mature marketplace is often better than not making a decision and waiting for the perfect solution.
Action: Broadcast up, down and across your organizations that decisions about the future merit of technology choices you make today are meant to be revised and updated frequently and that failure is an expected and natural part of this process if you want to remain competitive in your market. This is not the time to be 100% certain that you picked the right tools, or you will inevitably lag behind waiting for the perfect one. And in this age of big data, things are moving way too fast to be that vulnerable.
About the Author
Avi Kalderon is NewVantage Partners’ practice leader for Big Data and Analytics and heads the NewVantage Partners New York Office. He has extensive experience as a business leader and technology executive, most recently serving as senior vice president and head of the architecture and advanced technology group for FINRA (the Financial Services Regulatory Authority), the largest independent securities regulator in the U.S.