The Evolving Perception of Big Data – My Personal Experience

Sometimes you need to throw yourself out of your comfort zone to see what is really happening.

Over the past few months, I’ve been on the road presenting at events and meeting with clients (both current and prospective) across the globe. Now I’m no stranger to travel (having trekked similar international routes back in my data warehousing heyday) but this big data thing that is going on – is a wee bit different.

Over the past year, I’ve seen a major shift in the perception and adoption of a big data approach. Coming in to the job last year – I was constantly bombarded with complaints of ‘This is just hype…” “…big data is a horrible name…” “What is the 4^th V?… it should be this, not that…” “ Hadoop is big data…  Hadoop will take over the warehouse…”  “NoSQL means NO SQL and thus NO Database…”

<sigh>

For the most part – All rubbish.

A year later (and still getting my fair share of naysayer voice) I ended my tour in Warsaw, Poland delivering a keynote at the Computerworld Big Data event. Now, any time you have to present in English in a non-native English speaking country, you have to take into consideration the speed of your speaking, the usage of slang and the overall ability of the audience to comprehend your words. The week before I was in Japan using a translator after each sentence. This being my first time to Warsaw to present to an audience of over 300 (where the majority of sessions were in Polish …mine had a little asterisk next to it ‘in English’ on the conference brochure which did not promote confidence), I was assuming that conveying the overall message would be a crapshoot. Boy was I wrong.

Step back a year ago while presenting at a TDWI (The Data Warehousing Institute) event…. I asked the audience  “How many of you have heard about ‘Hadoop’?” Like one or two hands would go up…  I asked the same question here… and two thirds of the room raised their hands. Granted, this was a ‘Big Data’ event… but so was the theme of that TDWI event… Times had changed over the past year in a big way.

After my presentation I spoke with numerous conference attendees and the questions and topics were night and day different to what I was hearing a year ago. No more was it ‘What is Hadoop’…or ‘What is big data (and some crazy deep dive on the Vs and which ones were missing…)?’ It was more around…”So as you illustrated the use case for Data Warehouse Augmentation… Where did the actual loading of data occur?” or even better… “We have already starting using this Hadoop technology, yet we want to get a more real time perspective on our users… How can we do this?” Here I was in Poland at the end of winter (yup – cold and snowing), thinking that my talk might miss the mark – and I am engulfed by questions and briefings on actual usages of big data. This was awesome.

We all might not agree on what big data is, or what the big data challenge is at any one organization, but we can all agree that technology has been (and still is being) developed to answer questions and tap into data that was one deemed impossible. This is the big data challenge in my eyes.

But while specific technologies (like Hadoop) are fantastic… they are simply just one facet of a very complex answer. Business intelligence and reporting is irrelevant without trusted data underneath… A data warehouse appliance can be a foundation for new insights, but on its own – it is a just a dumb rack with disk. But together – all of these technologies combined – can offer the next frontier of insight – answers to problems that once were left unsolved.

It brings me back to that discussion in Warsaw that I had with a customer on how to augment their big data solution. They were already neck deep in using Hadoop to sort through clickstream data on their customers – yet could not leverage it fast enough to make a difference to their customer base. They were looking not just for a 360 degree view of their customer – but also for a way to reach out to their clients (with targeted offers) in real time.

This was the fantastic story that we were telling years ago when we built in continuous ingest into the data warehouse and brought InfoSphere Streams to market… yet here (last week) in Poland, a customer was coming to me with the same exact story. Awesome

As I sit in my office today and finish up on my trip reports (really missing that sushi breakfast at the Tsukiji fish market in Tokyo the other week) I’m increasing excited for 2013 and the advances that we will make this year. Organizations around the globe are making data a priority and leveraging the latest in technologies to ensure that they are making better decisions. What I have seen in meetings and briefings so far has been nothing short of astonishing. Clients are coming to the table with some absolutely innovative ideas on how they want to leverage big data technologies – ideas that could (and will) make front page on Wired magazine one day as they are developed and implemented.

It’s going to be a fun year in big data.

Changes at 30,000 feet en route to Japan

At the moment I am up in the air and about 5 hours away from touchdown in Tokyo, Japan. I’m heading out to our Japan offices for a number of client meetings, including some key sales enablement presentations.. At the moment, I’m amazed that I even have my laptop open in these horribly small seats (yeah, no upgrade today) – but if I’m not able to get some sleep, I might as well work.

I’ve noticed that my blogging skills (and actual postings) have been a little lacking lately – much due to the tradeoff of work and family balance (…can’t justify a blog entry on customer analytics, when my son is asking to play chutes and ladders on a Saturday afternoon) – but per one of my goals of this year, I’m going to make a more consolidated effort to document my travels, thoughts and communications. I have over 30 hours of HD footage from my stay in Nanning, China last May, and I only just offloaded it off of my camera. Sad.

One of the changes on the horizon will be the location of the blog. At the moment I’m using the free wordpress.com site to host my blog and other ramblings. In the near future, I will be moving to my own server to maintain a more granular control over the site, as well as create a portal for more interesting endeavors (not sure what they are yet, but the option is there)

The second of the upcoming changes will be the introduction of video media. Words only take us so far, …and well lots of this stuff that I talk about on a daily basis would be drastically assisted by some examples, live action and at least a diagram or two. We will see how this pans out, but bear with me as I migrate through the transition.

Lastly, for the purpose of my personal blog, I’m going to look to expand and alter my scope. To this point in time, I’ve become a bit stale trying to blog about technologies that I market, discuss and present all day, every day… So we will mix it up a bit to keep it fresh and include more intersections of day to day life, travels and customer experience – yet all supporting the drive for making better decisions. Will see where it ends up.

Not everything has to be some uber techo architecture manifesto when you talk analytics or big data – and I will attempt to maintain a simple, tangible approach where I can.

On to the hours of presentation, meetings and translations, yet need to try and get some sleep in on this flight – else I will be awake for 48 hours and I’m unfortunately not 18 years old any more – will make for an interesting business dinner upon my arrival.