Archive
The Evolving Perception of Big Data – My Personal Experience
Sometimes you need to throw yourself out of your comfort zone to see what is really happening.
Over the past few months, I’ve been on the road presenting at events and meeting with clients (both current and prospective) across the globe. Now I’m no stranger to travel (having trekked similar international routes back in my data warehousing heyday) but this big data thing that is going on – is a wee bit different.
Over the past year, I’ve seen a major shift in the perception and adoption of a big data approach. Coming in to the job last year – I was constantly bombarded with complaints of ‘This is just hype…” “…big data is a horrible name…” “What is the 4th V?… it should be this, not that…” “ Hadoop is big data… Hadoop will take over the warehouse…” “NoSQL means NO SQL and thus NO Database…”
<sigh>
For the most part – All rubbish.
A year later (and still getting my fair share of naysayer voice) I ended my tour in Warsaw, Poland delivering a keynote at the Computerworld Big Data event. Now, any time you have to present in English in a non-native English speaking country, you have to take into consideration the speed of your speaking, the usage of slang and the overall ability of the audience to comprehend your words. The week before I was in Japan using a translator after each sentence. This being my first time to Warsaw to present to an audience of over 300 (where the majority of sessions were in Polish …mine had a little asterisk next to it ‘in English’ on the conference brochure which did not promote confidence), I was assuming that conveying the overall message would be a crapshoot. Boy was I wrong.
Step back a year ago while presenting at a TDWI (The Data Warehousing Institute) event…. I asked the audience “How many of you have heard about ‘Hadoop’?” Like one or two hands would go up… I asked the same question here… and two thirds of the room raised their hands. Granted, this was a ‘Big Data’ event… but so was the theme of that TDWI event… Times had changed over the past year in a big way.
After my presentation I spoke with numerous conference attendees and the questions and topics were night and day different to what I was hearing a year ago. No more was it ‘What is Hadoop’…or ‘What is big data (and some crazy deep dive on the Vs and which ones were missing…)?’ It was more around…”So as you illustrated the use case for Data Warehouse Augmentation… Where did the actual loading of data occur?” or even better… “We have already starting using this Hadoop technology, yet we want to get a more real time perspective on our users… How can we do this?” Here I was in Poland at the end of winter (yup – cold and snowing), thinking that my talk might miss the mark – and I am engulfed by questions and briefings on actual usages of big data. This was awesome.
We all might not agree on what big data is, or what the big data challenge is at any one organization, but we can all agree that technology has been (and still is being) developed to answer questions and tap into data that was one deemed impossible. This is the big data challenge in my eyes.
But while specific technologies (like Hadoop) are fantastic… they are simply just one facet of a very complex answer. Business intelligence and reporting is irrelevant without trusted data underneath… A data warehouse appliance can be a foundation for new insights, but on its own – it is a just a dumb rack with disk. But together – all of these technologies combined – can offer the next frontier of insight – answers to problems that once were left unsolved.
It brings me back to that discussion in Warsaw that I had with a customer on how to augment their big data solution. They were already neck deep in using Hadoop to sort through clickstream data on their customers – yet could not leverage it fast enough to make a difference to their customer base. They were looking not just for a 360 degree view of their customer – but also for a way to reach out to their clients (with targeted offers) in real time.
This was the fantastic story that we were telling years ago when we built in continuous ingest into the data warehouse and brought InfoSphere Streams to market… yet here (last week) in Poland, a customer was coming to me with the same exact story. Awesome
As I sit in my office today and finish up on my trip reports (really missing that sushi breakfast at the Tsukiji fish market in Tokyo the other week) I’m increasing excited for 2013 and the advances that we will make this year. Organizations around the globe are making data a priority and leveraging the latest in technologies to ensure that they are making better decisions. What I have seen in meetings and briefings so far has been nothing short of astonishing. Clients are coming to the table with some absolutely innovative ideas on how they want to leverage big data technologies – ideas that could (and will) make front page on Wired magazine one day as they are developed and implemented.
It’s going to be a fun year in big data.
Changes at 30,000 feet en route to Japan
At the moment I am up in the air and about 5 hours away from touchdown in Tokyo, Japan. I’m heading out to our Japan offices for a number of client meetings, including some key sales enablement presentations.. At the moment, I’m amazed that I even have my laptop open in these horribly small seats (yeah, no upgrade today) – but if I’m not able to get some sleep, I might as well work.
I’ve noticed that my blogging skills (and actual postings) have been a little lacking lately – much due to the tradeoff of work and family balance (…can’t justify a blog entry on customer analytics, when my son is asking to play chutes and ladders on a Saturday afternoon) – but per one of my goals of this year, I’m going to make a more consolidated effort to document my travels, thoughts and communications. I have over 30 hours of HD footage from my stay in Nanning, China last May, and I only just offloaded it off of my camera. Sad.
One of the changes on the horizon will be the location of the blog. At the moment I’m using the free wordpress.com site to host my blog and other ramblings. In the near future, I will be moving to my own server to maintain a more granular control over the site, as well as create a portal for more interesting endeavors (not sure what they are yet, but the option is there)
The second of the upcoming changes will be the introduction of video media. Words only take us so far, …and well lots of this stuff that I talk about on a daily basis would be drastically assisted by some examples, live action and at least a diagram or two. We will see how this pans out, but bear with me as I migrate through the transition.
Lastly, for the purpose of my personal blog, I’m going to look to expand and alter my scope. To this point in time, I’ve become a bit stale trying to blog about technologies that I market, discuss and present all day, every day… So we will mix it up a bit to keep it fresh and include more intersections of day to day life, travels and customer experience – yet all supporting the drive for making better decisions. Will see where it ends up.
Not everything has to be some uber techo architecture manifesto when you talk analytics or big data – and I will attempt to maintain a simple, tangible approach where I can.
On to the hours of presentation, meetings and translations, yet need to try and get some sleep in on this flight – else I will be awake for 48 hours and I’m unfortunately not 18 years old any more – will make for an interesting business dinner upon my arrival.
Saint John Bound – The T4G Big Data Congress
Headed out this evening up to Saint John, New Brunswick for the T4G Big Data Congress. I’ll be up there speaking on a panel with associates from a number of Hadoop vendors including Cloudera and Hortonworks on our respective big data platforms.
It will be interesting to see how the audience is currently interpreting big data and the challenges that they face, and I will share my perspectives from the event in the coming days.
I’m personally looking forward to hearing from Tom Davenport, who will be speaking with us as well. Competing on Analytics came out as I was kicking off my 1st tour of duty for IBM in the warehousing space, productizing the old ‘balanced configuration unit (BCU)’ into our first data warehouse ‘appliance’, the Balanced Warehouse. Yeah seems like a decade ago. It will be very interesting to hear his current view on big data in his session and the associated application of analytics.
I’ve never actually been up to Saint John, but have already been warned by the local office that the temperatures will be a bit on the cold side. From 0 to -15 to be exact. Ouch. Reminds me of the MONY tower back in the day in Syracuse – Would always be interesting to make a run to Marshal Street at that negative temperature mark flashing in the sky.
Advancing big data technologies for the end users
Last week we formally announced and GA’d a slew of new core big data offerings in our big data platform (There was a reason that I have been quiet/offline) and I thought it would be a great time to share them with you all. We started the discussion of our new technologies at the Information on Demand conference at the end of October – but they are now all fully baked in the marketplace.
The 3 new offerings are:
I’ll plan on digging deeper into each one of these offerings over the next few posts – but in summary, we are building out a platform portfolio that is unmatched in the world of big data. We are making it easier for organizations of all sizes to leverage and exploit ‘big data’ to make better decisions.
On the hadoop front, BigInsights not only updates and includes the latest support/versions for the Apache hadoop initiatives but also starts implementing technologies from across the big data platform. In addition to making hadoop more enterprise (rather than a standalone, open source project) BigInsights 2.0 offers a slew of advanced visualizations and tools for users across the organization.
With InfoSphere Streams 3.0, making decisions in real time has just become easier. While Streams has always incorporated a rich programming language (spl) not every user has had the time and effort to master it on the fly. With version 3.0, InfoSphere Streams now incorporates a visual GUI ‘drag-and-drop’ interface to program your own streams… and yes, that interface also generates the proper code as well so that you can alter and enhance granularly as well.
Last but not least, the Vivisimo acquisition in late Spring has already been integrated into the portfolio with the new InfoSphere Data Explorer 8.2 (formally known as Vivisimo Velocity). This offers fast ROI and minimizes risk by helping customers understand their big data assets and unlocking the value – including federated search – leaving data where it is BEFORE you determine if you are going to use it/analyze it….
Yeah – I’m a product guy, and well – new things are cool – so I get a little nerded out when we release offerings like this. In my next blog installments, I’ll drill into each one of these offerings and show you why we added the features and capabilities that we die (Yes, we did listen) – and most importantly how it helps you with your big data initiatives.
Sorting Through the Clutter
Entering into this world of Big Data headfirst, I am overwhelmed with the amount of buzz and hype surrounding the topic. The other day I read the article ‘How Big Data Became so Big’ by Steve Lohr via the NY Times website and it really set the stage for the world’s challenge of Big Data. You know something has hit the big time when Dilbert references it in passing.
Per my previous post, I do not view Big Data as a product (or as a group of products), but instead as a challenge that organizations face in their journey to analyze ALL of the data made available to them to make better decisions. Hadoop is one tool to get there – yet not the only one. Over the years we have gone from machine readable punch cards to petabytes of data stored on an array of different disk types – commodity through high performance solid state.
Great – lots of storage for data – more clutter – just like my email account. Could end up being an episode of hoarders for techos.
It’s not just the analysis of the data that is important (think a superfast data warehouse appliance cranking through queries – ala Netezza) but also the determination if the data is actually worth being stored. It is like one big garage sale. There is so much to dig through, so many items old and new – You sure as heck are not going to take it all home with you – as most of the items are garbage and not needed – they would just sit around in your house (warehouse that is) and waste premium storage – and perhaps trip you up on the way to the car (or to your analytical appliance) that you have revved and raring to go
This is where hadoop comes in handy. Hadoop sorts through your ‘digital exhaust’ (as well as any other massive load of data) and culls insight or information from it. This result can then be sent to the data warehouse for analysis – It does not have to be sent there, but in most cases I’m assuming that many folks would like to include the new insights into their analytics.
Think customer churn models, if hadoop was able to determine 1 or 2 other hidden or unknown traits of a customer segment from lets say, web click through routines (the exhaust) – The analysis would be much more accurate and theoretically save (or make) the organization money.
There are many ways that hadoop technologies can be a part of your enterprise data warehouse or big data platform – this was just one simple example that I like to use to get my head around the technology.
At the end of the day, hadoop enables analysis of Big Data problems – It might not answer them all on its own – but it is a key player (if not ‘the’ key player) in Big Data Analytics.
Back into the swing of things, …now with a Big Data focus
It has been a busy summer so far, just getting back home from San Diego the other day for last week’s TDWI world conference. As usual it was a great event, but one thing that was quite apparent at the event was the advent of ‘Big Data’. I don’t mean just the addition of Hadoop based companies to the mix (ie Hortonworks and the like) but most every vendor had some sort of Big Data story to tell. Even SAS had their booth just plastered with Big Data messages, and does not offer any specific ‘Big Data’ product.
For full disclosure, I’m partial to this growth of Big Data promotion in the marketplace, as I recently migrated my professional focus at IBM to this specific area. There is a level of excitement that surrounds Big Data that I have not seen since the early days of Linux adoption. Folks are clamoring to get in on this technology and surrounding buzz. From developers through consultants, many of my warehouse discussions had some sort of ‘Big Data’ piece to it.
So what is ‘Big Data’? That is a question that I hear left and right – not just at the conference, but in general day to day business. In my limited layman type approach to the definition, I refer to ‘Big Data’ as the challenge organizations and professionals have to start using ALL of the data available to them to make better decisions. Are you using all of it – even the stuff you normally throw away.
Does Hadoop help this? – sure – it is a key enabler in the movement.
Does streaming technology ?– absolutely – decisions in motion – awesome.
Data Warehousing? – uh of course – what do you do with all of this Hadoop based data once you sort through it and deem it ‘relevant’?
Look the list goes on and on here, but the fact is that in my opinion ‘Big Data’ is a part of a larger information ecosystem. It is the challenge to leverage all of your data available to you – not just the items that are placed in front of you, but also the ‘digital/data exhaust’ (great term that was created that I openly support) that to this point in time you have not had time to analyze.
I would suggest that part of the issue of understanding is that every company that has a dog in this fight has crafted the message to suit their own needs. Many vendors only have a subset of the products that make up an exhaustive Big Data platform and skew the definition to support this.
At the end of the day – regardless of definition, the question that you have to ask yourself is ‘Would you be better off leveraging ALL of the information that is available to you, to make better decisions?’ If you desire to be an analytical competitor, your answer is definitely yes. This is why Big Data is big …and why the hype is warranted.
Coach always said that it was about basic blocking and tackling
Working in the Information Management / Business Intelligence space, we always search for those perfect ‘golden nugget’ type stories to share. “Company X leveraged their ‘Y data warehouse’ and increased sales ‘Z%”… Or the latest cutting edge business intelligence solution enabled company ‘C’ to grasp ALL of their information and insight and well, they are now destroying their competition, predicting the future and taking over the world… due to their investment in company ‘B’.
Sound familiar?
These stories are fantastic – and for the customers involved (as well as the right prospective customers) they are illustrations of smart IT decisions to grasp their own information for better decision making. Great. Yet before we run out the door, bottling solutions with a ‘one-size-fits-all’ mentality, we have to pause and acknowledge the fact that not everyone will benefit from this.
Over the last few months, while I was on assignment in China, I had the opportunity to interface and examine the operational business models of a number of companies and organizations. While this engagement had nothing to do with my day to day involvement in data warehousing and business analytics, I always kept in the back of my mind – “How are these organizations using their information systems?”
While many (if not all) of the organizations had a developed IT infrastructure, I was not able to see any business intelligence being reaped (minus one lone Finance manager investigating balanced scorecards) from the collected data. On that note – even in the B2C companies, information was not being captured via technology – it was interpersonal and ‘in-person’. Now, I am an advocate for more face to face business (Don’t even get me started on the ‘effectiveness’ of the virtual meeting) but there is no way that a company can grow efficiently when there is no information sharing across employees and departments.
Growth itself was the common issue with all of my clients.
When you are selling out of a product, you have to increase production to meet the rising demand. Yet, capital investment to increase manufacturing is only one side of the coin. You also need to scale your leaders, your sales force, your marketing, etc. How do you expand your reach and your organization, if all of the information assets are in your head? or your respective employee’s heads. We need to share information via technology to make better, more informed decisions. This it the foundation of business intelligence – You need to capture and record the data before you can make analytical decisions, correct?
Perhaps my head is still in China, consulting on corporate development – but I am honestly overwhelmed by the amount of marketing stories back home that are swinging for the fence. I open my first paper version of the Wall Street Journal in months (Their iPad app absolutely saved me from the great firewall) and I see some silly Oracle marketing on ‘fastest data warehouse, yada yada’. I kinda chuckeled – yet also cringed.
I must be on a usability kick, but if we put more effort into making business intelligence technology consumable, more usable, more tangible – rather than these uber expensive ‘shock’ ads – we’d get a lot more stories about success, which kinda pay for themselves.
Back to my original thought – The vast majority of companies need basic blocking and tackling when it comes to business intelligence and data warehousing – Basic reports, quality data, relevant data. I’m fully aware that we require the thought leadership stories, but let’s tone down force feeding clients solutions that only are consumable by 1% of the population. (Not everyone has $M+ to spend)
Product developers, analysts and marketers (as I am guilty of this) go off on technology rants to differentiate products as well as differentiate themselves. We have endless blogs on columnar vs relational, appliance vs custom, commercial vs. open source and anything else that we can jam pack into the latest product announcement or press release.
Not one company that I worked with while on assignment cared about any of the above. They only asked, “Can you help me figure out how to gain market share?” and “How can we understand our clients better?”
Sales 101 – Listen to the customer first. …It’s time to get back to basics.
Specialty Street: An Overwhelming Attack on the Senses
After a long day cranking through our 1st day of research and brainstorming, we decided for a change of pace. The decision was to hit up a Japanese restaurant (we have been eating Chinese cuisine pretty much every day, all day.) for once. Our interns recommended one that specialized in ‘Hot Pot’ and took a cab into the city.
Dinner was great (Sushi not the same as others that I have had, but beggars can’t be choosers) and the layout of the hot pot ingredients was fantastic. Mushrooms, beef, greens, flowers, pork and tofu were scattered about the table. Individual pots of boiling liquid were placed on a mini stovetop, integrated into the table. (I went for spicy, as usual)
After the feast (and I mean FEAST) we headed out to a street that Winky suggested that we ‘had’ to see. It was called ‘Specialty Street’ (Official translation pending)
This street was chock full of vendors selling everything from Durian fruit to sparrows, to vast assortments of live seafood. I personally only tried a mild sugar cane water, bit the selection of offerings was astounding.
With this being said. I cranked out a quick video from my iPhone yesterday (Very basic editing) and posted below – Apologies on the quality, but will hopefully get better at filming, editing and of course actually speaking.
Check out the team blog at: http://cscchina16.wordpress.com
On to Nanning, Guangxi Autonomous Region – The last post from Beijing
So apparently it’s been a few days since I chimed in. Has been a very busy time here in China. (so busy that I wrote most of this in pieces on the iPad 4+ days ago – just getting to posting)
The last time that I was in China was in 2007, right before the 2008 Summer Olympics in Beijing. At that time, Beijing was undergoing a complete overhaul both physically and culturally. The country (and organizations) were pouring money into economic development – especially into construction projects. I can still remember the sea of scaffolding spread across the city.
With the new, out goes the old – and in concert with this change for modernization comes a destruction of sorts of the old culture. The hutongs of Beijing were being bulldozed to make way for new and more modern buildings and neighborhoods. Some hutongs still remain, but are shadowed by a sea of high-rises across the city. All things must at some time change, but we must make sure to preserve what we can from our history and our heritage. (China has now begun to preserve them)
Since I wrote last, I’ve moved from the tourist mode to the data warehousing marketing work mode in Beijing.
I have never been to the IBM office in Beijing, but had a number of meetings scheduled on my way over to Nanning for my Corporate Service Corp assignment. I have recently gotten better at navigating the subway in Beijing (although it is a bit jarring being the only foreigner on the train) and decided to use it to get to the marketing office at Pudong Plaza.
Getting out of the subway, I was greeted by the 2008 Olympic Park and the giant birds nest-like stadium that became so famous. It was very cool to see all of these buildings that were so commonplace in the media – yet in reality these buildings were all much larger in person.
Turning to my left, once into the Olympic Park – I was able to see the very unique IBM building at Pudong Plaza. The main (and satellite) office buildings are built to look like a Chinese dragon, moving along the earth. This was very cool – and to think that this was where I would be holding meetings this week.
The meetings and enablement activities actually went very well. I understand the financial importance of virtual meetings, but there is something to be said about face to face – especially when you are using different languages. On the phone, we would be disconnected – not always sure that we understand one another, but after these meetings – I came to realize that just one meeting in person would have staved off much chaos and confusion. Perhaps I need to quantify the opportunity cost of it all.
At the end of the week, I met with one of our premier business partners at their office in Beijing. This partner, ADM, has been successfully building (and selling) their own custom business analytics solution on top of our Smart Analytics System 5710 in the China market. We had recently agreed to work on a series of case studies and videos together and I was stopping by to meet with them to brainstorm on the concept – and gain final approval.
It had been since my trip to Tokyo, Japan last year that I had to use a translator for a meeting – But the translator was one of our own lab services employees and did a wonderful job making the discussion as seamless as possible. I know that the month in Nanning will require even more assistance from a translator (a team of 8+ university interns to be exact) – but I’m starting to get irritated that I am completely helpless with the Chinese language. I need to figure at least a portion of this out to communicate.
I’m on the flight now to Nanning (China Air got WAY better since last time) and I’m not sure what to really expect. Beijing had grown into a very accessible, very major city from the last time I visited. Will Nanning offer the old China? The type of China that we read about as children with tea plantations, wonderful cranes and array of beautiful nature and scenery? (No offence to Beijing – but you are a very ‘concrete’ city)
Very excited for the unknown. Very excited for this adventure.
Here is a clip on Guilin from the Guangxi Autonomous Region – in the same area where I will be for the CSC program
Touchdown Beijing – Whirlwind first few days in the city
Right off of the bat, I have to admit that the full first night sleep in a plush, oversized bed was divine. It was so refreshing that the first thing that I did on this Sunday morning here in Beijing was head to the gym for an hour on the treadmill. This may have been my mental demise for the rest of the day as with my biological clock not necessarily in check, I started to fade fast throughout the day.
I met up with my cousin, Liz (and her friends) quite early in the day. Crazy to think that one of my relatives was out here for a fun trip, but very cool to have the opportunity to see the city together.
At 10am I was in the Silk Market being accosted by hoards of Chinese women pushing silk scarves and an assortment of fake products at me. Usually I would be invigorated at the frenzy of negotiation and noise, but today I was tired, …reserved, and overly passive. I just kind of smiled at it all. I wanted so bad to hit the watch counter – but I could not muster the strength to bargain.
Liz had been here for over a week and was in the throngs of it all. Her and her friend Ali were playing off one another to grab better and lower deals. They did well. Yet I could do better. I was the master at this . Yet for some reason, I was DOA.
We toiled around a bit more in the market and then hightailed it to their other friend’s apartment and then off to the local food market.
This food market was a new experience for me.
I have been to open air markets around the world – from domestic farmers markets in Pennsylvania and North Carolina to famous markets in Greece, Spain and Singapore. Two things stood out for me in this market that surprised me.
The amount of beautiful fresh produce that I would not have expected for a Beijing market. Absolutely beautiful array of colors and selection of fruits and vegetables. Add to that the stalls that had fresh noodles. (durian and mangosteens in full effect – still need to try them both… Hopefully Nanning will deliver)- The lack of refrigeration for the meat shops. Birds, beef, pork and lamb were opened up, butchered and spread across wooden counters… Fresh, yes – looked great. Yet, the shelf life had to be about 24 hours. It was warm in there and I had to keep forcing my self to forget that I most likely would be noshing on food that was butchered and sold in this manner. It was cool to see blue hens on display (random)
I unfortunately have no pictures of the establishment, but we hit up a dumpling house for a late lunch snack. Phenomenal. I’m actually craving the condiments more than anything else – Fresh garlic, dried chili in oil and an assortment of soy and vinegar to douse in. After the dumpling orgy, we hit up a Chinese shoe store. I looked and looked and could not find an 12 or 12 and ½ size shoe anywhere in sight. Crazy women’s shoes galore, but no Men’s past the 10 or 11 mark. (I would have to import my shoes if I lived here)
A great close to the night was a trip for some Peking Duck. Years ago when I was in Beijing, we visited Li Qun for a rustic, yet delicious meal. This place (name missing) was on the spot as well. An actual highlight was the Chinese eggplant – I could swear the there was Ketjap Manis in it (Sweet Soy Sauce). All in all a great meal and a great night – with locally brewed Swartzbier to boot (another killer highlight).
As I wished my cousin good bye (her trip had come to an end) I started to get that feeling I was expecting from the start. It’s great to have family and friends when you head out on a trip, but it does add to the withdrawal when you hang out with them when you first land for a day. You get used to having camaraderie – and then hits harder when you walk out in the street and realize that you are the outsider. You understand that you are alone.
Heading to Nanning with the CSC crew will most likely curb this feeling – but it is important to point out. Taking the subway to the office alone in Beijing during rush hour is a different kind of experience. The sheer number of people is one thing, the fact that you yourself are the spectacle is another.
As stated in the last travel post, you have to learn how to adapt pretty quick here. I have been fortunate enough to have been reading Scott Seligman’s book “Chinese Business Etiquette’ before and during this trip. This has saved me a number of times already – and I will most likely quote certain sections over the next month.
It’s closing in on mid-late week and I have a moving meeting with a major business partner – as well as a presentation/seminar with the local team here in Beijing. Day one felt long, yet now that I have become a bit more comfortable, I sense the days just whizzing by. I’ll try and post prior to leaving for Nanning and my CSC assignment on Friday, yet the next two days are chocked full of work activities so not really sure at this point. Time to clock off.
In postscript – Uber fast internet at the hotel here has allowed for great video chat (FaceTime) with the family. This is something rather new to me travelling internationally (quality of it all) and it totally reduces the strain of being away. I surely hope that this works for the time in Nanning as well.
Smart Business: Simple and Tangible 