Real-Time Big Data Analytics: Emerging Architecture white paper authored by Mike Barlow covers big data analytics topic and how real-time big data analytics (RTBDA) are different from traditional analytics. Mike describes the five phases of real-time big data analytics framework: Data distillation, Model development, Validation and deployment, Real-time scoring, and Model refresh.
He also discusses the four-layers of RTBDA technology stack proposed by David Smith:
- Data
- Analytics
- Integration
- Decision
InfoQ spoke with Mike about the current state of real-time big data analytics and the emerging trends in the Big Data space like Decision Science.
InfoQ: Can you define what Real-time Big Data is and explain how it works?
Mike: Real-time big data represents a convergence of technologies and techniques. Analytical processes that used to require month, days, or hours have been reduced to minutes, seconds, and fractions of seconds. Two years ago, many data analysts thought that generating a result from a query in less than 40 minutes was nothing short of miraculous. Today, they expect to see results in milliseconds. That’s practically the speed of thought – you think of a query, you get a result, and you begin your experiment. On the other hand, milliseconds would seem like an eternity to a high-speed trading system, which operates in a universe that is parsed into microseconds. So the term "real-time" depends a lot on the requirements of your job or the task at hand.
InfoQ: What are the differences between traditional analytics and real-time big data analytics?
Mike: Traditional analytics tend to be a process for generating reports from structured data that has been retrieved from a traditional data warehouse. Real-time big data analytics take it to the next step by generating insights or recommendations that can be used to drive business value at the point of sale, whether the point of sale is a brick and mortar store or an e-commerce web site.
InfoQ: In your white paper, you talk about the new era in which machines begin to think and respond more like humans and the shift from data science to the next logical frontier: decision science. Can you discuss more about these innovations and upcoming trends?
Mike: The trend is towards faster, more automated, more intelligent and more business-friendly analytics that summarize insights quickly and offer specific recommendations that are more likely to result in greater sales and higher profits.
InfoQ: You also talked about the creation of analytics and the consumption of analytics being two different things. Can you give an example of these two steps?
Mike: Data analysts - the people working directly with the data - are different from users or consumers of data, who tend to be closer to the company's actual customers. Both require tools for visualizing and understanding data, but the tools are going to be different. Data analysts and data scientists need tools for modeling data. Users and consumers, whether they are sales reps or marketers -- need tools for understanding how the data is likely to impact customer behavior and result in a sale.
InfoQ: What are the emerging trends in real-time big analytics?
Mike: Because there are many different kinds of analysts and many different kinds of users/consumers, there must be a choice or range of tools for visualizing, interpreting and acting on big data in meaningful ways that drive real business value. At the present moment, there is no "one size fits all" solution for launching a real-time big data platform. On the bright side, lots of very intelligent and highly motivated people are working hard to develop techniques and strategies for converting big data into tangible corporate assets. Stay tuned, this should be exciting!
About the Interviewee
Mike Barlow is an award-winning journalist, author and communications strategy consultant. Since launching his own firm, Cumulus Partners, he has represented major organizations in numerous industries. Mike is coauthor of The Executive’s Guide to Enterprise Social Media Strategy (Wiley, 2011) and Partnering with the CIO: The Future of IT Sales Seen Through the Eyes of Key Decision Makers (Wiley, 2007).
He is also the writer of many articles, reports, and white papers on marketing strategy, marketing automation, customer intelligence, business performance management, collaborative social networking, cloud computing, and big data analytics. Over the course of a long career, Mike was a reporter and editor at several respected suburban daily newspapers, including The Journal News and the Stamford Advocate. His feature stories and columns appeared regularly in The Los Angeles Times, Chicago Tribune, Miami Herald, Newsday and other major U.S. dailies.
Mike is a graduate of Hamilton College. He is a licensed private pilot, an avid reader, and an enthusiastic ice hockey fan. Mike lives in Fairfield, Connecticut, with his wife and two children.