Big Data is All the Rage, Now and in 2015

Posted on by

We’ve recently seen a flurry of activity on our LinkedIn Group. We went from 55 to 800 people in just a couple of weeks. I’m always astonished by how quickly we connect with people these days. I’m not sure how to measure it, but I’m guessing that those 800 new group members represent a network of somewhere around 100,000 people.

In the midst of the flurry, I asked the following questions: what is your interest in big data analytics? how do you think big data analytics will change the way we do business by 2015?

Here are a few of the answers that caught my eye:

Kelly Moertle: We live in a time where companies are trying to have a more personal experience with their customers.  The only way they can do this is if they treat their data as the valuable asset that it is.  I think it’s fascinating what companies can learn about their customers by making proper use of their data and the opportunities this presents for more targeted marketing, better customer service, lower customer attrition, etc.  It’s a fun place to be right now.

Greg Petraetis: What I am hearing from companies is that the amount of data in their world is set to explode or has been exploding, and analyzing large data sets—so-called big data—will become a key basis of how they compete, underpinning new waves of productivity growth, innovation, and consumer surplus. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, device proliferation, the importance of customer experience management et al will fuel exponential growth in data for the foreseeable future in terms of variety, volume and the speed at which organizations need to analyze information.  Old technology standards aren’t the answer to meet this new demand in a price performing fashion so that companies can see economies of scale in their returns.  I joined ParAccel because we have the technology and subject matter expertise that can address this need today so that companies will see 2015.

Ed Bernier: Never before have companies had access to so much data about their customers.  Companies that figure out how to harvest this detail data can gain a tremendous amount of insight into the lifetime value and lifestyle of their customers.  Combine a high performance analytical database platform like ParAccel’s PADB with in-database analytical functions and it’s amazing how accurately things like “propensity to buy”, “high churn risk” or “cross selling opportunities” can be predicted.  As Kelly said it’s an exciting time to be delivering business intelligence and analytics solutions.

And of course, my take: My interest in big data analytics has to do with the amazing things that people and companies can do when they realize the potential of big data and the capabilities of new analytic platforms. As the guy responsible for solutions at ParAccel, my favorite thing to do is work with customers dreaming of what could be done on our platform, then watching customer’s amazement when they realize they can actually do them. By 2015, I believe big data anlytics will be mainstream and will be processed so quickly that even complex analytics on big data will be embedded in business processes. We will also be at the forefront of mathematically-driven, automated decisioning.

If you haven’t yet, come join the discussion!

Proprietary Comms Protocol Designed for High Performance Analytics

Posted on by

I had the pleasure of speaking with Robert McMillen, one of many top notch innovators and engineers working on the ParAccel Analytic Database. We have said alot about our foundational design principles and the fact that we build our product for speed, agility, and sophistication. After my conversation with Robert, I realized just how serious we are about these principles. I asked Robert if he would share his excitement with the rest of the world. He said, “YES.”

Guest Post by Robert McMillen, Comms Engineer and Innovator at ParAccel:

One of the key differentiators in ParAccel’s PADB analytic engine is its proprietary communication subsystem.   We built our own communication protocol primarily for speed but also with reliability in mind. Speed is a basic design principle for everything we do at ParAccel. To achieve greater speed, the communication protocol doesn’t carry all the overhead of the more generally designed TCP/IP protocol. Instead it is designed to fit like a hand in a glove with our columnar database and multi-parallel processing engine. It has just the right amount of error protection and lost packet handling for the short distance data travels within our system, compared to much more stringent requirements placed on long distance packet delivery.

The most fundamental entity in our system is a slice.  Associated with each slice is a fraction of the total memory and storage for data in each table and a fraction of the available processing capacity. The total number of slices in a system represents the degree of parallelism available to process queries against the data. Having such an abstract entity to work with, we can tailor our software to maximally utilize the available hardware by selectively configuring the number of slices assigned to each compute node.

In keeping with this, the comm subsystem, which is built on top of the UDP protocol, supports addressing based on global slice ID. A given query process will be responsible for a single slice of data and executing a user’s query against that data. At any point at which the query requires the exchange of data among slices, a comm channel object can be instantiated. Each channel consists of a source facility and a sink facility.

For example, in a data distribute operation, each query process could scan its slice of a table, perform a hash operation on one of that table’s columns and based on that, determine which global slice each corresponding row belongs to. Using a channel source, it need only hand the row and destination slice ID to the comm subsystem.  From there the row will find its way to the proper compute node, query process, and comm channel in that process, on that node. Upon arrival, it will be delivered to that channel’s sink facility at the destination. Similarly, the original query process will receive all the rows assigned to it via the hash, from every other slice in the system from its channel sink.  At the end of the operation, every slice will be in possession of all the rows from the original table that were assigned to its slice ID.

The example illustrates just one of the services provided by the comm subsystem but makes clear how perfectly fitted it is to the needs of doing query processing in a multi-parallel processing environment.  Underneath the covers, the comm subsystem efficiently manages movement of bundled data, retransmission of any dropped packets, and even automatic reconfiguration if a component fails.

AHA! So That’s What Goes On at the Gartner BI Summit

Posted on by

I got to attend the Gartner Business Intelligence Summit last week in Los Angeles, a conference which affectionately became known as #GartnerBI. What could we possibly do in a world without Twitter hash tags?

One of the most interesting sessions to me was the wrap up they did around AHA moments that were captured by a combination of Gartner analysts, vendors, and attendees. Following, and taken straight from the Twitter feed, is my take on what surprises people at the conference:

AHA: Fast is no longer enough, accessibility via visualization is critical to
successful adoption. #GartnerBI

It is visualization, tablet computing, and the popularization of data that is truly driving adoption. Check out www.flowingdata.com!

AHA: Case studies from companies like eBay and how much they can do on a large scale, how much more we can do on a smaller scale. #GartnerBI

I loved that eBay was talking about the analytic sandbox, something they have developed over the last several years. It hit home since ParAccel can provision an analytic sandbox for new users in just 1 minute and 8 seconds. Check out the Lab
Validation Report: ParAccel PADB and NetApp SAN Optimized Solution.

AHA: Business and IT roles are collapsing, causing the need for changing roles. #GartnerBI

I had a great conversation with our friends from OfficeMax about the idea of creating an analytics center of excellence to help drive the collapse. When companies expand their analytic capabilities, it takes a combination of business and technical people to drive adoption and educate people about what can be done with analytics.

AHA: Case study where a company put all data functions in a single group enabled common understanding of data. #GartnerBI

This kind of collaborative thinking opens up the opportunity for cross-pollination, as well as, combined analytics, where different parts of the organization combine analytic results to create more value for the company and a more sustainable competitive advantage.

AHA: There are untold numbers of counterintuitive performance indicators out there. Analytics can help discover them. #GartnerBI

Simple truth, but hard to accept by some: New and emerging indicators that help predict outcomes will not be uncovered by human exploration; they require mathematical or machine-driven discovery.

AHA: We are not alone; we can all collaborate on better solutions for making sense out of our technology. #GartnerBI

It would be interesting to create a truly collaborative environment where people from different companies could come together to work on problem solving in more of a workshop, as opposed to the classic “sit and listen” conference structure.

AHA: Mobile BI used on tablets opens executives to more interactive experience with data, information, and analytics. #GartnerBI

I love the idea of executives “playing” with data for the first time, because they have it on their iPad. Thank you, Apple, for helping change the way we interact with data!

AHA: The number of organizations creating a BI strategy for the first time. #GartnerBI

This speaks to the fact that the business intelligence and analytics world continues to grow at astonishing rates. Having been in this industry for 17 years, I always feel like we’ve done everything possible. But is it possible that we have only touched the surface of what can be done with analytics? The pioneer in me likes to think that there are many new universes to explore!

AHA: No one tool will supply all of our needs #GartnerBI <Thanks again to Mark Beyer and the idea of the logical data warehouse #GartnerBI

I am a firm believer in a collaborative architecture where different platforms work together and workloads migrate to the platform best suited for them. Check out the new YouTube whiteboard on Analytic Offload and Big Data – Free Up Valuable Resources

AHA: In executive meetings look for opps to apply BI resources to address real business needs. #GartnerBI <the role of the imagineer!

In my 16 years in business intelligence and analytics, the role I have enjoyed most is working with companies as an imagineer, taking what I know about technology and helping people see the potential of what could be done with it, many times beyond their dreams.

AHA: We should have come with a business champion to this conference. #GartnerBI

If information technology professionals give their business counterparts the sense that they were “champions” they would be sitting next to them. One of my basic tenets of life is toward the success of the people around me. I don’t always succeed, but I try. It seems to be something that could bridge the gap between business and IT.

AHA: The democratization of business intelligence can make more information available to all the people. #GartnerBI

I wonder if it’s more of a republic, where really amazing data engineers, data scientists, and information architects represent the masses and help create information hubs that are compelling, valuable, and easy to use for all.

AHA: One person can change a company through the innovative use of analytics! #GartnerBI

Yes, and AMEN! Every company is waiting for someone to rise up with analytic innovation and lead the charge. One of the reasons I joined ParAccel is because I believe their analytic platform is capable of supporting amazing innovation.

AHA: So many new, cool new BI vendors #GartnerBI <@ParAccel has been around for a while but new to many.

I’m quite impressed with the number of new vendors taking a “design thinking” approach to product development. The designer is the new hero in the world of business intelligence and analytics. However, as popularity grows, a back end analytics platform to support the influx of new users is equally important and cool.

Kurt Schlegel AHA: Consumerization of BI is finally catching on, more mainstream than ever. Proof point: interactive visualization #GartnerBI

Nice work, Kurt. Your AHA is sort of reminiscent of Alvin Toffler and “Future Shock.” Maybe before I die, we will be walking through virtual representations of data. I look forward to the holodeck for data.

So, that’s it for the AHA moments at #GartnerBI. I look forward to next year to see what has changed and what’s on the horizon.

Reflections on the Gartner BI Summit : Visualizing the Power of Collaborative Social Data

Posted on by

A Guest Blog by Candice Santaferraro (@csantaferraro), Social Media Manager for ParAccel

We’ve now wrapped up an invigorating four days at the Gartner Business Intelligence Summit in Los Angeles.  We gathered and conversed  about data and analytics with thousands of professionals from across the globe. While much was discussed, one thing stood out to us: the clout of social data resonated loud and clear. And for that reason, I wanted to begin this blog with the words of Jesse Hamlin: “Real innovation happens when teams collaborate on and socialize data.”

With 1,492 tweets  and 343 contributors in 4 days, the conference made over 2,000,000 impressions. Not only were we there talking about power of data, but we were participating in a massive stream of collaborative information.

However, more interesting than the numbers was our ability to track conversations and points of interests across the entirety of the event. This word cloud at the right gives us some visual insight into what people were really talking about.

Gleaning though the conversations and visualizations of data, I found a clear emphasis on the trend toward mobile opportunities and interactive data with websites like Flowingdata.com and applications like Tableau that allow you to visualize and play with the information before you. Collaboration is innate in this social data interaction, and the ability to share data and ideas proliferates innovation.

In a recent article published in the New Yorker, science writer Jonah Lehrer illustrates the famed Building 20 at MIT which served as an incubator for the linguistic work of Noam Chompsky, the sound engineering of Amar Bose and the Bose Corporation, and many other brilliant minds. While the architecture of the building was rather drab and the school nearly tore it down midway through the 20th Century, it essentially bore a space for scientists and engineers to work in solitude. However, the horizontal layout in which Lehrer describes in more depth, spurred interaction and therefore spontaneous conversation between individuals coming from diverse disciplines, yet still working in the same building. Consequentially, Steve Jobs also designed the Pixar building with a similar thought in mind. But Lehrer concludes his article with this poignant statement:

“All these errant discussions add up. In fact, they may even be the most essential part of the creative process. Although such conversations will occasionally be unpleasant—not everyone is always in the mood for small talk or criticism—that doesn’t mean that they can be avoided. The most creative spaces are those which hurl us together. It is the human friction that makes the sparks.”

No algorithm has yet had the ability to absolutely recreate the social intelligence that we navigate as human beings. Whether through a mobile device, machine or another tool: there is much insight to be gained though the conversations we share with those around us. Technology that allows us to glean information from those flowing conversations renders consequential value: and nothing was more clear than this as the conference wrapped up last week.

A Guest Blog by Candice Santaferraro (@csantaferraro), Social Media Manager for ParAccel

Nothing but Net: Wisdom and the NCAA Championships

Posted on by

All the blood, hype and sweat have culminated to this one final game: a face off between the Kansas Jayhawks and the Kentucky Wildcats. The NCAA Championships have just now wrapped up and as Kentucky takes the title, what would have been a mere television marketing opportunity in the past has now expanded into a massive social data goldmine. Beyond simply speculating the what team will win the game. we now have the ability with apps like Microstrategy Wisdom to analyze the fans from both sides. Wisdom released this infographic earlier showing that the fans are not as similar as one may think.