Knotice is a provider of data management services, specializing in the intake and activation of data for advanced messaging and analytics across digital channels. With the Knotice DMP (Data Management Platform), customer information across disparate platforms and touch points can be consolidated within a big data environment, while individual data profiles (of both known and anonymous individuals who engage with the brand) are automatically enriched with a high degree of data fidelity.
Leading names from the consumer products, retail, and cable and telecommunications industries trust Knotice to manage their data. These clients benefit from the ability to perform in-depth analytics, to use dynamic content to personalize their websites on the fly, and to send thousands or even millions of email messages at once. To provide these services effectively, Knotice must process massive volumes of data in seconds.
“When we architected our system years ago, we had a vision of a central hub that’s focused on the individual profile,” explains Bill Landers, Chief Technology Officer, Knotice. “There are other analytics programs that are great for analyzing data on the aggregate. We do that too, but we really excel at enabling profiles.”
Complex ETL Jobs Delay Big Data Analytics
On a typical day, Knotice receives hundreds of millions of lines of data—at high velocity—that it must ingest, normalize, enrich, and analyze so that it can deliver insights that let clients take action. Much of this data defines individual profiles that contain rich data about the characteristics and behavioral history of consumers—information that’s critical to segmentation and modeling. By leveraging this data with agility, Knotice’s clients can begin to form a unified view of their customers across marketing channels, devices, and locations.
“We envision a future in which marketing and advertising professionals will obtain actionable insights in real time during executive briefings, rather than at the typical snail’s pace of hours or days later,” says Landers. “High-value audiences will immediately be pushed to multi-channel campaigns and the raw performance results will be fed back into the system to further enrich profiles and models.”
But until recently, it was difficult for Knotice to execute on this vision. The company batched data using complex, multi-stage extract, transform, and load (ETL) jobs on Microsoft SQL Server Enterprise. As a result, the largest batching processes could take hours to complete while consuming valuable system resources the whole time. Analytics and database development also took far longer than they should have because validation of results took hours. These delays threatened Knotice’s competitive advantage.
“It’s critical for us to be able to deliver results—for segmentation, analytics, or dashboards—in seconds or minutes,” says Landers. “We wanted to make sure we could exceed the expectations of our clients and stay ahead of our competition, so we began to explore some of the better solutions on the market for managing Big Data.”
After reviewing the offerings of various massively parallel processing (MPP) vendors, Knotice narrowed its choices to HP and Pivotal. As soon as product trials began, Pivotal quickly jumped ahead not only in terms of its products’ performance, but also in the strength of the relationship it formed with Knotice’s evaluation team. “During the trial, the Pivotal system was taking complex problems that our current system would chew on for 45 minutes, and reducing them to a single SQL statement that would be run in 45 seconds,” recalls Landers. “Equally important, though, was the fact that Pivotal’s entire team had retained that ‘startup’ mentality of innovation and openness. With the investment they’ve made in data science, they’re clearly in the Big Data space for the long haul.”
Pivotal Greenplum Provides the Power
Today, Pivotal products form the new foundation of the Knotice Data Management Platform. Knotice uses Pivotal Greenplum—running on a EMC Data Computing Appliance (DCA)—to ingest, normalize, segment, and analyze millions of customer profiles related to billions of time series events. The company leverages the open source MADlib in-database analytics library to compute models that are used for campaign optimization and customer segmentation. At the same time, Knotice’s distributed data bus—built on a topology of RabbitMQ nodes—orchestrates the replication of data changes between the company’s analytic data environment and sharded data activation environment, which is optimized for high concurrency, low latency transaction processing. Knotice also ensures low latency responses during individual user sessions on its website by caching appropriate data using Redis.
“Pivotal delivers pure performance at scale for all the areas that matter most to our business,” says Landers. “That includes data ingestion, complex queries, analytics, and data output.”
“ Pivotal’s strategic vision and product roadmap is keeping pace with the rapidly changing Big Data environment. We’re confident that Pivotal will continue to meet the demands of the ever-increasing wave of data we must manage for our customers.” — Bill Landers, Chief Technology Officer, Knotice, Ltd.
Producing Reports in a Fraction of the Time
The EMC DCA returns results in seconds or minutes, and greatly simplifies Knotice’s data management approach by eliminating manual tasks. Preparing a complex business intelligence report used to take 45 minutes and require a scheduled delivery. With Pivotal, Knotice can complete this task in 45 seconds on demand. When it comes to performing complex multi-table update/inserts of flat files, Knotice can now complete 77,000 per second—a dramatic increase from its previous rate of 1,400. The company has also reduced the time needed to produce Tableau dashboard reports from 15 minutes to 5 seconds, and has streamlined ad hoc segmentation—across millions of profiles and billions of events—from 30 minutes to 60 to 90 seconds.
Enabling Real-Time Data Exploration
As Knotice customers derive insights from their data, they can now take action on them in seconds—performing rapid data exploration to iteratively evolve audiences and analytic insights. Whereas they might have waited 30 to 45 minutes to get the insights they needed under Knotice’s previous architecture, they now get these answers in seconds. “We’re giving our clients the ability to sit in an executive meeting and bring up real-time insights that actually change the course of that meeting—rather than just dropping a request in the data team’s queue and waiting hours or even days for a response,” says Landers. “With Pivotal, the response times on our platform are blazing fast, which will result in greater customer productivity and more satisfied users.”
Going from Software Vendor to Strategic Partner
Pivotal’s solutions have done more than give Knotice the performance it craved—they’ve also helped the company raise its value among its target market. “Our vision for our platform has always been to break down the information silos that often exist in companies and give customers the tools to execute a cohesive messaging strategy across channels and on all devices,” says Landers. “Our new architecture based on Pivotal technology enables us to have more strategic conversations with C-level executives about how Knotice can be the hub they need. We enter these meetings fully confident that our product can deliver not only the features, but also the performance they expect.”
Now that Knotice is live on Pivotal solutions and achieving measurable results, the company plans to expand its use of the product line. The company is exploring ways to use Cloud Foundry to manage and deploy Knotice Mobile View (mobileview.knotice.com)—a free product offering that helps prospects test out Knotice’s capabilities. The company also plans to tap into the powerful Hadoop capabilities of Pivotal HD as it begins to include unstructured and semi-structured data in its Big Data environment.
“Pivotal’s strategic vision and product roadmap is keeping pace with the rapidly changing Big Data environment,” Landers concludes. “We’re confident that Pivotal will continue to meet the demands of the ever-increasing wave of data we must manage for our customers. That’s why we made Pivotal products the foundation of our Data Management Platform.”