From announcements at EMC World and Hadoop Summit to major updates on Pivotal’s data platform, this month’s Build Newsletter shows just how hard we worked in the past month! While the 60+ links below should provide some interesting discussion points for your next meet-up, one of the high-cool-factor links is the new performance study by GE using our in-memory data grid product with 5 terabytes of data in memory!
EMC World: Open Source Updates and Pivotal Customers—BMW and BC Hydro
Since our last newsletter, some of the most newsworthy and interesting information came out of EMC World. Given Pivotal’s open source DNA and historically closed-source parents, it was exciting to hear both EMC and VMware launch open source projects. VMware announced two—Lightwave is all about container identity and access management, and Photon is a lightweight Linux distribution. As well, EMC open sourced some of its ViPR controller software to combine pools of storage across vendors.
We participated in many sessions, and two big elements of the themes at EMC World included a deeper look at data lakes (slides) as well as keynotes on IoT (slides). We were excited to hear our joint customer, BMW present and explain how they used big data to perform predictive maintenance—predicting the malfunction of car components. BC Hydro also presented at EMC World and is using EMC storage and Pivotal for big data analytics—specifically Pivotal HD, Pivotal HAWQ, and Pivotal Greenplum Database to perform IoT-centric processing on smart meter data to reduce energy theft.
Data Platforms: The ODP and Horse Manure, HAWQ on Hortonworks, and an Update to Pivotal Big Data Suite
On the topic of data platforms, industry announcements flew out of Hadoop Summit 2015 in Europe. Of course, the Open Data Platform (ODP) was addressed and discussed by many. To date, ODP members include GE, Hortonworks, IBM, Infosys, International TELCO, Pivotal, SAS, Altiscale, Capgemini, CenturyLink, EMC, PLDT, Splunk, Teradata, Verizon, VMware, and WANdisco. WANdisco’s CEO recently had some insightful remarks in an interview with Datanami—he even refers to views opposing the ODP as something that horses leave behind.
Pivotal made big, back-to-back news headlines on several fronts with regards to data. First, we now officially support Pivotal HAWQ, our SQL on Hadoop engine running on Hortonworks Data Platform and it works in the Hortonworks Sandbox with some detailed instructions on deploying via Apache Ambari™. At EMC World 2015, we made a splash with open-source-based improvements to Pivotal Big Data Suite. Then, we announced Pivotal HD 3.0, now aligned with the ODP core. This included updates to many Apache™ elements, including HDFS, YARN, Ambari, Pig, Hive, HBase, Zookeeper, Oozie, Nagios, Ganglia, Ranger, Knox, Tez, and Spark.
The possibilities with data are now so vast, companies are struggling how to transform their corporate culture as well to innovate with data.
Real-Time Big Data: Pivotal GemFire, Stock Prediction Architectures, and 5 Terabytes of Memory
Since Pivotal GemFire, well known as a distributed in-memory database, was pushed out into the open source world as Project Geode last month, several really cool things happened. The website and project source code moved over to the Apache ™ incubator. Some excellent, new content has been put out there—a series of videos on the architecture, a concise history about the problems GemFire was built so solve, and a pointer to a great talk given by Pivotal GemFire’s chief architect on in-memory data grids.
We also just had an awesome meetup titled “Creating a Highly-Scalable Stock Prediction System with R, Geode, & Spring XD.” Over 50 people were in attendance, and our own Fred Melo presented, covering some of the same material in his recent article on real-time streaming and machine learning.
As we mentioned earlier, GE also released a performance study white paper on Pivotal GemFire, proving that it is truly industrial strength using a cluster of 46 machines and 5 terabytes of memory to ingest data at a rate of 100,000 time series data points per second over 5 days. We are happy to report the test ran flawlessly.
If that wasn’t enough for Pivotal GemFire and Project Geode, we also made Pivotal GemFire available as a service on Pivotal Cloud Foundry—this allows session state to move into its own, separate, highly scalable service. Application instances can now die, but user sessions will continue.
Oh, Apache Geode will be at OSCON Portland, July 20-24!
Programming on Clouds: Python, PHP, Ruby, Go, Spring, Microservices and More
PyCon 2015 took place late last month, and there is a good roundup of videos, and so are the videos for RailsConf 2015. Did you hear that RubyGems.org passed the 5 billion download mark? People are saying the upcoming PHP 7 is going to have 100%+ performance improvements, and it is really good to hear that the Node.js fork, IO.js, is joining the Node.js Foundation to ultimately merge the two back. Pivotal is also proud to be a sponsor of the upcoming GopherCon 2015.
In Spring, 11 Pivots spoke at Spring I/O. If you missed the Microservices Conference earlier this year, the replay of our own Oliver Gierke is now on InfoQ, and he explains Spring Cloud—A Toolbox for Distributed Systems (or you can just grab the slides). We also posted the past webinar of Smarter Service-to-Service Invocations with Spring Cloud (actually, it was called “The Bootiful Microservice” and really explains a view of microservices with Spring Boot and Spring Cloud). The SpringOne2GX replay of Centralized App Configuration with Spring and Apache Zookeeper™ is also posted, and there was a popular video posted on Domain Driven Design and Microservices from JAX 2015. We also shared how Spring XD lays a foundation for streaming and machine learning systems. Whew. For the last two Spring topics, the team posted a great write up that explained Spring and Apache Kafka™—how it works with Spring Integration, Spring XD, Lattice, and Docker. Did you know Apache Kafka™ powers parts of Netflix, AirBnB, Goldman Sachs, Paypal, Foursquare, Twitter, and LinkedIn? Spring Social Facebook 2.0.0 was released and targets version 2.3 of the Facebook Graph API. Oh, Spring Boot broke 1 million downloads a month in April too.
Containers and PaaS
Google was in the news for a couple of big announcements: their support for Rocket, Docker’s rival, on Kubernetes and they announced Google Cloud Bigtable. Not to be outdone, IBM announced that its Watson cognitive computing platform just got bolstered to run on hybrid clouds.
The world of Cloud Foundry continues to move in rapid fire fashion, and we will have more to share coming off the heels of CF Summit. For now, we recently gave a deeper dive on Lattice and Cloud Foundry, announced new capabilities in Pivotal Cloud Foundry, which includes high availability for MySQL. In addition, HP donated its .NET server and development stack to Cloud Foundry so MSFT workloads will run on it along with MS SQL Server.
IoT and Mobile App
In the world of mobile apps, Mobilegeddon is upon us with the new mobile focused update to search, leaving non-mobile friendly sites out of search results, the race to be mobile savvy has never been as important before. We also saw Google’s mobile search results prompt us to install mobile apps that were relevant to your search results.
If you are still thinking IoT is just a buzzword, just look at the hardware trends—oops—we mean the application runtime trends. Chip companies are headlong into IoT, moving past sensors into local micro-compute nodes. For example, Qualcomm is talking about smarter devices that compute on the device instead of the cloud (meshing too), and Samsung’s Artik includes 8 ARM cores and 2GB of RAM in a 29mm by 29mm footprint. Fujitsu just showed protoypes of stamp-sized, wearable sensor tags with common smartphone components like accelerometers and GPS. With those micro-platforms in mind, GE announced its Industrial Dojo for Cloud Foundry and partnership with Pivotal on a TCP Router to connect industrial protocols like CoAP, DDS, and MQTT with Cloud Foundry services. Want to learn more? At QCon New York next month, we will be hosting the Mobile and IoT at Scale Track. Not convinced you need a different platform for IoT?
Facebook open-sourced React Native for mobile app development—it apparently goes a long way to solve the HTML versus native problem. The 300 million Tango app users will now get an addition to their messaging functionality—the Tango Shop will allow users to buy from Walmart.com and Alibaba.
Lastly, guess who has the fastest growing mobile app in the land?
It’s not Uber, Tinder, or Nike. It’s Kohl’s—about 2180 miles from Silicon Valley.
Open Source: The 2015 Future of Open Source Survey
In closing, we’ve been eyeballing the 2015 Future of Open Source Survey. Over 1300 people responded. There are lots of statistics. Notably, only 3% of companies don’t use open source at all and 78% say they run part or all of their operations on open source. In 2007, survey comments had people calling open source a gimmick. Nice.
About the Author
