This month, the big news for app developers and architects spans across open source, digital transformation, big data, in-memory data platforms, machine learning, data science, programming language popularity, developer salaries, the world of Cloud-Native, and security issues of the Internet of Things.. There is so much good stuff in here, starting off with a roundup of OSCON 2015—the predominant open source conference. Upcoming events are listed too!
Open Source: OSCON Roundup
Last month, OSCON 2015 took on it’s annual role as the predominant open source pulpit—we were excited to be a Gold Sponsor. O’Reilly has already pumped out 75 videos from the event, and there was plenty of anticipation for Kubernetes 1.0.0, which Wired also covered.
On to recaps and notables. Ben Kepes mentioned one of the more unexpected announcements, that Hitachi’s Unified Compute Platform will support Google Kubernetes. The talented team at The New Stack organized a pre-event dialogue and published it as an article, podcast, and on YouTube. The article talked about things like the advantages of Go on Docker, but it really focused the fact that microservices talks, in one year, went from zero to 30 talks—actually the only talk in 2014 is one that Pivotal did. The New Stack team also did a Live at OSCON recording. As well, Opensource.com published a large number of live reports and speaker interviews.
Digital Transformation, Strategy, Agile, and DevOps
Focusing on IT leadership struggling to demand change, Steve Dennning, Director of the Scrum Alliance, penned a great, in-depth piece on Forbes about the corporate need for innovation, the success of agile models like Scrum, and the issues these methods have inside corporations.
The Altimeter group worked with Capgemini to research business investments in innovation centers—the paper starts off by explaining the importance of innovation in business and says, “Fifty-two percent of the Fortune 500 have merged, been acquired, or gone bankrupt since 2000.” Now, we wouldn’t say this if we haven’t heard it from many of our customers. But, if you haven’t experienced a Pivotal Labs environment yet, then it is really hard to get a sense of a truly innovative environment. This is why Inc.com just named us to The 10 Web Dev Companies You Want to Work With list.
With Agile and DevOps, there have been a ton of great perspectives published in the last month. One of the keynotes from Agile2015 promoted adaptability for Agile, making it OK to tweak for fit in specific environments. One expert recently covered “Agile as a Process vs. a Concept,” and another penned “Why Agile is Fragile,” covering the importance of quality and planning not just speed and iteration. We ran across three other really good pieces of advice—what character traits to avoid when recruiting for DevOps, how CIOs can build a new operating model, and enabling agile for distributed teams.
Lastly, Matt Asay recaps the 2015 State of DevOps Report and highlights the most interesting findings. As well, we are excited to have a new leader for our digital transformation practice and welcome Siobhan McFeeney, who is considered one of the most influential women in the Bay Area.
Big Data: Data Platforms and Data Lakes
At the end of the day, only one V of the “Big Data Vs” matters. That V is value. Datamation published an in-depth view of the current Hadoop landscape and ecosystem, pointing out that SQL and Java are two pillars that make adoption much easier. This is why Pivotal Greenplum and HAWQ are two key keys to our growth.
As an example, a recent SQL for big data article was recently published—to stop over $100M per year in electricity theft, BC Hydro is feeding data from two million smart meters into Pivotal Greenplum and using SAS predictive analytics to understand where energy losses are happening. In this case, EMC provided the data lake storage.
On to other topics. Sometimes, industry veterans offer very worthwhile things to think about. This article covers the concept of “data swamps” and reinforces the importance of schemas, metadata, profiling, cleaning, refining, enriching, and validating data. If data is the new oil and analytics is the combustion engine, dirty data must be addressed.
In Memory Data Platforms: Geode and Apache Spark™
First off, we are excited to announce the winner of our Apache Geode Ambitious Apps at Amazing Scale Hackathon. Taking first place, Dynamic Geode Warping built a pattern matching capability for real-time scenarios—a quite impressive, yet simple demo and GUI.
In the Apache Spark™ world, KDnuggets wrote up an excellent interview with the creator of Spark. There is also a recent article by InfoWorld’s Serdar Yegulalp that points out the recent investment in Spark by companies like IBM, Microsoft, and Huawei. Adrian Bridgwater explains Huawei’s “Project Astro,” combining the best bits of Spark, SQL, and Hbase. Microsoft recently added Apache Spark support for Azure, and IBM has been investing heavily in Spark. Lastly, the team from theCUBE noted some important points about Apache Spark as they concluded the Hadoop Summit 2015—importantly, that Spark isn’t as enterprise-ready as Apache Hadoop® and that the Open Data Platform is going to serve a critical function.
From our perspective, Apache Spark is one components within our Big Data Suite, and it has long been on the road map for Spring XD as well. This article on InfoQ explains how Spring integrates with Spark, including the use of Spring Batch and Spring Integration. There is also an Introduction to Spark for the Spring Developer presentation. Looking at the Spring XD documentation, you can also see support for Pivotal GemFire, JDBC, JMS, Kafka, Redis, MongoDB, RabbitMQ, Spark streaming and much more. Here was Databricks’ view of our supportfrom last year and the Apache Geode (Pivotal GemFire) proposal gives a high level comparison of Geode and Spark.
One key difference between Spark and Apache Geode (Pivotal GemFire)—one of them has been performing Wall Street’s trades for a long time and is enterprise-ready.
Advanced Analytics: Machine Learning and Data Science
Machine learning (ML) interest continues to grow. If you aren’t familiar, here is an excellent overview of a modern, real-time ML system based Spring XD, Apache Spark, Apache Geode, Apache Hadoop, and Pivotal HAWQ.
Machine learning is solving all types of problems that have historically been hard. There are evolutionary and disruptive examples in advertising, finance, manufacturing, media production, supply chain, retail, energy, IT security, news, and every other industry. Of particular note for software folks, some say machine learning will revolutionize the software industry itself. Check out MADlib, our open source machine learning project or read about some advances of machine learning into deep analytics, as published by our data science team.
In the “hey, that’s cool” category, there is a new programming language to manage heterogeneous robot swarms. Also, some new research found that nearly 8 out of 10 “IoT cloud development tasks” have to do with big data and analytics. Out of what was left, the next two were middleware and server-side development. Lastly, there is some news out of London that software exists to transplant and re-use code automatically, much like an organ donor.
Cloud-Native Foundations, Platforms, Containers, and Companies
The race to adopt Cloud-Native architectures is gaining speed. At Pivotal, we have been working on this for a while largely through our efforts with Cloud Foundry, but also regularly posting blogs and even writing a book. As a signal more companies are working together to make this easier for companies, the Linux Foundation announced the Cloud-Native Computing Foundation at cncf.io, and twenty two companies were part of the announcement, including AT&T, Cloud Foundry Foundation, Docker, eBay, Goldman Sachs, Google, IBM, Joyent, Twitter, VMware, Weaveworks, and many more. Importantly, Google is taking Kubernetes to the new foundation. As Brandon Butler explains, the group plans to create a stable, operable, well-integrated group of projects for cloud-native applications—apps that are container—packaged, dynamically scheduled, and micro-services oriented. It will also work together with the Open Container Initiative (OCI), another Linux Foundation project.
Speaking of the OCI, John Waters gave a an update on the group since its initiation a month ago. He goes on to explain how the number of members doubled in one month and interviewed several people from Docker, who made the first, big upfront investment. Waters also wrote a separate article, interviewing Pivotal’s James Watters, and explaining the intersection between Spring Cloud, containers, microservices, and Cloud Foundry.
From Pivotal’s perspective, one of the most exciting things we’ve seen in the past month is the groundswell around structured versus unstructured cloud-native application platforms—Wikibon’s Brian Gracely points out some key differences and explains how very-early-adopter web-innovators, like NetFlix and Pinterest, built their own app platforms that were unstructured. He then explains how structured PaaS offers a lot more tooling to support and address enterprise needs. He does cite one of our talks at Cloud Foundry Summit, but there is a also a good set of questions to help people make a decision between the two.
As covered by InfoWeek’s Charles Babcock, GE Software is moving Predix to the cloud. Later this year, GE is moving internal systems to the Predix Cloud and will then make it available to customers is early 2016. GE says this business had $4B in revenue in 2014 and will have $6B in 2015—that means GE is a pretty big software company these days, a revenue stream all created since 2011.
Two key articles were also written about Pivotal Cloud Foundry telco partners—CenturyLink and Telstra. We are partnering with the biggest telcos in the world to innovate and delivery innovation with cloud-native platforms.
Hackers Kill a Car on the Highway: Things that Connect to the Internet
The Internet of Things is a hot new area with massive expansion throughout industries of all types, and, as expected, has its fair share of growing pains. To help guide companies through some of the challenges that arise, our CTO EMEA, Chris Mills, penned an article on IoT called Future Proofing your Business with the Internet of Things.
Of course, we are already getting some stories of some failures in IoT. Wired featured some remote hackers who killed a Jeep on the highway. There are also some scary stories about hacking chemical plants or power plants. We expect that security will be a continued area of concern for some time, and will be sure to continue to cover developments here.
Upcoming Pivotal Events
- Apache Geode Clubhouse: Roundtable – Tuesday Aug 18, 2015 9AM to 10AM
- Pivotal Open Source Hub: SF: Introducing POSH Unplugged! – Thursday Aug 20, 2015 6PM to 9PM
- Golang UK: Pivotal Gold Sponsor – Aug 21, 2015
- Apache Geode Clubhouse: Roundtable – Tuesday Aug 22, 2015 9AM to 10AM
- Pivotal Open Source Hub: SF: Deploying Microservice Architectures with Spring Cloud on Cloud Foundry – Wednesday September 23, 2015 5:45PM to 8:30PM
- QCon Rio 2015: August 24th
- Pivotal Toronto User Group: Tomcat Overview with Mark Thomas – Wednesday Aug 26, 2015 7PM to 8PM
- OpenStack Silicon Valley: Conference: Aug 26 – 27, 2015
- VMWorld 2015 US: Pivotal Booth/Presentations: Aug 30 – Sep 3, 2015
- Very Large Data Bases: Aug 31 – Sep 4, 2015
- Jenkins Conference / CD Summit: West: Pivotal Diamond Sponsor: Sep 2 – 3, 2015
- Pivotal Open Source Hub:PA: Enabling R for big data using open source tools PL/R and PivotalR – Thursday September 3, 2015 7PM to 8PM
- Pivotal Open Source Hub: PA: Cloud-Native Development Using Lattice – Thursday September 10, 2015 5:45PM to 8:30PM
- Cloud Platform Seminar : Atlanta: Presented by Pivotal and CenturyLink: Sep 10, 2015
- SpringOne 2GX: Sep 14 – 17, 2015
- Postgres Open: Pivotal Platinum Sponsor: Sep 16 – 18, 2015
Editor’s Note: ©2015 Pivotal Software, Inc. All rights reserved. Pivotal, Greenplum Database, GemFire and HAWQ are trademarks and/or registered trademarks of Pivotal Software, Inc. in the United States and/or other countries. Apache, Apache Hadoop, Hadoop, Apache Lucene, Apache Cassandra, Apache Geode and Apache Spark are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
About the Author