Since our last Build Newsletter, much has happened in open source software. Namely, Pivotal made a bold move announcing we will be open sourcing all of our core big data technologies, including GemFire, HAWQ and Greenplum Database. In addition, this month’s Build Newsletter covers other key happenings in open source, the Internet of Things, and devops/agile transformation.
Open Source: Standards and Compatibility
After the big stir at Strata + Hadoop World, one might ask, “Why are companies joining the Open Data Platform Initiative (ODP)?” Well, check out what ODP participants say for yourself: Verizon, Teradata, Altiscale, SAS, InfoSys, IBM, Hortonworks, and, the newest member, WANDisco.
Like the others, we think enterprises want more solutions based on open source, and the Apache Software Foundation® (ASF) has achieved tremendous respect for furthering this cause. However, debate still surrounds our announcement of the ODP and its alignment with the ASF since:
- Cloudera has been public about their disagreement with the new initiative, firing a shot with an “Only Dollars Pay” comment and questioned the use of the word open. That day, they also announced paying to be a Platinum Sponsor of the ASF. Interesting.
- Our own Roman Shaposhnik, who left Cloudera because of differing views on this topic, addressed their assumptions and questions on the ASF relationship. One of his comments sums it up—some Apache Hadoop® vendors claim 100% compatible versions of Hadoop when they have made 700+ code changes to the core, making compatibility a big question mark. The aim of the ODP is to make certification of every participating distribution of Hadoop open source, and to coordinate a huge amount of engineering contributions to the ASF by member companies.
In other words, we are opening up certification for member Hadoop distributions and encouraging a huge amount of investment to advance open source big data technologies. Since media outlets originally covered the news, Datanami, Gartner, and SiliconANGLE have all taken a deeper look—please feel free to tell us what you think and share in the comments below.
Open Source: More Highlights
Apache HBase 1.0 was released after 7 years of development and begins to take on Cassandra and MongoDB—it seems like 7 years is a long time coming. A newer project, Apache®Tajo v0.10.0 (also SQL on HDFS), was also announced alongside releases of Apache Solr™ and Apache Lucene™.
In case you missed it (and related to the above ODP info), we are open sourcing Pivotal GemFire (big data in-memory), Pivotal HAWQ (SQL on HDFS), and Pivotal Greenplum Database (massively parallel analytics) in a quarterly cadence. While specific dates are not available, we expect to see progress on this in next month’s edition of the Build Newsletter.
As published on InfoQ, Groovy has found a home at the Apache Software Foundation. As well, Node.js is getting its own foundation with support from IBM, Paypal, Microsoft, and the Linux Foundation.
In the past month, Spring.io released new versions of the Spring Framework, Spring XD, Spring Boot, Spring Cloud, and more. For those new to it, “Spring XD is to Hadoop® what Spring was to JEE” (now with 25 sample projects), and Spring Cloud does the same for distributed systems.
Mobile developers received a new open source framework from Apple, and Stanford University researchers discovered that, in 24 hours, 11,000 people signed up for their study using Apple’s brand-new ResearchKit.
In the world of containers, Docker acquired their way into networking, Docker 1.5 came out, and CoreOS added Docker support. As well, a Microsoft leak shared some of the company’s vision for a tiny, container-based cloud-centric OS. In the midst of this shifting container landscape, we worked with partners to make .NET available on Cloud Foundry using a Windows container library from Iron Foundry.
Internet of Things (IoT)
While IoT has been a tremendously hyped technology, O’Reilly Publishing founder, Tim O’Reilly, went on record to say that many developers and companies in the space are missing the point. Wow.
Another industry-led, open-source, interoperability-focused standards organization, the AllSeen Alliance, made IoT-related announcements at Mobile World Congress with 22 new members and also gained 350 members from the EnOcean Alliance.
Canonical announced IoT news with Microsoft and Amazon, while Eclipse released MQTT projects for IoT. At Pivotal, we have a new IoT leader who hailed from both Microsoft and Amazon, and while he is busy building Pivotal’s new IoT Lab, other Pivots have been showing IoT progress with the connected car and Internet of Humans.
DevOps/Agile Enterprise Transformation
We recently heard first hand from our customers that Pivotal Labs and Cloud Foundry are helping them transform into software companies, much like we are doing with Sundance.
The devops movement continues. Organizational learning is being applied to devops with excellent anecdotes, and devops patterns for microservices may even impact organizational models. Yet, we are still all learning about the barriers to overcome so that 2016 becomes the year of devops. Assuming we do these things, perhaps we can update production 70 times per day like Etsy. It looks like this devops thing might even catch on with big data.
Recently, some of our experts learned a lot about devops and the way operations runs with Cloud Foundry, but the devops type of change isn’t for every company unless you see the writing on the wall and decide to build a software factory.
Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
About the Author
