Born from the experience of working with over 3000 of the largest in-memory data grid projects out there, including China Railways, GIRE, and Southwest Airlines, and we’ve invested more into the needs of the most demanding enterprises: more scale, more resilience, and more developer APIs.
This release is a significant enhancement for developers looking to take their big data NoSQL apps to massive scale. For the complete technical details, you can check out the new datasheet, and official product documentation.
Here’s what’s new, sorted by the 5 areas that GemFire does best in the industry:
Providing Scale Out Performance
This is why most of Pivotal’s customers begin looking at GemFire in the first place—because they can’t make traditional RDBMS’s scale with the number of concurrent transactions and data they need to manage.
Pivotal GemFire manages data in-memory distributed across multiple systems on commodity hardware—100’s of nodes if you like—in a shared-nothing architecture. So there’s plenty of compute and memory to host all your data to get real-time response.
We’ve added in-memory compression, effectively giving each node the capacity to hold up to 50% more data. Compression is achieved through Snappy, a speed-optimized algorithm, although the compression codec is replaceable to whatever algorithm you want to use.
Maintaining Consistent Database Operations Across Globally Distributed Nodes
The drawback to a massive shared-nothing in-memory data grid is that your data is not durable, and its very hard to keep changes in one node consistent with all the other copies of data. Here is where many other products in the category fall down, especially as data volumes scale up.
Pivotal GemFire provides numerous features, allowing it to operate as a massively distributed operational database. Consistency mechanisms ensure that changes to data in one node do not collide with other changes, even if network problems split nodes off from one another. It provides write-optimized disk-based persistence to ensure no data is lost in the event of a total cluster failure. This then allows us to build up all the other kinds of database operations you’d expect such as queries and transactions.
Allowing High Availability, Resilience, and Global Scale
Funny thing about large scale-out operational applications: they tend to be very strategic and important. Here’s where you collide with the CAP theorem. Most of the solutions in the market that provide high availability of data can mess up consistency in concurrent update situations. Those that are consistent have trouble maintaining availability.
To be clear, Pivotal GemFire emphasizes partition tolerance and consistency over availability of data, although you can configure the system differently. GemFire, however, has many features to enhance availability of nodes and thus data. This includes node fail-over to replicas in the event of system failure or network partioning, and the ability to run multiple clusters connected via a WAN, giving you multi-site disaster recovery capacity. This same cluster-to-cluster WAN connectivity also allows your to scale your massive application to a global scale. The gateway filtering feature helps you deal with annoying localization problems, such as private data needing to remain within certain country borders.
We’ve made resiliency—bouncing back after failure—even more automated. Now nodes will start up much faster, reconnect automatically, and get updated on the latest state of the data that they missed. In addition, we’ve enhanced the software update process, so that even when you update the core GemFire software running on a node, you can still leave the rest of a cluster running, so there’s no need for maintenance downtime!
Powerful Developer Features
GemFire is an in-memory distributed database for custom applications. So yes, there’s a ton of developer functionality. I’ll save it for the data sheet to list most of them, however, a few special capabilities make Gemfire unique in the market. These include a powerful, persistent asynchronous event queue which can act like an enterprise messaging queue for your application. Related to this is the continuous query feature, which continues to return query results whenever a change to data matches your criteria without having your application continuously poll the database.
First, we’ve added a full featured REST API, which means that any programming language used for high scale applications, including Ruby, Python, Scala, and Node.JS can now take advantage of GemFire capabilities. And, you don’t need to add a client library to your application. Second, we’ve updated to comply with Spring Data GemFire version 1.x. This is simply the fastest way to develop powerful applications with GemFire.
Administration of Distributed Nodes
Taking care of a grid of systems requires some good administrative tools. Pivotal GemFire provides a lot of automation to minimize administrative tasks, such as auto-tuning distribution of data to optimize usage of available system resources between nodes. Several tools are available for analyzing current status, analyzing historical performance to diagnose potential bottlenecks and command-line tools, that can be easily automated, for taking action on nodes.
We’ve improved configuration so you can address all nodes in a cluster with a single service. We’ve also enhanced command line tools to better work in cloud deployments, and further enhanced the capabilities of the browser tool to look at data stored within your nodes.
If you are an existing GemFire customer, additional incremental improvements have been added that you’ll appreciate. See the complete release notes for details.
About the Author
Greg Chase is an enterprise software business leader more than 20 years experience in business development, marketing, sales, and engineering with software companies. Most recently Greg has been focused on building the community and ecosystem around Pivotal Greenplum and Pivotal Cloud Foundry as part of the Global Ecosystem Team at Pivotal. His goal is to to help create powerful solutions for Pivotal’s customers, and drive business for Pivotal’s partners. Greg is also a wine maker, dog lover, community volunteer, and social entrepreneur.More Content by Gregory Chase