Mesosphere Links To Key Supporting Systems For Container Use- Valutrics

Mesosphere announced its Data Center Operating System now works with data and message management systems from Lightbend, Confluent, and DataStax.

(Click image for larger view and slideshow.)

Mesosphere’s Data Center Operating System for container orchestration is being teamed up with key open source systems to build more complex microservice applications. The combination is designed to give developers the ability to use containers in state-aware systems, as well as stateless ones.

A stateful application can order updates to a database system, read messages in required sequence, or synchronize operations of interdependent applications. Stateless code can’t, and that can make all the difference in the kinds of applications that can be built, said Florian Leibert, Mesophere cofounder and CEO.

Docker, CoreOS, and Mesosphere have acknowledged containers in their early days were essentially stateless packages of code, able to be constructed in a uniform manner and moved around by developers in predictable ways.

But Leibert claimed in an interview that Mesosphere is changing what containers can be used for as they become parts of more sophisticated applications. By implementing its DC/OS Enterprise Edition with one of three partners’ platforms, containers can take on stateful operations in a combined system.

(Image: chuyu/iStockphoto)

The partners are: The DataStax commercial version of the open source Cassandra NoSQL data management system; Confluent’s version of open source Apache Kafka data streaming, the Confluent Platform; and the Lightbend integration of Apache Spark data-handling, Apache Akka messaging system, and Lightbend’s own Lagom framework.

The combination is called the Lightbend Reactive Platform. Lightbend built the Lagom framework for Java-based microservices. It was previously known as Typesafe and renamed itself on Feb. 23.

Liebert said it’s essential for a container orchestration system to be able to work with other systems for the containers to function in modern apps. Mesosphere’s DC/OS can do so because it was built with “two-level scheduling,” which allows DC/OS to maintain general container operations on a large computer cluster, but yields the specifics of workload scheduling to the relevant subsystem, such as Lightbend Reactive Platform, DataStax, or Confluent Platform, which have their own schedulers.

Mesosphere DC/OS is based on the open source Mesos container orchestrator. A UC Berkeley paper first published in 2010, “Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center,” describes how its two-level scheduling was built in.

Leibert said such a capability is essential if containers are going to live up to their hype as a future critical component of applications. He called two-level scheduling a required element of the second phase of container operation, or Containers 2.0. “Container 1.0 systems, including Kubernetes and Docker Swarm, use a single monolithic scheduler,” Leibert wrote in an Aug.1 blog post on the Mesosphere website.

The drawbacks of the so-called Container 1.0 world are that it’s hard to deploy a stateful application for data synchronization and database use, among other things, he said.

Architects may do so by designing a separate cluster for a stateful service, then trying to get it to work with other elements of the application. Any upgrade to the stateful service will require the application to shut down and await a restart, he added.

“If you wanted to add a Kafka node to a cluster to get it to scale up, you have to stop the application, provision the node, and restart the application,” he said.

Kafka, on the hand, was architected to communicate with the open source Mesos container orchestrator, which sits at the heart of Mesosphere’s DC/OS. DC/OS can start a cluster of a 1,000 servers, and then follow a systems administrator’s directions on which subsystems to prioritize on it.

Those subsystems might include Cassandra, Kafka, and Spark. Provided they’re supplied by DataStax, Confluent Platform, and Lightbend Reactive Platform, the DC/OS cluster operating system would be able to hand off scheduling of workloads to each, depending on its priority.

[Want to learn who’s backing Mesosphere? Read Microsoft, HP Help Pump $73 Million Into Mesosphere Container Management.]

“If Spark was being used for Server log analysis, that might be a lower priority than website clickstream analysis,” and could be scheduled accordingly through DC/OS supervision and handoffs.

Since Spark is a batch process, the log file analysis could be run during a slack period, with the virtual servers returned to the cluster as soon as traffic picked up again, said Leibert in the interview.

Administrators may set up a similar system with products other than DC/OS, but Leibert claimed they’ll have to do work and have fewer automated functions to rely upon.

Mesosphere is trying to make it easy to add nodes to a cluster or containerized microservices to an application, he said. It also wants a system on a cluster to survive a node failure without losing track of the application’s state.

The two-level scheduling “is a big differentiator for Mesos,” he said, and by extension, Mesosphere. It can scale over a cluster of tens of thousands of servers. “No other system has the scalability of Mesos” at this time, he claimed.

Stateful container deployment is currently in use at Verizon, Esri, Autodesk, Wdellframe, and Time Warner Cable.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive … View Full Bio