techponder

OpenLMI is included in Fedora 19, which was released on July 2, 2013. (A preview release of OpenLMI was included in Fedora 18.) The focus of this release is on delivering a set of OpenLMI Providers which have the capability to configure and manage resources on a system, with a focus on storage, networks, services, software and users.

This involved solving some of the most challenging issues for Linux system management – providing a remote interface capable of configuring and changing core hardware and software settings. More information is available at http://www.openlmi.org.

Fedora 19 includes a new set of OpenLMI Providers with extensive capabilities for configuring a system:

Storage

• Enumerate local drives (block devices).
• Obtain information on drives and luns including I/O statistics.
• Partition, including support for GPT partitions.
• Format – EXT2, EXT3, EXT4, XFS and btrfs.
• Build RAID sets with mdraid.
• Perform logical volume management with lvm.
• Mount/unmount file systems.

In addition to managing local storage, the provider can manage network storage services that present a lun – for example, iscsi, Fibre Channel, and FCoE. This means that OpenLMI can partition, format, RAID and lvm against these luns.

Networks

• Enumerate NICs.
• Get information on NICs (MAC address, link status, etc.)
• Bring up/bring down NICs.
• Set IPv4 address and netmask.
• Set IPv6 address.
• Set default gateway.
• Set dns servers.
• Manage static routes.
• Configure network bridging and bonding.
• Notifications (events) for changes in network devices and settings.

Power Management

• Halt, reboot, suspend or hibernate a system.
• Report and set power states – Pstate and Cstate.

Since OpenLMI runs inside the OS, other tools are needed to power-on a system.

Service Management

• Enumerate system services and their status.
  • Service name.
  • Service description.
  • Service status.
• Start/Stop/Restart/Reload a service.
• Enable/Disable a service.
• Configure a service to start at boot.
• Event based monitoring of service status.
• Installation/Update/Removal of a service (using Software Management)

Software Management

• List all rpm based software installed on a system (including version).
• List available repositories.
• List available packages per repository.
• Search repositories.
• List files owned by a package.
• Perform a package integrity check.
• Install/Update/Uninstall software.

Account Management

• Manage local accounts.
• List user accounts and settings.
• Create/Update/Remove user accounts and groups.
• Query and Change user account settings.
• Notifications (events) for account changes.

Hardware Information

The Hardware Provider delivers a wide range of information on the hardware, ranging from CPU architecture to details of memory configuration to detailed chassis information. There is enough in the Hardware Provider to justify a separate post – stay tuned for more details.

Since OpenLMI is based on open industry standards, it is useful to take a closer look at these standards, including the organizations that develop and support them.

Distributed Management Task Force (DMTF, formerly “Desktop Management Task Force”) is an industry organization that develops, maintains and promotes standards for systems management in enterprise IT environments. These standards allow for building systems management infrastructure components in a platform-independent and technology-neutral way. By creating the open industry standards, DMTF helps enable systems management interoperability between IT products from different manufacturers or companies. Full details are available at http://www.dmtf.org.

DMTF was founded in 1992. It is a Standards Development Organization where companies, other organizations and individuals can become members. In 2005, DMTF had more than 3500 participants out of more than 200 organizations and companies (such as AMD, Broadcom, CA, Inc., Cisco, Citrix, EMC, Fujitsu, HP, Huawei, IBM, Intel, Microsoft, Oracle, Red Hat, VMware). The DMTF is organized in working groups where the participants jointly develop and maintain the standards. DMTF has alliances with a number of other organizations and with academia.

The goal of DMTF is to provide an industry standard for managing computer systems and subsystems. In addition to specifying standard data model and communications specifications, the intent is to ensure that implementations of the standard are interoperable – that low level service providers work with multiple management frameworks, that a single framework can call low level service providers from multiple sources, and that implementations of a single low level service provider from different sources provide consistent behavior.

Core of DMTF is the Common Information Model (CIM) – The CIM schema is a conceptual schema that defines how the managed elements in an IT environment (for instance computers or storage area networks) are represented as a common set of objects and relationships between them. CIM is extensible in order to allow product specific extensions to the common definition of these managed elements. CIM uses a model based upon UML to define the CIM Schema. CIM is the basis for most of the other DMTF standards.

The DMTF has specified a number of CIM models addressing storage, servers, services, and other functional domains. Implementations of many of these models, called Service Providers, are available on Linux. In addition to the service providers, there are specifications for communications and language bindings, which are also available on Linux.

CIM providers can be used to provision, configure, manage and monitor hardware, services, and applications.

CIM is typically configured as a distributed service, with a CIM server on a local system talking to CIM clients on other systems.

Relevant DMTF Specifications

The primary data models and functions we are interested in are the server centric SMASH (System Management Architecture for Server Hardware) profile and the storage centric SMI-S (Storage Management Initiative Specification). These specifications have been significantly updated over the last few years and are under active development.

The infrastructure is defined through a set of other DMTF specifications that cover object specifications, communications, APIs, API language bindings, security, and other specifications required to define and implement a standard environment.

DMTF Technology Stack

In addition to the CIM object models, there are standards for communication and software development. The major communications interfaces are WBEM (Web Based Enterprise Management), an XML over http/https protocol, and WSMAN.

CIM Providers talk to the outside world through a CIMOM (CIM Object Monitor). OpenLMI uses the OpenPegasus CIMOM. OpenPegasus supports WBEM and CMPI.

While you can work with XML directly, it isn’t easy. WBEM includes a set of language bindings for Python, Java, and C/C++. These language bindings present the CIM objects as native objects – as Python objects for Python, Java objects for Java, and C++ objects for C_++. This greatly simplifies using OpenLMI.

CIM Providers are standardized through the CMPI specification, which is used in OpenLMI. Toolchains are available for developing OpenLMI agents in Python, C, or C++.

The value of the DMTF technology stack is that it provides a common framework for talking to all providers. Since the interface is standardized, it is only necessary to understand the specific commands needed for various tasks.

OpenLMI Strategy

There is considerable value to the DMTF models, but they are not designed for implementation, at least not on Linux. While the models have great scope and coverage, they have been designed by groups of experts with a strong modeling focus. Further, the intent is that the models should include everything, and any implementations should choose subsets of the complete model. DMTF specifies recommended subsets as profiles.

OpenLMI takes the approach that the OpenLMI models are based on the DMTF models but are not required to exactly match the DMTF models. OpenLMI starts with the DMTF models, derives a subset that is appropriate for the function being addressed, and extends the model as needed to support Linux.

These LMI extensions are clearly identified by using the lmi namespace. In other words, the LMI extensions have a lmi_ prefix.

There is no goal or requirement in OpenLMI to exactly conform to the DMTF models. The goal is to develop a system manageability API that works. Note that this approach is common to all implementations based on DMTF CIM – everyone customizes and extends the standard models to meet their needs.

The greatest need for improving the manageability of Linux systems is to provide a standard programming interface – an API – for system management functions.

The API should be a low-level interface that provides the needed control over managed systems. It should also support a higher level abstraction, making it easy for system administrators to use it for routine tasks.

The API should support scripting and CLI use. Studies at Red Hat have shown that roughly 75% of Linux system administrators use CLI and scripts. They manage remote systems by logging in through SSH and running the CLI and scripts locally.

This API should provide both a remote interface and a local interface. It should support managing remote systems without having to directly login to each system. The API should support multiple language bindings, allowing it to be easily used from the common tools used for system management.

The API should be supported by a formal object model. Many of the idiosyncrasies of managing Linux systems are due to the organic growth of independent subsystems over time – this makes it difficult to design and provide a consistent interface.

Finally, any system management interface should be built on top of existing Linux tools and capabilities. Any attempt to build complete new subsystems from scratch is doomed by the sheer size of the task – and this doesn’t even touch the challenges of getting new tools accepted by the upstream community and users!

From a real-world perspective, note that Linux subsystems like the I/O subsystem contain tremendous amounts of “institutional knowledge” – these systems are complex because they are solving complex problems on a wide range of real world hardware. In theory, it should be straightforward to replace the I/O subsystem with a clean design. In practice, by the time a new I/O subsystem was able to do what the current I/O subsystem can do, it would be roughly as complex and ugly.

Customer Challenges

Customer feedback indicates that the greatest immediate need is to manage the physical configuration of systems, with emphasis on the ability to configure storage – especially local storage – and networks – on production servers.

A “production server” can be loosely defined as a remote system (you don’t login through a local, physical, keyboard/mouse/monitor), with no graphics (no X-windows or desktop environment installed), with multiple drives and NICs. A production server will often be configured with 4-8 NICs and may have several dozen local drives as well as network storage.

In a Nutshell

The system manageability challenge can be summarized as:

• Provide a standardized remote API to configure, manage and monitor production servers.
  • Physical servers
  • Virtual machines
• Supporting CLI and scripting.
• Providing language bindings for the main languages used for system management.
• Built on top of existing Linux subsystems.
• That Linux System Administrators can – and will – use.

The DMTF/CIM technology stack has been chosen for the agent infrastructure. This provides several building blocks that accelerate development and reduce cost of implementation:

• A standard object broker, called a CIMOM (CIM Object Monitor), which provides a standard method for interacting with Agents. It includes multiple language bindings, a CLI, a standard communication infrastructure, security, and support for both callable and asynchronous events. The CIMOM supports compatible Agents from any source, including Red Hat and third parties. The agents can be open source or proprietary. The OpenLMI Controller is a standard CIMOM. CIMOMs have been included in RHEL since RHEL 5.
• An agent infrastructure which allows functional agents to be developed and deployed. In DMTF/CIM terminology, OpenLMI Agents are called CIM Providers. The agent infrastructure supports introspection, meaning that each agent provides information on the attributes and methods it supports; there is no need to pre-define or hardcode the agents object model. This also allows agents to be easily extended – new capabilities can be added to an agent, the old agent unregistered with the CIMOM and the new agent registered with the CIMOM, and the new capabilities used. If the existing capabilities were not changed, the new extended agent is a transparent direct replacement for the previous agent.

  • Agents can either be called by a script or client tool or can respond to events occuring on the underlying system and produce asynchronous notifications. A callable agent, called a CIM Provider, supports attributes, methods, and associations. A CIM Indicator produces asynchronous notifications that are received by a CIM Listener.

    

• Tooling for building agents. This includes a UML-like tool for defining the attributes and methods of an agent, a compiler for creating an object template or skeleton from this definition, a standard for developing agents and their interfaces, and a test framework. Agents can be built in Python, C, or C++.
• A set of rich and robust (alternatively, large and complex…) object models for enterprise computing systems. Of particular interest are:
  • SMI-S (Storage Management Initiative – Specification) is a comprehensive storage specification, developed with the Storage Networking Industry Association. It covers both attached and networked storage, and has been ratified as an ISO standard.
  • SMASH – Systems Management Architecture for Server Hardware.
• An existing set of agents. These existing agents are mainly focused on reporting and monitoring.
  • Most of the existing agents are contained in the SBLIM package, which has been included in Fedora since at least Fedora Core 5 and RHEL since RHEL 5.

OpenLMI, the Linux Manageability Infrastructure, is building the infrastructure to standardize the management of Linux systems. It is an open project, hosted at www.openlmi.org.

OpenLMI consists of a set of system management agents installed on a managed system, an OpenLMI controller which manages these agents and provides an interface to them, and client applications or scripts which call the system management agents through the OpenLMI controller.

OpenLMI is, at its core, an implementation of open industry standards for remote system management. It builds on top of a standards based infrastructure, and provides a set of system agents that perform operations on managed systems. The infrastructure used by OpenLMI already exists in Linux. However, the existing Linux system agents mainly implement monitoring and reporting functions – they do little work, and do not include the ability to configure or modify a system.

OpenLMI is focused on building system agents that perform these management tasks, with the primary target being bare metal production servers. The initial agents implemented in OpenLMI include storage configuration and network configuration. Later work will address additional elements of system manageability, with the long term goal of providing a complete systems manageability infrastructure.

OpenLMI does not, by itself, deliver a complete systems management solution. OpenLMI provides the low-level functions, capabilities and APIs that can be called from scripts or system management consoles. Thus, OpenLMI complements existing management initiatives by providing a low-level interface to server hardware configuration that can be used by these management systems.

Interfaces provided by OpenLMI include C/C++, Python, Java, and a CLI. A REST API is under development, but has not been released. These interfaces are implemented as language bindings to the underlying system agents – this means that all interfaces provide the same full access to all capabilities implemented in each agent. You have exactly the same capabilities in OpenLMI no matter which programming interface you use.

The basic model for Unix and Linux systems is to use small, single purpose programs or utilities to accomplish a single task and to combine these simple modules to perform a complex task. For system management, these modules are typically used by calling them from shell scripts and piping the output of one module to the next.

The other common element is to use text based configuration files. These files are designed to be human readable – to be edited and understood by people.

The result is a very flexible system that can be configured and maintained though human-centric, text based interfaces. It supports automation, works with simple interfaces (shell and SSH), and enables an experienced Linux system administrator to manage a large number of systems. Little software needs to be installed; you do not need a graphical environment or a large set of additional software; all of the software you need is part of a normal Linux installation. The approach is also portable – you use the same tools to manage systems whether they are physical, virtual, or in the cloud.

There are, of course, some drawbacks to this approach. It depends on a skilled system administrator, and has a fairly steep learning curve. Human readable configuration files can be challenging for software outside of the targeted application to parse. Shell scripts can be fragile, with limited flexibility and exception handling. Each individual subsystem tends to have its own “look and feel”, meaning that you have to learn the idiosyncrasies of each subsystem.

Many of the elements of Linux system management that support direct human interaction have the unintended consequence of making it challenging to automate, resulting in few integrated management suites or integrated graphical interfaces. (Many tools have their own GUI, which talks directly to the underlying application or tool.)

Finally, the entire management model is built around running local tools directly on the system being managed. This means that you have to login to each system to manage it – the most widely used management tool is ssh!

Let me be clear: Linux is a very manageable system. The existing tools, techniques, and approaches work and work well. We believe that it is possible to improve the current state of Linux system management without discarding all of the work that has been done. We believe that Linux system management can be made easier, more flexible, more automated, and support both local and remote systems.

We not only believe this, we are doing it in the OpenLMI project. We will be covering OpenLMI in a series of blog postings and welcome your feedback and comments. We invite you to join us on this journey to move the management of Linux systems to the next level – to provide a common, consistent way to manage Linux systems, from the most complex Enterprise Servers to virtual machines.